Training data formats

Two things produce training positions: the teacher converter, which distils an external corpus, and self-play, which searches its own games. Both write the same raw stream, MPK1, and both are converted into the same parquet schema. Nothing downstream has to ask which one it is reading.

Raw first, parquet second, in both cases. The raw stream is what a producer can append to as positions arrive; parquet is what training and publication read. The stream is the only thing a shard can be rebuilt from, so it outlives the conversion rather than being deleted at it.

The raw stream stores quantities that add. Visit counts, not probabilities; one game's outcome, not a rate. A probability is a quotient, and quotients cannot be combined without the denominators that produced them — a search of 100,000 playouts and one of 100 both normalise to 1.0 and become indistinguishable. Folding duplicate positions together is therefore done at the parquet conversion, on counts, and the division happens once, at the end.

Corpus origin and curation are in teacher data.

MPK1, the raw stream

Defined in manaka_core::stream. Little-endian throughout.

header, 24 bytes
  magic u32 | version u16 | packed_len u16
  pov u8 | policy_source u8 | mate_scale u16
  fv_scale u16 | reserved u16 | eval_coef f32 | flags u32

record, variable length — 12 + packed_len + 8·n_cand bytes
  ply u16 | value_z f32 | value_q f32 | n_cand u16
  packed [u8; packed_len]
  candidates × n_cand : mv u16 | visits u32 | cp i16

• packed — the position, as manaka_core::pack wrote it. 96 bytes, laid out below. It is the position and not an encoding of it, so that a feature space can change without every shard being rebaked.
• value_z — this game's outcome from the position's side to move. Exactly -1, 0 or 1, because one record is one game.
• value_q — the search's root value for the same position, same point of view, squashed into [-1, 1].
• ply — move number. Not necessarily consecutive within a game.

packed_len is in the header rather than fixed by the format, so a producer with a differently sized position rides the same stream. There is no train/validation flag: the split is not a property of the data (see invariants).

What packed holds

96 bytes, manaka_core::pack. Every byte is used; there is no padding.

byte
 0..81   the board, one byte per square, in square-index order
81..88   Black's hand, one count per kind
88..95   White's hand, same order
    95   side to move, 0 Black / 1 White

Squares. Index 0 is 9a and index 80 is 1i, running down a file before moving left:

square = (9 − file) × 9 + (rank − 1)
file   = 9 − square / 9        9 down to 1, the order SFEN writes them
rank   = square % 9 + 1        1 at the top, 9 at the bottom

So ranks 1–3 are White's camp and 7–9 are Black's. Measured over 20,000 rows of a teacher shard: Black's king sits on rank 8 or 9 in 80% of positions and White's on rank 1 or 2 in 84%, each side has exactly one king in every row, no Black pawn ever appears on rank 1 and no White pawn on rank 9.

A board byte.

 bit  7 ── 5 │  4  │ 3 ──── 0
      unused │ col │  kind

Bit 4 is the colour, 0 Black and 1 White; the low nibble is the kind. Black pieces are 0x00..=0x0D and White 0x10..=0x1D. Empty is 0xFF, which cannot collide because no valid byte sets the high bits.

kind	0	1	2	3	4	5	6	7	8	9	10	11	12	13
	K	R	B	G	S	N	L	P	+R	+B	+S	+N	+L	+P

Hands. Seven counts each, one byte apiece, and not in the board's kind order — hands run P, L, N, S, G, B, R, which is the reverse direction. This is the one place in the format where the same seven kinds appear under two orderings, so it is worth checking against rather than assuming. Maxima are 18, 4, 4, 4, 4, 2, 2 respectively; a count is a plain integer, not a bitfield.

The layout is bytes rather than bits throughout. It costs 96 bytes raw and 5.63 in a shard, which is where the argument for packing it tighter falls apart.

A candidate is a move, not a label

mv is a move16 — chisaki's own packing.

 bit  15 │ 14 │ 13 ──────── 7 │ 6 ──────────── 0
         │ ✗  │ pr │  from or 81+kind  │      to

to in bits 0–6, the origin square in bits 7–13, promotion in bit 14, bit 15 unused. A drop puts 81 + hand index in the origin field instead of a square, so the dropped kind is in there too.

This is what lets the format be read without a move generator. A policy label is a direction and a destination, so a pawn stepping to 7f and a lance running there are the same number, and which piece was lifted is not recoverable from it. Recovering it means regenerating every legal move of the position and trusting that the reader's movegen enumerates in the order the writer's did — two crates agreeing, which is not a property a file can state. A move16 carries the origin outright, so both the label and the lifted piece fall out of bit arithmetic.

Candidate::from_square(), to_square(), is_drop() and is_promote() are that arithmetic; a reader in another language needs seven lines of it, not a shogi library.

It is also what makes folding possible. Two records for the same position have to have their candidate lists matched up before the counts can be added, and a move16 is a key that means the same thing in both. A label is only a key relative to the move list it was enumerated against.

Only moves the search visited

Each record states its own n_cand, and moves the search never reached are not written. A reader that wants a legality mask regenerates it from packed, which is the one thing movegen is still good for and is needed only at inference.

There is no fixed-width mode. Measured over two self-play streams:

	a late generation (55,504 rec)	an early one (75,172 rec)
legal moves, mean	74.8	71.7
moves with policy mass, mean	37.03	43.46
same, p50 / p99 / max	27 / 183 / 351	30 / 188 / 368

A fixed n_cand = 5 would truncate 88.6% of the first stream's rows; 64 would still truncate 17.0%. MultiPV rows would fit under some fixed width, and a fixed width would make such a stream seekable by index for free — but the width that fits is whatever MultiPV the generator happened to be run at, so it is a bound borrowed from a setting rather than one the data has. Supporting both means two parse paths and a header field that changes what a record means. The self-play distribution decides it, and it has no such number.

n_cand is in the record, not beside the file. A candidate count that lives in the flag that wrote the corpus has to travel separately, and a reader that guesses wrong reads a valid-looking record at the wrong offset.

visits or cp, and the header says which

Every candidate has room for both, and the two producers fill opposite halves. Both fields are optional and each has an absent marker, so a candidate always occupies the same eight bytes and always says which half it filled. Neither stores a probability.

Teacher generation fills cp and leaves visits absent; self-play does the reverse. Which one a stream did is in policy_source, and the rest of the differences are tabulated further down.

A MultiPV search has no visit counts, so it writes none. It ranks moves and stops; the policy comes back out of cp. Writing 1 there instead is the tempting alternative — it would make merging a single rule, sum the counts on both sides — and it buys nothing, because the one quantity it would carry is one the fold can see for itself. How many searches listed a move is the number of records in the group that mention it, and the fold is holding those records. A constant that says "I am one record" is redundant with being one record.

What it would cost is not redundant. A reader that normalises visits without consulting policy_source would get a uniform distribution over the MultiPV candidates: plausible on inspection, wrong in every row, and raising no error anywhere. Nor would folding smooth it away — only 6.47% of the published corpus's rows share a board with another, so nearly every teacher row folds with nothing and would come out flat. An absent marker makes the same reader divide by zero instead. That is the exact failure this format exists to make impossible, traded for an accumulation step that is three lines either way — the derivation of prob branches on policy_source regardless, so the branch does not go away, it only moves.

The marker is 0, which looks inconsistent with NO_CP being i16::MIN until the rule behind both is stated: the absent value has to be one the data cannot take. For cp that rules zero out, since an even position is a legitimate score. For visits zero is exactly right, because a move with no visits is never stored in the first place — the format only carries moves the search reached. So no constant needs inventing, and a stray zero among real counts is a mixed record, which validation refuses.

What does not change is the layout. The field is written either way, at the same offset, so a candidate can be parsed before policy_source has been looked at. A record whose shape depends on a header field is one a stranger cannot read, and being readable by a stranger is the point.

policy_source therefore decides one thing only: how prob is derived at conversion.

Where the counts are playouts — self-play — they are one search's and not a running total, so a record's counts sum to the simulation budget and nothing more. Measured over 5,000 self-play records and 175,160 candidates at the default 800 simulations a move: every record totals exactly 800, the largest single candidate reached 799, and 46.4% of candidates were visited exactly once (11.2% twice, 6.5% three times). A handful of moves take most of the budget and a long tail sees one playout each.

That tail is the reason to store counts. A move seen once is noise on its own, and as a probability it is 1/800 with nothing to say how thin it is; folded across a hundred games it is either still one visit or it is forty, and those are different claims. Adding pre-divided numbers cannot tell them apart.

DLManaka's counts are not playouts. Its Gumbel search trains on the improved policy — softmax(logits + σ(completedQ)), a float distribution the raw visit counts cannot reconstruct, because the completed Q-values that shaped it are gone once the search returns. So its writer quantises that distribution into the same field at a fixed scale: visits = round(prob × 65536), and a count of 1 means a probability of 2⁻¹⁶, not one visit. Moves below half a quantum drop out. The fold does not care: every record totals the same scale, so summing counts and dividing is an equal-weight mean of the distributions, which is what folding playout counts computes too.

The field is u32 rather than u16 because the total is a setting: playout counts are bounded by the simulation budget, and a run at more than 65,535 simulations a move is configuration rather than fantasy — and the quantisation scale above already sits at that edge. Sums are a different matter and do not live here; folding accumulates candidate counts in u64 and divides them away before anything is written (see folding).

Carrying the scores rather than a distribution over them means a corpus can be re-softmaxed at a different temperature without regenerating it. Carrying the counts rather than their quotient means it can be folded.

The header states the conventions

Point of view, mate scale, FV_SCALE, policy source and win-rate coefficient are properties of how a corpus was generated, and they do not show up in the bytes. Two streams written under Eval_Coef = 600 and under 2 × 756.0865 are byte-identical and put the same win rate 1.26× apart in centipawns: p = 0.90 is 1318 cp under one and 1661 under the other.

field	values
pov	0 side to move, 1 Black
mate_scale	u16, the centipawn value mates are folded to: 30000 for ±(30000 − ply), 32000 for YaneuraOu's CP_LIMIT; 0 not applicable
mate_slack	u16, how far below mate_scale this corpus's mates reach — the threshold is the difference; 0 not stated, and a reader falls back to the card's 320
fv_scale	u16, the FV_SCALE the generating engine's evaluation was read at; 0 not stated
policy_source	0 visits, 1 softmax over cp
eval_coef	f32, the centipawn scale value_q was squashed against; 0.0 not applicable

A reader is not required to match them, only to read them — a converter that refused everything but its own build would refuse the corpus it exists to convert. What the header prevents is reading them as something else.

policy_source is the one a reader cannot infer. The rest are conventions it can be wrong about quietly; this one it has to act on, because it decides which field becomes the policy.

mate_scale, fv_scale and eval_coef describe the centipawn scale, so a stream with no centipawns on it declares none of them. Self-play carries NO_CP on every candidate and gets value_q from the search as a win rate already inside [-1, 1], so nothing in it is measured in centipawns and all three are written absent. This is the same rule as NO_VISITS, applied one level up: a header that named a mate scale it has no mates to state would put two self-play runs into a scale conflict neither of them has any values in, and the fold would refuse them for a disagreement about nothing.

mate_scale holds the number, not an index into the numbers we happen to have seen. An enum would make 1 mean 32000 only for a reader holding the table that says so, and a corpus generated at some third scale would be undescribable until the format grew a variant for it — which in practice means it gets written as the nearest existing one instead, and the header lies. A u16 says 30000 or 32000 outright, covers anything a clamped evaluation can reach, and leaves 0 free as the absent marker, because no mate folds to a score of zero. That makes both centipawn-scale fields the same shape: a number, with zero meaning the stream is not on this scale at all.

Mate scale is the one that bites in practice, and the scale alone does not locate the threshold, which is why there are two fields. Mates are stored discounted by something, and how far that discount reaches is a property of the generator rather than of the scale.

V3 is on the 30000 scale: over 14.3M cp and 66.7M cand_cp it tops out at exactly 29999 with nothing at or above 30000, and 7.0% of its rows are mate distances at the card's threshold of |cp| >= 29680. Its discount is a ply count that the generator replaces with a PV length when YaneuraOu reports a mate out of score cp range, and those run long — measured to ply 315 — so the slack has to be 320.

V4 is on the 32000 scale, and its discount is the real mate distance rather than a PV length. Measured over all 142 corpus files, cp runs −31998 to 31999; over a 40.1M-row sample its mates occupy 31971..31999 and its ordinary evaluations stop at 31753, leaving a gap of 218 between the two bands. A slack of 320 would put the threshold at 31680 and claim the top of the ordinary band — 0.32% of rows and 0.27% of candidate scores — as mates, each one switching from a softmax over evaluations to a one-hot and from a tanh value to a signum. V4 therefore declares a slack of 100, and the threshold is 31900.

An earlier revision of this document said both corpora were on the 30000 scale, on a 1.44M-score sample that topped out at 29999. That sample predates gensfen-parquet, which converts the generator's records and by default leaves the engine's own 32000 − distance alone; it has a --mate-scale 30000 mode that folds to V3's scale, clamping ordinary evaluations under it, and the published corpus was not built with it. A shift applied to a corpus already on the target scale moves every mate 2000 cp past it, and 7% of the rows are mates, so this is not a number to assume.

fv_scale is what makes the centipawns centipawns. An NNUE's accumulated output is in the net's own units and becomes a score only after division by FV_SCALE, and that number does not live in the net file — it ships beside it, and the distributions differ: the benchmark config runs engines at 16, 20, 28 and 40. So one net read at 20 rather than 16 hands back every opinion at 0.8×, and a corpus generated from it carries that calibration in every cp it wrote. Two teacher streams that disagree here disagree about what a hundred means.

A reader does not divide by it. The division already happened inside the generating engine; cp in the stream is centipawns and needs nothing done to it. fv_scale is stated for the same reason pov is — so that two streams can be told apart, and refused — and not for the reason eval_coef is, which a reader actively undoes. It is the one field here that is provenance rather than decoding, which is exactly why it has to say so.

Self-play writes 0: no NNUE evaluation produced anything in it. A teacher converter writes what the generator was run at, and 0 when nobody can say — which is where the published corpus starts from, since its shards carry no metadata at all.

The header carries what a reader has to act on, and nothing it merely wants to know. Every field here changes how a number is read or whether two streams may be folded together. What produced the stream — which engine, at what version, under what options — changes none of that, so it is not in the header and the header stays a fixed 24 bytes. A stream is an intermediate that lives until the shards it produced are known good; provenance belongs on the shard, which is what gets published and kept, and the converter writes it into the parquet footer there.

version is the format's, and the digit in MPK1 is part of the name. They look redundant, and the split is deliberate: the magic says what kind of file this is, the u16 says which revision. Putting both jobs in the four bytes is what produced CZR1 — a format that was renamed, not changed, its layout identical after the header, and because its identity carried its revision the rename cost a second accepted magic that is never written, a reader branch outliving the shards, and a filename predicate matching both .czr and .mnr. All for a change that moved no field. So when the layout changes the u16 goes up and the magic stays put, and a reader can say it accepts revisions 1 and 2 rather than two different file types.

The two producers, side by side

The layout is identical — same header, same record, same offsets — and most of the fields mean the same thing on both sides too. Start from what does not differ, because a difference table invites the reading that everything in it is different:

	both producers
packed	the position. Not an encoding of it, not a corpus row, not a game state — a position
ply	the move number that position was reached at
value_z	one game's outcome: -1, 0 or +1, from the side to move
pov	the side to move
mv	a move16

Where those came from differs, and it does not reach the field. A position read out of a SFEN and a position a game walked into are the same 96 bytes; an outcome copied off a corpus row and one the network just earned are the same three values. Everything else is what actually changes:

	teacher (MultiPV)	self-play (MCTS)
policy_source	1	0
cp	the MultiPV score — the policy	NO_CP
visits	NO_VISITS	the search's visit count — the policy
n_cand	as many as the engine was asked to report — a setting, not a bound	every visited move; measured mean 37.03 / max 351
value_q	the root score squashed through eval_coef	the search's root value, already in [-1, 1]
eval_coef	the coefficient that squashing used	0.0, not applicable
mate_scale	the centipawn value the corpus's mates are folded to	0, not applicable
fv_scale	what the generating engine's evaluation was read at	0, not applicable
written	one record per corpus row, streaming	one game at a time, buffered until it ends

The axis is the search, not the network. A MultiPV search ranks a fixed number of moves and scores each in centipawns; an MCTS spends playouts and counts them. That is what the two columns divide on, and it is the whole content of policy_source. Reading them as "the NNUE one" and "the DL one" is wrong in the direction that matters: teacher shards feed both trainers, so nothing about a row says which network will eat it.

Several of these need more than a table cell.

n_cand is not comparable across the two columns. Self-play's count is produced by the search — every move it reached, however many that turns out to be. The teacher's is dictated: the engine reports the width it was told to report, so the number describes whoever ran the generator, not the data. The published corpus was made at MultiPV 5 and its rows have at most five candidates, and that is a fact about that run and nothing else. multipv goes into the shard metadata for exactly this reason — the count on the row cannot be read as a property of the position without it.

value_z is the same field with a different provenance. In both streams it is one game's outcome — -1, 0 or +1 from the side to move, three states and not two, because draws are real: 0.32% of the published corpus, and self-play writes 0 whenever a game ends drawn. A record is written per position but the value belongs to the game, so every record of one game carries the same outcome seen from its own side to move.

What differs is only what the outcome is evidence of. Teacher data reports a game someone else played, recorded at generation time; self-play reports the game the network being trained just finished.

A stream never holds a score rate. value_z stays categorical for exactly as long as the data is raw, and becomes a real number in the fold, where the outcomes of every game that reached a position are averaged. That is the same rule as counts-before-probabilities, applied to the value head: the thing that adds is stored, and the division happens once, at the conversion.

value_q is derived on one side and native on the other. A search that returns win rates hands value_q over directly. A search that returns centipawns has to be squashed, and the coefficient that squashing used is a convention that changes the target without changing a byte — which is why it is in the header on the side that needs it and absent on the side that does not.

A game is the unit on one side and a row is the unit on the other. Self-play cannot write value_z until the game ends, so records are held and flushed together; a teacher converter already knows the result on the row in front of it and never buffers. This does not show up in the format, and it is the whole reason the self-play writer has a shape the teacher writer does not.

Validation

stream::validate runs at the writer and again on read, so a stream corrupted after it was written fails at load rather than during training. It refuses a record with no candidates, a position that is not packed_len wide, a value_z that is not exactly -1, 0 or 1, and a non-finite or out-of-range value_q.

Both optional fields are checked the same way, and neither check needs the header: within one record a field is present on every candidate or on none. A record with some counts and some absences is one where a move would be dropped from the distribution or counted as never visited, and there is no way to tell which was meant. When visits is present its total must be positive, since a distribution that normalises to nothing divides by zero at conversion.

Only one rule reads policy_source, and it is a cross-check rather than a different check: the half named by the header has to be the half that is present. A score-sourced record with no cp anywhere is a record whose policy cannot be derived at all.

The checks live at the writer because every failure they catch is silent downstream. A record with no candidates has every logit masked, contributes zero loss, and looks exactly like a sample that teaches something. A value outside [-1, 1] is a target the tanh head can never reach.

mv must also look like a move16: bit 15 clear, to a square, the origin field no greater than 87, and no drop flagged as promoting. Two of those fields have ranges narrower than their width, so a record read at the wrong offset usually lands outside them — a cheap check on a stream being read against the wrong packed_len.

The one thing not checked is that mv appears at most once in a record. It matters more after folding than before, so it is enforced there, where the lists are being merged anyway.

Teacher generation

manaka-teacher reads the corpus and writes MPK1 with policy_source = 1. The corpus holds what USI could observe — SFEN positions, a root score at a depth, MultiPV candidate moves and their scores levelled to a second depth, the move actually played, and the game's result — and none of the fields a record needs. Each one is derived:

• value_z — the corpus's result flipped into the side-to-move view. It is stated from Black there and from the side to move here, and mixing the two inverts the target on half the rows while the loss curve still looks like training. side has to be read rather than inferred from ply % 2: the starting positions are a mix of sides and more of them begin with White.
• value_q — the root score squashed into [-1, 1] against eval_coef.
• cp — the candidate's own score, carried through unshifted. The corpus's root score is not max(cand_cp); the two are measured at different depths, and only the candidate scores go into a record.
• mv — each candidate move parsed into a move16.
• visits — NO_VISITS on every candidate. The corpus has no visit counts to carry, and says so rather than inventing a number.
• ply — carried through unchanged, so it stays 0-based and stays non-consecutive. The published corpus had repeated boards deleted — not folded, in the sense this document uses the word — which takes rows out of the middle of a game; games are still contiguous blocks in ply order.

No softmax happens here. The scores go in as scores, and the temperature is applied at the parquet conversion, so changing it does not mean regenerating the stream. Where the position is not a mate, that is a softmax at 100 cp; where it is, the distribution becomes a one-hot on the highest-cp candidate, because levelling the depths can leave the leader tens of thousands of cp above candidates still sitting at thousands, and a softmax over that is not a distribution.

The one-hot is on the leader, not on the move the game played. The generator draws its move at random from the candidates, so in exactly the positions where the search is most certain, the played move is the least informative label in the row. It is also not in the stream: a record carries no played move, and the leader is derived from fields it does carry, which keeps the conversion a function of the stream alone — and of the folded stream, since the fold averages cp per move and the one-hot falls out of the result, where a played move per record would have had to fold to something.

A row whose side disagrees with its SFEN, or whose played move is not among its candidates, is refused rather than encoded.

A record is 108 + 8·n_cand bytes. The published corpus averages 4.37 candidates a row, which puts it at 143 — a figure that moves with whatever MultiPV the generator was run at.

Self-play

manaka-selfplay and dl-manaka-selfplay write MPK1 with policy_source = 0: the search produces visit counts over moves, not a centipawn opinion per move, so every candidate carries NO_CP. The counts go in as counts. value_q is the root value the search settled on, taken as it comes — it is already a win rate in [-1, 1], so no coefficient is applied and none is declared: the header carries eval_coef = 0.0, mate_scale = 0 and fv_scale = 0.

value_z arrives only when the game ends, so records are held and the whole game is written together. That buffering is the one structural difference between this writer and the teacher's, which knows the result on the row in front of it.

packed is the position, not the network's input planes — including for DLManaka, whose feature space (KP256, HalfKP256) is expected to change between generations. Baking features into a shard means rebaking every shard when the encoding moves.

At the measured 37.03 candidates per row, a self-play record is 404 bytes.

Parquet

manaka-pack reads MPK1, folds duplicate positions, and writes <prefix>-NNNNN.parquet, zstd at its lowest level, one row group every 8192 rows. Every shard is a complete file with its own schema and footer, so a converter that dies partway leaves behind shards that still open. Parquet is used because sparse policy data compresses well and stays easy to inspect with standard tools.

column	type	meaning
packed	fixed binary (96)	the position; side to move in the last byte
ply	uint16	move number; the lowest one this position was reached at
value_z	float32	outcome target, side-to-move view. Not restricted to −1/0/1
value_q	float32	search-value target, same view
weight	uint32	how many searches were folded into the row
candidates	list<struct<mv uint16, prob float32, cp int16>>	the policy

cp is null for self-play rows. There is no legality mask and no move list: a reader that needs either regenerates it from packed.

Folding, and where the division happens

Records with identical packed are one row. Merging them is addition, which is the whole reason the stream stores counts:

• candidates — matched by mv, and the merged list is the union. Two records listing A, B, C and A, B, D fold to A, B, C, D. Whichever half carries the substance is merged: visits summed, or cp averaged over the records that listed the move. That mean's denominator is the size of the group, not a stored count. The fold has concatenated the records and grouped them by mv already — it has to, to average cp at all — and counting a group is the same reduction as summing a column over it, one column lighter. So the policy gets denser as more searches of the same position are folded in, which is the thing probabilities cannot do — two separately normalised distributions added together have already lost the weights that would say how to add them.
• value_z — the mean over the folded records, each counting once. A position that appeared in a game that was won and a game that was lost lands between −1 and 1, and that number is the position's measured score rate. This is why the parquet column is not restricted to −1/0/1 the way the stream field is: one record is one game, one row is however many games reached the position.
• value_q — the mean over the same records.
• ply — the smallest: the earliest move number the position was observed at. Nothing trains on ply — the NNUE loader never reads it and the DL trainer carries it into the batch untouched — so the fold keeps it honest rather than exact, and a minimum stays a uint16 where a mean would be a move number no game contains.
• weight — the number of records folded, which is the number of searches that reached this position. Carried so a trainer can tell a position 400 games agreed on from one a single game wandered through, and down-weight the second.

The key is the board, and ply is deliberately not part of it. Two searches of the same position are two observations of the same thing, whether the game reached it in 40 moves or in 48 — a search is handed a board, not a route. Nor is this a rare tie to break: over one shard of the pre-fold corpus (14,287,893 rows), 25.65% of the 173,622 duplicate-board groups span more than one ply, with a median spread of 8 plies and a maximum of 92 — games converge on the same board along paths of different lengths. Keying on (packed, ply) would leave a quarter of the groups partly unfolded to protect a column nothing reads. What the board-only key blurs, it blurs slightly: a mate under mate_scale = 30000 carries its ply inside the score, so averaging across plies moves a 30000-scale value by a few centipawns, and near a maximum-move draw the same board does not have the same outcome distribution. Both are worth naming; neither is worth a key.

A move missing from a record does not mean the same thing on both sides, and the union is only exact on one of them. Self-play enumerates every legal move and stores the ones it visited, so a move absent from a record got zero playouts. That is a measurement, and summing it in is arithmetic with nothing lost. MultiPV is a cut: a move absent from a teacher record was ranked below the width, which is an upper bound on its score and not a zero. Folding A, B, C with A, B, D therefore puts C and D into the same softmax as A and B on the strength of one search each, while the opinion the other search held about them — worse than its own worst listed move — is discarded. Moves seen once come out slightly over-weighted relative to what the two searches jointly believed.

Nothing in the format fixes this, because the information was never in the corpus to begin with. What bounds it is how often teacher records meet at all: 6.47% of the published corpus's rows share a board with another, so the bias rides on one row in fifteen and on none of the rest. That is a property of one corpus and not of the format, and it grows with every generation that searches positions someone has already searched.

The old pipeline made the number look smaller by deleting the duplicates before anything could fold them, which is not a fix — it is the same censoring applied harder, with every search but the first thrown away instead of merged badly.

weight counts searches and not visits, which is a correction rather than a choice. Summing visits looks like the better evidence measure until a forced move goes through it: a position with one legal reply produces one candidate, so its total is 1, while an ordinary position from the same single search totals the whole budget or the whole MultiPV width. The row that is more certain comes out weighing less. Counting records is the only definition that means the same thing for both producers and is not distorted by how many moves the search happened to list.

It also fixes the width. The most-repeated position in any self-play run is the initial position, which appears once per game — measured at exactly 479 occurrences in a 55,504-record, 479-game stream — so a row's weight is bounded by the number of games in the fold, and uint32 is not a number of games anyone reaches. The u64 accumulator is still needed while candidate counts are being summed; it just never becomes a column, because prob is what survives the division.

What weight does not capture is how hard each search worked. A generation run at 3200 simulations counts the same as one run at 800. That is a mixing hazard of the same family as pov and mate_scale — see what must not be folded together.

prob is computed last, once, from the merged candidate list — normalised visits when policy_source is 0, a softmax over cp at the configured temperature when it is 1. Nothing upstream of this point holds a probability, so a temperature change or a different fold does not need the stream regenerated.

The division is one-way, and the shard is the far side of it. Two shards are not folded into a third — prob has already lost the counts, and prob × weight recovers them only approximately. Refolding means going back to the streams, which is why they are kept until the shards they produced are known good rather than deleted at conversion.

The fold is the only place duplicate boards are handled, and the pre-pass that used to remove them goes away. Dropping a repeated board at corpus level keeps the first row and discards the rest, which throws out exactly the candidates and outcomes the fold exists to add up — the second search's opinion of a position is evidence, not noise, and 6.47% of the corpus's rows were being resolved by picking one arbitrarily. Removing rows before the fold cannot be undone by the fold, so the order is fixed: fold once, here, with all the records present.

What must not be folded together

Addition is only meaningful between records that mean the same thing by the number, and two of the header fields decide whether they do.

• Different policy_source. A visit count and a centipawn score are not the same quantity and there is no exchange rate between them. A teacher record and a self-play record for the same position stay two rows.
• Different pov, mate_scale, fv_scale or eval_coef. The values are on different scales, and averaging across them produces a number that is on neither.

One that is permitted and still wants thinking about: folding across generations. Two self-play runs at different strengths produce visit counts in the same units, so the sum is well defined — but the weaker network's visits get equal say per playout, and the simulation budget is nowhere in the file. weight will not show it either, since a 3200-playout search and an 800-playout one each count as one search. Nothing in the format prevents this and nothing detects it; if runs at different budgets are mixed, the budgets belong in the shard's metadata, written by the hand that mixed them.

The position stays 96 bytes

The mailbox is large raw and small in a shard, because empty squares are 0xFF at consistent column positions and zstd eats that. Measured on the published teacher shards, packed is 96.00 bytes a row raw and 5.63 after zstd. The corpus's sfen column, over 13,383,862 rows, is 80.3 raw and 13.28 after zstd — more than twice as much for the same information.

Fixed width also reads faster. A whole column comes out in one np.frombuffer(packed_array.buffers()[1], ...); a string column is parsed per row, every epoch.

list<struct>, not three parallel lists

Three parallel lists are three columns that have to be the same length, which is an agreement between writer and reader rather than something the file states. A list<struct> says it in the type.

It costs nothing. On 100,000 synthetic rows averaging 37 candidates, three parallel lists are 25,685,483 bytes and the list<struct> is 25,685,403 — 80 bytes apart. Parquet shreds a struct into one leaf column per field, so the physical layout is the same and column pruning still works: reading candidates.list.element.prob alone does not touch mv or cp.

Key-value metadata

The stream's conventions are written into the parquet footer: pov, mate_scale, mate_slack, fv_scale, policy_source, eval_coef, the softmax temperature and what packed_len means. This is the same argument as the stream header — a shard that does not name its point of view is a shard whose targets can be inverted silently — and it is the gap that matters most in practice. The shards published before 2026-08-28 had kv metadata: None; both Knowledge_distilled_dataset_by_NAGISA_V3 and _V4 now carry the full set.

mate_slack reaches the footer as the effective slack rather than the header field verbatim: the writer subtracts the threshold from the scale, so a header that declared nothing lands in the footer as the 320 the fold actually used rather than as the zero it carried. A stream with no mate scale has no threshold either and says 0 for both. V3's published shards predate the key and carry no mate_slack at all — its absence means that same 320, and V4's 100 is the first value that had to be stated to be right.

The temperature belongs here specifically because it is the one convention the shard cannot be re-derived without: prob has already been divided by the time it lands in a column, and cp alone does not say what curve produced it.

Every pair is copied from the stream header or from the converter's own constants. There is no flag for adding more, and that is the point: a footer nobody can hand-edit is a footer that cannot disagree with the rows beside it. The conventions reach the header at manaka-teacher time, stated by the hand that knows how the corpus was generated, and travel from there into every shard without being retyped. What the header has no field for — which engine and evaluation produced the scores, which corpus revision they were distilled from — is the dataset card's job, written once where a reader can see it rather than copied into every shard as a string only the operator's memory could check.

multipv is not written either, in the footer or the header. It is the length of each row's candidate list, so a reader counts it rather than trusting a declaration.

Rows leave shuffled

The converter shuffles rows before writing. A folded row is an independent position — nothing ties it to its neighbours once games and transpositions have merged — so the one place that holds every row anyway is the right place to scramble them, and the trainer is not asked to know that a shuffle is its job or that one is needed at all. The raw streams stay in whatever order the producer appended; order is destroyed at the same step that destroys duplicates.

The permutation is a Fisher–Yates from a fixed seed, so conversion stays deterministic: the same streams still produce byte-identical shards.

--buckets does not weaken that to a shuffle within blocks. An earlier build wrote each content-hash bucket out in turn and permuted inside it, which leaves the partition itself standing in the row order — a function of the position bytes, and therefore of the position: measured on a shard of that build, the spread of mean ply across 24 blocks was 2.97× what independent sampling gives. So pass 2 now scatters each folded row back across a second set of spools chosen uniformly at random, and pass 3 loads and permutes one of those on its own before writing it. Where a row lands depends on nothing about the row, which is what makes the result one permutation of the corpus rather than shuffled blocks laid end to end. On the V4 shards the same 24-block ratio reads 0.978, against a noise floor of ±0.15.

Each spool gets SEED + spool rather than SEED, or every one of them would replay the same permutation prefix.

Row groups

8192 rows to a group, set explicitly with set_max_row_group_size. A row group is the unit a reader pulls in and decompresses, so it is the reader's working set; left unset, ArrowWriter's default of 1,048,576 applies and a shard ends up as a single group.

Invariants

• Raw data stores what adds. Counts, not rates; one game's outcome, not a win rate. Normalisation is a lossy step and belongs at the last possible moment, which is the parquet conversion.
• Encoding definitions live on the Rust side. Python keeps sizes, not a second move or position specification, and the stream header carries both so a mismatch fails before training rather than looking like a weak network.
• A record stores the position, never an encoding of it. That is what makes a feature-space change cost nothing on disk.
• Both producers write one format, down to the byte offsets. A field one of them has nothing to say in is written as absent, never as a plausible value and never dropped: a record whose shape depends on a header field cannot be parsed by a reader that has not already been taught to branch, and a fabricated value is a wrong answer that raises no error.
• An absent marker is a value the data cannot take. Which value that is depends on the field, so NO_CP is i16::MIN — zero is a real score — while NO_VISITS, mate_scale, fv_scale and eval_coef are absent at zero, which none of them can otherwise be.
• Validation happens at the writer and again at the reader, because everything it catches is silent otherwise.
• A format states its own conventions. Point of view, mate scale, FV_SCALE, policy source, candidate count, softmax temperature and win-rate coefficient belong in the file, not in the flag that wrote it or the reader that assumes it. Provenance is a different question and is answered on the shard, which is what outlives the stream.
• The magic names the format and the version numbers it. A revision that changes the magic turns one format into two, and CZR1 is what that costs.
• Validation data is a directory, not a predicate. There is no holdout field and no split rule. Every position roots at the initial position and transposes into the same middlegames, so a hash of the opening does not keep a position out of both sides — it only looks like it does. Pass a separate set of shards to validate against, which is how NNUE training already works.