Teacher data

DLManaka and Manaka use one shared teacher corpus. manaka-teacher writes packed teacher shards; each trainer builds its own input representation when loading them. There is no separate DLManaka teacher dataset.

The source is the NAGISA_V3 distillation corpus published as qleap/Knowledge_distilled_dataset_by_NAGISA_V3.

Build path

uv run --script tideborn/tools/fetch_teacher.py --shards 0 1 2 3
cargo build --release -p manaka-teacher -p tideborn-encode
./build/target/release/manaka-teacher data/deduped/*.parquet --strict \
  --out data/mteacher-deduped
uv run python -m tideborn.tools.check_shards data/mteacher-deduped --limit 50000

tideborn-encode is loaded by path on the Python side and builds DLManaka's planes from the packed position at load time.

Why positions stay packed

Pre-encoding the entire corpus separately for both engines would duplicate expensive legal-move/encoding work and create two large artifacts that must remain synchronized. Instead, shards store a compact position and sparse targets.

Measured on the DLManaka path, load-time unpacking was small compared with one GPU training step, so keeping a shared packed corpus did not starve the GPU.

Corpus curation

The source was shuffled at the game level and then deduplicated:

uv run --script tideborn/tools/shuffle_corpus.py --input data/raw --out data/shuffled
uv run --script tideborn/tools/dedup_corpus.py   --input data/shuffled --out data/deduped
uv run --script tideborn/tools/upload_corpus.py --input data/deduped --yes

Recorded size after deduplication: 281,061,093 rows, down from 300,045,733 (−6.327%). Holdout games are not folded into training duplicates.

check_corpus.py verifies that game blocks remain intact and that the shuffle actually mixed games.

Teacher-shard row

field	meaning
packed	compact position, not either engine's final input tensor
label_hash	fingerprint of regenerated legal-move ordering
prob_slots / probs	sparse teacher policy
value_z	final game outcome
value_q	teacher root/search value
aux_value	optional auxiliary target; NaN when absent
ply / game_id	weighting/split metadata

Recorded size: about 43.8 bytes/row, 12.3 GB for the 281M-row corpus.

Train / validation split

The split is based on the starting position, not random rows. Positions one move apart from the same opening therefore cannot land on opposite sides merely because their row ids differ.

Both trainers use the same holdout rule, so their validation sets are aligned at the source-data level.

Integrity checks

Three checks prevent silent label/encoding drift:

1. manaka-teacher --strict verifies candidate moves are legal on the reconstructed board.
2. label_hash verifies regenerated legal-move order for every row.
3. check_shards.py validates position structure and sparse-policy invariants after conversion.

The encoding source of truth remains Rust (crates/dl-manaka-core/src/encoding.rs for DLManaka); Python does not maintain a second independent plane/label definition.

For the byte-level self-play and parquet schemas, see training data formats.