DLManaka improvement plan — gen11 checkpoint

Historical plan drafted 2026-08-20 and last reconciled with repository records on 2026-08-26. It is a checkpoint of the plan at gen11, not a claim that every item is still pending. Later work should update or replace this page rather than append another full narrative.

State at gen11

External RATED measurements against the fixed YaneuraOu reference showed:

generation	Elo	note
gen4	−522	first complete automated-loop run
gen7	−481.8	earlier peak
gen9	−638.0	collapse while weak children were promoted
gen11	−450.0	window 12 + gate −20; best recorded here

Two changes were retained because they addressed observed failures:

• replay window widened from 5 to 12 generations;
• a −20 Elo parent gate stopped unconditional promotion of clearly worse children.

Internal versus alone is not a strength claim; external anchor Elo and validation must agree.

Search is no longer the first bottleneck

The decisive search measurements were:

sims	fitted Elo	gain from previous doubling
32	−1398	—
64	−1132	+266
128	−926	+206
256	−738	+188
512	−612	+126
1024	—	+16, rejected

Earlier, simply changing the same weights from 32 to 256 simulations gave +557 Elo. That established that the original loop was teaching from a search that was too shallow. By gen11, however, the 512→1024 doubling had essentially run out of value, so brute-force simulation increases were no longer the primary plan.

What remains weak

Recorded validation diagnostics:

• policy top1 about 49.9%, top5 about 88.8%; the final searched move lies outside policy top5 about 14.2% of the time;
• value-sign agreement about 0.75.

The remaining work therefore focuses on evaluator/teacher quality rather than just search depth.

Priority axes

1. Keep the current network size unless evidence changes

The 15b256 network has not shown a clear capacity ceiling. Making it larger also slows search. Expansion should be tested only after better teacher/value targets stop helping, and only as a controlled side run.

2. Value target

value_weight=2.0 was tested and rejected; it delayed the value-head decay but worsened final validation metrics. The next cheap hypothesis is the target blend: test a higher value_q_ratio in a side run, not inside the generation loop.

See value-head decay.

3. Teacher search

Deeper search produced better distillation targets, but the search-doubling curve says the useful ceiling is around 512 simulations, not 1024. A 512-sim teacher is therefore the relevant next controlled comparison.

Virtual loss can make teacher generation much faster, but teacher quality under that setting must be checked before adopting it.

4. Data diversity

Candidates with plausible value:

• mine positions where the network evaluated highly but the game was lost;
• mix a fixed external engine into a fraction of self-play games;
• keep the replay window and gate unless a controlled measurement shows harm.

Simply generating more of the same data was already a weak lever.

5. Search engineering

Use search work to make a good teacher cheaper, not to push playing simulations indefinitely. Batch width must remain tied to simulation count because excessive width can reduce distinct nodes.

6. Policy quality

The policy still misses a non-trivial fraction of the moves found by search. Policy changes should be judged by validation and equal-time games; a better prior that slows each node may still lose overall.

Separate architecture line

The old plan discussed a CPU/incremental “zero-knowledge” alternative. That is no longer hypothetical: it became the separate Manaka project under crates/manaka-* and docs/manaka/. DLManaka and Manaka should now be evaluated as separate engine lines rather than as mutually exclusive proposals inside this plan.

Next controlled measurements

1. value_q_ratio side run.
2. 512-simulation teacher comparison, including the virtual-loss quality check.
3. Per-generation external measurements that expose both low-search and normal-search behavior.
4. Only after those: network-size or policy-architecture experiments.

Each experiment changes one main variable. A result counts as an improvement only when game-based external measurement and held-out metrics are consistent with the claim.