Value-head decay
Historical training diagnosis for the 15b256 supervised run. The value head improved early, then its training accuracy declined while policy metrics remained healthy.
Observation
Over 100k steps, train/value_sign peaked around 20k–30k and then fell toward the high-0.7 range. This did not look like ordinary overfitting:
- the run had consumed only about 35% of the full corpus;
- policy train/validation curves stayed close;
- the training value metric itself degraded.
The useful conclusion is simply that the value objective and shared representation stopped moving in the same useful direction partway through the run.
value_weight = 2.0 experiment
A second run doubled the value loss weight. Unfortunately it also changed BatchNorm momentum from 0.1 to 0.01, so this was not a clean one-variable experiment.
Final validation values:
| metric | baseline | value_weight=2 run |
|---|---|---|
| policy loss | 1.854 | 1.898 |
| policy top1 | 0.4943 | 0.4797 |
| value loss | 0.1743 | 0.1805 |
| value sign | 0.7615 | 0.7454 |
The weighted run started with better train-side value accuracy but decayed with a similar shape and ended worse on every validation metric.
Verdict
Do not adopt value_weight = 2.0 from this experiment. The joint change did not improve the endpoint and hurt policy validation.
Because BatchNorm momentum changed at the same time, the experiment does not cleanly prove that value weight alone caused every difference. The stronger statement in the old document has been narrowed accordingly.
Leading hypothesis: label variance
Teacher data stores:
value_z— final game outcome;value_q— teacher search value at the position.
Measured in the teacher data, almost all value_z labels are ±1, while corr(value_z, value_q) is only about 0.65. A balanced position can therefore receive an extreme outcome label because of events much later in the game.
As the network learns that many early positions are near even, minimizing squared error against noisy ±1 outcomes may become misaligned with producing a useful evaluation.
The cheapest test is therefore the target blend, not a larger loss weight: increase value_q_ratio so the target leans more toward the lower-variance search value.
This test should be a side run with no simultaneous architecture or normalization change.
Other hypotheses
- Value-head capacity: possible, but it does not naturally explain “improve first, then decay”.
- Learning-rate interaction: possible because the decline begins in a similar region of the schedule; requires a controlled run.
- Metric mismatch: value-sign on teacher targets may not map perfectly to game strength, so external games remain the final check.
Current role of this record
This page is evidence for one thing: increasing the scalar value-loss weight was not a successful fix, and the next low-cost experiment should alter the z/q target blend cleanly.
Search implementation and later self-play work continued independently; see self-play roadmap and gen11 improvement plan.