Twelve Manaka shapes on one validation set

Measurement record, 2026-08-26. Twelve one-epoch Manaka weight files were evaluated on the same 16,384-position holdout to compare policy cross-entropy (CE) and policy top1.

The question is whether the two metrics rank architectures the same way. They do not.

Conditions

manaka-train has no validate-only command, so each weight file received one extra training step before validation. That is 1 step against a roughly 293,000-step epoch and is treated as negligible but recorded here explicitly.

All runs used the same holdout and default --q-ratio 0.5.

Results

Sorted by lower CE. gap is top1 rank minus CE rank.

shape	val CE	CE rank	top1	top1 rank	gap	value_sign
f4	2.6336	1	.3325	3	+2	.6885
pairtie	2.6550	2	.3267	6	+4	.6909
ks	2.6911	3	.3307	4	+1	.6973
m16	2.6924	4	.3262	7	+3	.6802
f2	2.7213	5	.3369	1	−4	.6936
order2	2.7272	6	.3292	5	−1	.6973
pair0	2.7392	7	.3219	9	+2	.6980
lse4ks	2.7407	8	.3159	11	+3	.6789
h2256	2.7720	9	.3105	12	+3	.6637
pair64full	2.7871	10	.3339	2	−8	.6755
base	2.8009	11	.3199	10	−1	.6902
lse4	2.8109	12	.3232	8	−4	.6915

Spearman rank correlation between CE and top1: +0.476.

What this establishes

• CE and top1 are not interchangeable architecture metrics.
• pair64full is the clearest disagreement: 2nd on top1, 10th on CE.
• pair0 beats base on both metrics in these weight files, but this does not contradict the older §10 Elo result because those games used different weights.
• h2256 is weak on both metrics, consistent in direction with its older Elo result.

For the eight shapes that also have old §10 Elo rankings, top1 correlates more strongly with that old ranking than CE (+0.738 versus +0.405). Because the weights differ, this is reference only, not validation of either metric.

Next measurement

Run Simulations = 0 round-robin games using these exact twelve weight files. That puts Elo, CE and top1 on identical weights and removes the main confound in the historical comparison.