Manaka throughput work

Historical optimization record from 2026-08-25. For current absolute speed, use the benchmark / common benchmark. This page answers only: what made training faster, what did not, and what remains.

Result

On RTX 4070 Ti, batch 1024, the training step fell from roughly 44–45 ms to 13.3 ms (about 3.3×) during this work.

The winning idea was not a new scheduler or graph API. It was avoiding huge temporary tensors that were immediately reduced and discarded.

What worked

GatherDot

The policy head previously materialized a padded tensor of gathered policy rows with shape roughly [batch, max_legal_moves, head_width]. In the measured run that intermediate was about 732 MB.

GatherDot fuses row lookup and dot product so the intermediate is never written. Training moved from about 45.1 → 18.5 ms/step.

PowerSums

The symmetric-polynomial path previously gathered embeddings into a large [batch, 2, 40, dim] tensor for each order before reducing over tokens.

PowerSums performs the token sweep and powers inside one kernel and emits only the reduced result. Training moved from about 18.5 → 13.3 ms/step.

The two fusions also reduced VRAM substantially and removed the original memory-controller saturation.

What did not help

Delaying loss readback

Reading the loss to the CPU looked like a synchronization point, but restricting it to logging steps produced no measurable speedup. The actual bottleneck was memory traffic elsewhere.

Reducing kernel-launch count / CUDA Graph

A launch-count study found hundreds of launches per step, but reducing the count by roughly a quarter did not improve wall time. GPU execution remained the critical path and launch issue cost was hidden asynchronously.

CUDA Graph was therefore not pursued. Revisit only if profiling shows the CPU launch path has become the bottleneck.

More producer threads

The default four batch producers were faster than 8 or 12 on the measured 20-core host. More CPU workers are not automatically better.

Generation-side finding

Manaka inference did not contain an equivalent order-of-magnitude temporary-tensor waste. Evaluation is mostly real dense arithmetic plus search work, so the optimization ceiling there is much smaller than it was in training.

Current evaluator/search breakdowns belong in the benchmark.

Remaining candidates

• Lower precision for the pair-table optimizer state may still reduce bandwidth, but the original “memory controller is saturated” premise no longer holds after fusion.
• The pair table remains a significant fixed training cost because AdamW touches tens of millions of parameters every step.
• Any batch-size increase must be validated for strength as well as throughput; fewer parameter updates can change training behavior.

Measurement rules learned here

1. Profile before choosing an optimization target.
2. Check utilization.memory and power, not only GPU busy.
3. Use median-of-N and reject runs with co-tenant GPU processes.
4. Measure a copied benchmark binary if concurrent sessions may rebuild the workspace.
5. Never count a failed fast exit as a fast run.
6. Numerical-equivalence tests remain mandatory after kernel fusion.