Measuring playing strength — openbench

Speed and validation loss do not determine playing strength. openbench runs game-based measurements and SPRTs (sequential probability ratio tests) across local and cluster workers. The infrastructure itself lives in the separate rshogi/openbench repository.

Engine ↔ worker contract

usi-arena reports one machine-readable line to stderr:

[arena] metric games=4 wins=3 losses=1 draws=0 plies_mean=103.8 arena_score=0.7500 …

The worker parses wins, losses, and draws; a scalar score alone is insufficient for trinomial statistics. Engine commands, deployment paths and cluster placement are configured under tideborn/tools/openbench/.

When a binary changes, update the deployment path named by remote.toml as part of the same operation. A stale deployed binary usually fails every game immediately.

Anchor ladder

Internal head-to-head Elo can drift with the whole project. The external ruler is a fixed YaneuraOu evaluator registered at several search node counts, for example YaneuraOu KP256 @10k.

Adjacent rungs must be close enough for useful statistics. climb calibrate inserts missing rungs at the geometric mean of neighboring node counts:

climb calibrate remote "YaneuraOu KP256 @"
climb calibrate remote "YaneuraOu KP256 @" --dry-run
climb calibrate remote "YaneuraOu KP256 @" --climb 100 --floor 100000

--climb ELO extends the ladder upward until doubling the reference search no longer clears the requested Elo gain. Keep a finite --ceiling for unattended runs.

Rating a challenger

A challenger climbs from weak rungs upward, one SPRT per rung:

climb start local  "DLManaka v0 @32" --min-nodes 1000 --games-per-job 32
climb start remote "YaneuraOu Material Lv1 @1m" --min-nodes 1000

• accepted — continue upward.
• rejected — stop; the challenger did not clear the rung.
• inconclusive — also stop for this purpose.

Starting low is deliberate: a 500–0 loss against a strong rung only says “somewhere below”. --min-nodes skips rungs that are already known to be irrelevant.

flag	default	meaning
--min-nodes	0	first rung to try
--elo0 / --elo1	0 / 10	SPRT hypotheses
--max-games	1024	cap per rung
--games-per-job	16	pairs per worker job

Watching and stopping

climb watch
climb list
climb stop --with-reap
climb reap

The dashboard shows node capacity, live tests and local climbers. ▒ means games are in flight but not yet reported; it is not a stalled test.

Stopping a test stops new job distribution, not containers already playing. Use --with-reap or follow with climb reap. To stop everything explicitly:

uv run python tideborn/tools/openbench/stop.py \
  --config tideborn/tools/openbench/config.toml

Job sizing

Current intended ratios:

• server → worker: 128 pairs/job, at most 256 pairs in flight;
• worker → node: blocks of 32 pairs, at most 128 pairs/node.

One pair uses two engine threads. Oversizing only queues work locally and delays reporting because a job reports after all of its games finish.

Operational constraints

• Do not run local- and cluster-backend climbers concurrently. The server job queue is shared and placement is currently take-then-discard, so workers can steal and abandon each other's jobs.
• Every registered engine name must resolve in [engines] in config.toml, either exactly or after stripping the @nodes suffix.
• Re-fetch /openapi.json before writing manual API calls; the /api prefix is required.
• Prefer an external anchor plus validation metrics over internal head-to-head alone. A single narrow Elo difference is not evidence when its confidence interval overlaps zero.