Training

How to run tideborn.trainer, the trainer for DLManaka (the deep-network side that runs on GPU). What the teacher data contains and how it is made is covered in Teacher data.

Running it

uv sync --extra cpu     # on a GPU machine, --extra cu129
uv run python -m tideborn.trainer summary --config configs/smoke.toml
uv run python -m tideborn.trainer train   --config configs/smoke.toml

configs/smoke.toml is a plumbing check: it runs a small 2-block × 32ch network for 20 steps on synthetic data (dummy data made from random numbers), needing neither a GPU nor teacher data. CI goes through this same path. A step here is one parameter update over one batch (a bundle of positions processed together).

Production runs take hours, so they are launched in a form not tied to a terminal. tideborn/tools/train_run takes care of detaching, logs, the pid, and anomaly detection.

python -m tideborn.tools.train_run start  --config configs/rtx4070ti.toml
python -m tideborn.tools.train_run status --out-dir runs/ckpt/rtx4070ti
python -m tideborn.tools.train_run stop   --out-dir runs/ckpt/rtx4070ti

status reports progress plus a list of anything that looks wrong, and exits non-zero if there is any. It watches for NaN or divergence in the loss, the process disappearing, 15 minutes with no output, and OOM / CUDA error / traceback lines in the log. It never touches the training run itself. Whether to stop or keep going is a human decision.

watch streams one event per line, so its output can be fed straight into a monitor.

torch sits behind the mutually exclusive extras cpu / cu129. A bare uv sync installs no torch, so one of the two must always be specified. This prevents the accident of a CPU machine grabbing CUDA wheels; the development environment's setup script picks which extra to install automatically based on whether nvidia-smi is available.

Configuration

The list of configs is in Training, along with the axes shared by both engine families (--type / --network / --mode) and the mechanism by which settings come from tideborn.toml. This page limits itself to the DLManaka trainer proper.

Settings are frozen dataclasses parsed from TOML and can be overridden with --set. Values are interpreted as TOML, so 1e-3 and true can be written as-is.

uv run python -m tideborn.trainer train --config configs/rtx4070ti.toml \
  --set runtime.max_steps=5000 --set optim.lr=1e-3

To confirm what --set resolved to, python -m tideborn.trainer config prints the fully resolved settings as JSON. The settings are also stored whole inside every checkpoint (a file saving the weights and training state mid-run), so "how were these weights made" can always be traced from the checkpoint alone.

Distributed training

Training on multiple GPUs uses PyTorch's DDP (DistributedDataParallel: the same model on each GPU, with gradients kept in sync). DDP is assembled from torchrun's environment variables. A single-process launch creates no process group, so the same code runs on 1 GPU and on 8.

torchrun --standalone --nproc-per-node 8 -m tideborn.trainer train --config configs/a100x8.toml

When using optim.accumulation_steps (gradient accumulation: collecting several small batches and updating only once), every micro batch except the last runs inside no_sync() to skip the all-reduce. Without this, each step performs as many synchronizations as there are accumulation rounds.

Resuming

Checkpoints are written atomically to runtime.out_dir as step-XXXXXXXXX.pt. train resumes from the newest checkpoint in out_dir by default, so after a crash, re-running the same command is enough. --resume <path> names one explicitly; --fresh ignores them and starts from step 0.