An environment does not train the model by itself. Verifiers defines the task, runs the episode and supplies a reward. vLLM generates tokens. The orchestrator turns completed episodes into training samples. PyTorch and FSDP2 execute the update.

Start the serving process

The launcher starts inference and the environment server. In the shared-GPU profile, vLLM needs an uncontaminated startup memory profile. The serial path waits for engine liveness before starting the trainer. The early-start path overlaps the trainer’s CPU imports and setup with inference startup. It still delays trainer CUDA initialization until inference is live. That ordering is the useful part: imports overlap without allowing trainer allocations to distort vLLM’s initial memory assessment.

Collect a group

The orchestrator requests several rollouts for one task. Each model reply becomes part of a trace, with token masks distinguishing trainable actions from prompt or observation tokens. Single-turn arithmetic and a six-turn game have very different rollout costs even when the model size is identical. A warm fork server can reuse an already-imported interpreter when starting subprocess harness programs. Each rollout still gets its own process group and runtime state. This addresses Python startup overhead; it does not replace model generation or environment logic.

Assign credit and update

For GRPO, the algorithm scores a group of completed episodes and assigns advantages to trainable tokens. The trainer receives those annotations, computes its configured loss, performs backpropagation and updates the trainable weights. Qwen’s LoRA profile updates adapters; the small SmolLM2 profiles update full parameters. The new policy must then reach vLLM. A trainer step that finishes while inference keeps sampling old weights is an incomplete account of the loop. The pipeline tracks policy versions and bounds off-policy progress, and final shutdown waits for the final policy refresh.

Where the time goes

Whole-run time includes startup, rollout waiting, updates, refreshes, checkpoint I/O and shutdown. Timers from concurrent processes overlap. Adding the orchestrator’s duration to the trainer’s duration would count some wall time twice. RL entrypoint · Trainer loop Continue with moving the policy.