GRPO compares several rollouts of the same task. In this implementation, the baseline weights each reward by the number of trainable sampled tokens. If all shaped rewards tie, there should be no relative credit to assign.

The failure

For rewards 0.4 and 0.4, with lengths 33 and 34, the older FP32 calculation produced an advantage of about 2.98e-8 for both episodes. The admission rule checked exact nonzero credit, so it treated numerical residue as trainable signal. In the stopped Wordle investigation, 360 of 524 admitted episodes had advantage magnitude below 1e-6. This was worth fixing even though it did not explain every gameplay failure.

Center before taking the mean

The current function subtracts one reward first. Tied rewards then become exact FP32 zeros before the weighted reduction: This preserves the estimator algebraically. It does not change optimization/reduction dtype, the environment’s rewards or the exact-nonzero admission rule. Unequal rewards still produce signed advantages. Optional reward shaping runs before this function. For a simple unequal example, rewards 0.2 and 0.8 with equal lengths have mean 0.5 and advantages -0.3 and +0.3. With unequal lengths, the longer rollout contributes more to the baseline.

Edit one function, keep the rest of the pipeline

Initialization creates algo.py with a working starter:
Inputs are one-dimensional CPU FP32 tensors. Return one finite scalar per trainable trace. The algorithm rejects a wrong shape or non-finite values before assigning them to tokens. The installed integration probe changed the return value to default_advantages(rewards, lengths) * 0.5. It completed two updates, and all eight trainer-consumed trace annotations matched the edited values. Both gradient norms were nonzero. This proves the hook reached training; it is not evidence that halving advantages improves learning. Implementation · Tied-reward diagnosis · Edited-function evidence Continue with the hillclimb.