The failure
For rewards 0.4 and 0.4, with lengths 33 and 34, the older FP32 calculation produced an advantage of about2.98e-8 for both episodes. The admission rule checked exact nonzero credit, so it treated numerical residue as trainable signal.
In the stopped Wordle investigation, 360 of 524 admitted episodes had advantage magnitude below 1e-6. This was worth fixing even though it did not explain every gameplay failure.
Center before taking the mean
The current function subtracts one reward first. Tied rewards then become exact FP32 zeros before the weighted reduction: This preserves the estimator algebraically. It does not change optimization/reduction dtype, the environment’s rewards or the exact-nonzero admission rule. Unequal rewards still produce signed advantages. Optional reward shaping runs before this function. For a simple unequal example, rewards 0.2 and 0.8 with equal lengths have mean 0.5 and advantages -0.3 and +0.3. With unequal lengths, the longer rollout contributes more to the baseline.Edit one function, keep the rest of the pipeline
Initialization createsalgo.py with a working starter:
default_advantages(rewards, lengths) * 0.5. It completed two updates, and all eight trainer-consumed trace annotations matched the edited values. Both gradient norms were nonzero. This proves the hook reached training; it is not evidence that halving advantages improves learning.
Implementation · Tied-reward diagnosis · Edited-function evidence
Continue with the hillclimb.