How Model Growth, Recursion, and Boundary Operators Influence Scaling Exponents
Notes on how growing transformer depth during training changes the fitted compute-scaling exponent, and why looping helps when data is reused.
Notes on how growing transformer depth during training changes the fitted compute-scaling exponent, and why looping helps when data is reused.