Hyperconnections
Notes on Hyper-Connections, which replaces fixed residual paths with learnable depth and width mixing across multiple streams so transformers can trade off gradient flow and representation diversity.
Notes on Hyper-Connections, which replaces fixed residual paths with learnable depth and width mixing across multiple streams so transformers can trade off gradient flow and representation diversity.