Small-batch autoregressive decode repeatedly reads a model’s weights to produce very few output tokens. Launch overhead, memory traffic and register pressure can dominate. This is the reason to try a model-specific fused path on the laptop.

What a megakernel changes

The 135M route combines decoder work into a cooperative launch instead of dispatching the full layer stack through many kernels. Prefill and unsupported batch compositions use the regular path. vLLM still owns logits processing and sampling. The 360M work adds its own dimensions, weight layout, numerical gates and serving integration. A kernel specialized for SmolLM2 does not become a Qwen kernel merely because Qwen is also small. Qwen3-0.6B stays on stock vLLM in these presets.

Follow the dispatch boundary

The serving path must distinguish eligible decode from prefill, mixed batches and unsupported shapes. Unsupported work falls back to native execution. Weight refresh must invalidate or refresh cached packed representations, including any graph-captured references. The acceptance work checks final hidden states and every layer’s KV values, not only the top token. Matching a greedy token can hide errors in close logits. Conversely, one BF16 ULP can fail an exact gate even if the generated token stays the same.

The experiments that shaped the route

Lower shared memory does not automatically increase occupancy. Registers can remain the limiting resource. A lower-level PTX instruction also does not guarantee a faster composed decoder.

What I count as a gain

The accepted B1 serving change lowered full-generation medians by 0.75–0.88% at the measured contexts. That experiment excluded startup and RL training. Its number belongs on a serving panel, not on the whole-RL wall-time curve. Kernel fork · Experiments 233–240 Continue with credit assignment.