AivexaNewsSearch
AI news for builders and product teamsChecked every hour

Last year we built megakernels. This year we deleted them: The road to Together Compiler

Collected Nov 16, 2026

Together AI has published a technical post describing Together Compiler, a framework it built for continual AI inference optimization, in which AI agents rewrite and fuse the GPU kernels executed during a model's forward pass. The company reports that this approach outperforms the hand-written megakernels it released last year by 28% on NVIDIA B200 hardware, hitting 2,118 tokens per second versus 1,649 tokens per second on Llama 3.2 1B. The post traces a two-year arc from hand-crafted megakernels, which proved hard to productionize, to an agent-driven compiler.

The starting point is the difficulty of squeezing performance out of standard tooling. Rendering models as sequences of individual PyTorch operations introduces kernel launch overhead, limits overlap between compute and memory movement, and leaves GPU resources idle. Torch Compile, powered by Inductor, fuses operations and cuts some of this cost, and CUDA Graphs reduce launch overhead further. On the Llama 1B workload Together studied, plain Inductor delivered 669 tokens per second. The team argues that Inductor's heuristics and its benchmarking harness leave substantial performance on the table, and it presents three findings to support that claim.

The first finding concerns the sheer size of the configuration space. As of the third quarter of 2026, Inductor exposes 607 tuning flags, 198 of them added in the past year, and their effects interact in ways that are not predictable from the flags alone. Together cites an example where the freezing flag, which tells the compiler that weights will not change during inference, improved throughput by 10 to 26% at larger batch sizes but caused a 49% slowdown at batch size 1, because a conversion of weights to FP32 was not reversed for matrix-vector operations. Elsewhere, enabling an aggressive fusion flag alone cost 6 tokens per second on Llama 1B, yet enabling it alongside several other flags produced a 3 tokens-per-second gain.

The second finding is that the fastest kernel measured in isolation is not necessarily the fastest in the context of a full inference pass. Together reports that Inductor clears the L2 cache before timing kernels individually, producing timings 39 to 58% slower than the same kernels inside a live forward pass where cached data is reused. Conversely, benchmark inputs can distort results in the other direction: when timing torch.embedding, Inductor initializes token IDs to zero so every token reads the same embedding vector, which for a 64K-token sequence made the kernel appear 2.1 times faster than with random IDs. Measurement overhead also matters for short kernels, with the harness reporting 2.5 microseconds for work that does no GPU computation and other measurements running roughly 3.6 microseconds above Nsight Systems figures. On Llama 1B's output-projection GEMM, this led Inductor to pick a Triton kernel taking 7.4 microseconds over a cuBLAS option at 6.4 microseconds in the live pass. Together also reports 4 to 21% slowdowns for a mixture-of-experts grouped-GEMM kernel on GLM 5.3 when benchmarked with random inputs.

The third finding concerns fusion itself. Inductor uses a greedy, pairwise algorithm that weighs candidate fusions by estimated memory savings. Together points to two consecutive kernels in Llama 1B decode, an elementwise activation and a KV indexing kernel, that share no memory accesses and are therefore left separate, even though the first launches only six thread blocks and leaves most of the GPU's 132 streaming multiprocessors idle. The team's reading is that fusion should consider occupancy and utilization, not only memory traffic.

Megakernels were the response: one persistent kernel that executes an entire forward pass, keeping thread blocks resident, coordinating work with specialized warps and dividing shared memory into pages that operations allocate and release as execution proceeds. This reached 990 tokens per second on Llama 1B, a 48% gain over Inductor's 669. But productionizing it took months of work on paged KV caches, porting to Blackwell GPUs and debugging race conditions, and yielded a single megakernel supporting one batch size (1), one precision (BF16) and one customer.

That outcome prompted a decomposition experiment. Starting from Torch Compile's 669 tokens per second, the team ablated the benefits megakernels provide and found that CUDA Graphs plus Programmatic Dependent Launch, a CUDA feature that lets a following kernel begin before its predecessor finishes, on top of default Inductor kernels recovered 98% of megakernel performance without writing a kernel. Launch overhead is the reason the headroom exists: Together measures 37 microseconds of ATen dispatch for an H100 Llama 1B down-projection whose GPU execution is 15 microseconds, and pure C++ launches still cost about 2 microseconds each. CUDA Graphs cut launch overhead to roughly 1 microsecond and lifted Llama 1B to 864 tokens per second, 87% of the megakernel. PDL was inserted by compiling Inductor's kernels to PTX and adding instructions, since the Triton backend did not expose the needed controls.

From there, the team pursued "multi-kernels" plus agent loops, which it says beat the old megakernels by up to 1.7 times and exceed 1,100 tokens per second for Llama 1B on H100. The post frames these as the seeds of a system that searches the optimization space systematically rather than relying on fixed heuristics, and says it plans to detail that system in a follow-up.

A likely trade-off for readers is operational: the compiler depends on agents exploring kernels, which implies search and verification cost and a need to validate generated kernels against output correctness before serving, a concern the post does not address in the material available. The results are also reported for a 1B-parameter model, so how the gains scale to larger models and mixture-of-experts architectures remains to be shown.

Why it matters: Teams serving models on NVIDIA GPUs face a multiplication problem: new architectures, batch sizes, sequence lengths and hardware keep appearing, and hand-tuned kernels are expensive to maintain across all of them. Together's account suggests a large share of megakernel gains can be recovered with CUDA Graphs and PDL on ordinary compiler-generated kernels, and that agent-driven kernel search can go further, which in inference is likely to shift effort from writing kernels by hand to building and validating automated search pipelines. Developers evaluating serving stacks should treat these numbers as vendor-reported on specific hardware and model sizes.

Read at Together AI

Based on reporting from the original publisher. Visit the source for full context and later updates.

Publisher excerpt

Together Compiler uses AI agents to rewrite and fuse the GPU kernels in a model’s forward pass. It beats hand-written megakernels by 28% on B200 (2,118 vs 1,649 tokens/s on Llama 3.2 1B). This is how we got from one to the other.