TorchTitan Update: New SOTA Models, Performance at Scale, TitanRL

The TorchTitan team has published a year-in-review update on the PyTorch-native training platform, covering new frontier-model support, the introduction of reinforcement learning through a component called TitanRL, out-of-the-box performance work for mixture-of-experts models, and new distributed infrastructure. TorchTitan is described as a minimal, clean-room implementation of PyTorch-native training techniques for large-scale pre- and post-training of generative models, with extension points meant to keep it fork-free for users with custom needs.
On models, TorchTitan added PyTorch-native support for Muse Spark Open, Muse Glimmer, Qwen3-VL and Qwen3.5, Kimi K2.7, Kimi K3 and DeepSeek-V4. The team says these were full training-stack integrations spanning its 5D parallelism stack, not partial ports. To make that manageable, the release introduced a composable configuration system built on typed Python recipes, model-config transforms and targeted component overrides, so a user can swap parallelization strategies, model components or hardware-specific kernels without maintaining a separate model fork. Those same interfaces wire in optimized kernels from Attention Gym (Gated DeltaNet, Kimi Delta Attention and sparse-attention kernels) and opt-in Helion kernels such as fused RoPE, while staying compatible with torch.compile. The shared toolkit accumulating around these models includes multimodal data pipelines, sample packing, shared vision components, Hugging Face checkpoint interoperability, declarative sharding and numerical validation.
The headline addition is TitanRL. A common pattern in RL stacks is to keep separate training and inference implementations of the model, which means architecture changes must be kept aligned across two copies and it becomes hard to tell a reward regression caused by an algorithm change from one caused by the two copies disagreeing. TitanRL keeps the same model from pretraining through post-training and inside vLLM, so the work lives in one codebase. Rollouts run asynchronously, and training and generation each scale with multiple forms of parallelism. The team reports stable training on MoE and dense models for hundreds of steps on tasks involving long context, code execution, compaction, harness use and self-play. Its two reported evaluations use Qwen3.5-35B-A3B base: SWE-Rebench easy at 64K context on 40 GB300s, and a train run on Tmax 15k with evaluation on terminal bench 2.1 at 128K context on 20 GB300s.
A correctness feature worth noting is batch-invariant mode. In supported on-policy configurations, the trainer and generator produce bitwise identical log probabilities, so the gap between them is exactly zero at every step. That removes one source of noise when diagnosing a reward change after enabling a new algorithm or kernel. In practice, this kind of guarantee is valuable because the trainer and the inference engine usually run different kernels and batching schedules, so small numerical divergence can accumulate; a bitwise-identical mode lets a researcher isolate a real behavioral change from numerical drift, at least within the configurations the mode supports.
The performance work concentrates on MoE models across three areas: faster kernels for higher compute utilization, better communication-compute overlap, and finer-grained memory control. local_compile applies native torch.compile benefits to selected regions without committing to full-graph or full-block tracing, preserving numerics without hand-maintained custom kernels. torch_remat is a modern activation-checkpointing API with fine-grained control over which regions get recomputed, avoiding CPU overhead and custom op wrappers. Dist-MoE provides fused communication-compute kernels for NVIDIA SM100+ GPUs with activation packing for dynamic tensors that remains fully CUDA-graphable, and a revamped pipeline-parallel package delivers CUDA-graphable, deterministic execution with knobs for maximizing overlap when composed with FSDP. TorchTitan overrides also allow model-specific kernels such as fused MLA, fused SwiGLU and fused Complex RoPE to be integrated without complicating the core code.
The team benchmarked DeepSeek V3 671B with the MTP1 module on the MLPerf training benchmark on an internal GB300 cluster, scaling only the PP, DP and EP axes with MXFP8 quantization. Across five configurations from 256 to 512 GPUs, throughput per GPU ranged from 5,324 tokens/sec on the PP2-256 run down to 4,780 on PP8-512, with global throughput reaching 2,694,493 tokens/sec on the PP2-512 setup at 1,535.93 TFLOPs/GPU and peak HBM around 221 GiB. The team states that TorchTitan matches Megatron loss convergence on this benchmark. For MuseSpark 1.2, the team reports long-context training efficiency with TP, EP and CP scaling: a 512-GPU, 192K-context run with forced balanced routing reached 1,588.22 tokens/sec per GPU and 813,168 globally, at 929.33 TFLOPs/GPU and 37.17% MFU.
FlexShard is a new data-parallel library for sharding model and optimizer state, integrated to go beyond what FSDP2 allows. It uses a bucket-centric API where each parameter bucket follows a custom Placement policy, with runtime overlap of bucketed communication and computation. Beyond FSDP1/FSDP2-style placements it adds Owned, where a single rank owns an entire parameter, and BlockShard, which allows 2D tile-aligned sharding. That lets the distributed Muon optimizer shard Muon-optimized parameters with Owned and BlockShard, eliminating All Gather collectives before the optimizer update; the team measured up to 79% improvement in optimizer time on DeepSeekV3 at 256 H100s versus an FSDP-only implementation. BlockShard also aligns sharding boundaries with blockwise quantization, so FP8-quantized parameters can be all-gathered with scaling factors needing no extra collectives, cutting communication volume by roughly 50% and improving end-to-end step time by up to 20% on Llama3 405B at 256 H100s.
GraphTrainer offers a graph-mode training path built around an explicit training graph, with graph-based and eager-based paths producing bitwise-matching numerics. It enables cross-module fusions, graph-level debugging and tailoring for heterogeneous accelerators. On infrastructure, PyTorch 2.14 ports the torchcomms NCCL implementation into c10d as an opt-in backend registered under the standard nccl name, adding one-sided RMA windows, in-place process-group reconfiguration for fault tolerance, raw access to NCCL shrink/grow and suspend/resume APIs, and eager communicator initialization in PyTorch 2.15. A new torch_checkpointing library removes per-save synchronization from default distributed checkpointing: one collective shares metadata before the first save and none follow. The team says this matters past 16k ranks, where GLOO-based synchronization scaled poorly, and measured improvements at 512 H100s training DSv3 671B with a checkpoint interval of 50.
Ecosystem adoption is also reported: TorchTitan is a training backend in veRL and has been added to Miles, is the default training backend for Meshy, and inspired the core implementation of Areal's Archon training engine.
Why it matters: Teams training large open-weights models increasingly choose between Megatron-style stacks and PyTorch-native tooling, and a framework that ships frontier architectures with validated parallelism, RL support and checkpointing in one codebase lowers the cost of switching. The FlexShard and Dist-MoE results suggest that when sharding layout is co-designed with the optimizer and quantization scheme, meaningful speedups follow, which is likely (inference) to push framework authors toward more flexible placement APIs. For product teams, the practical change is that agentic RL experiments can now run on the same model definition used in pretraining and serving, with a debugging mode that separates numerical noise from real regressions.
Based on reporting from the original publisher. Visit the source for full context and later updates.
Publisher excerpt
TorchTitan is a PyTorch-native platform for rapid experimentation and large-scale pre- and post-training of generative AI models. It is designed as a minimal, clean-room implementation of PyTorch-native techniques, with flexible... The post TorchTitan Update: New SOTA Models, Performance at Scale, TitanRL appeared first on PyTorch .