Introducing TorchTPU

Google started TorchTPU in mid-2025 in collaboration with Meta's PyTorch team, and the project is now available as open source. It is a native PyTorch backend for Google TPUs that aims to run existing PyTorch models out of the box, with migration reduced to small edits such as changing a device string from "cuda" to "tpu" and swapping the distributed backend from NCCL to a TPU-specific one. The authors describe it as in active development but already used in production for both inference and training workloads for some of the largest Google Cloud customers.
A good part of the post is a critique of what came before. PyTorch/XLA, Google's compiled PyTorch-on-TPU path since 2020, was built on lazy tensors and, per Google, required intrusive source changes to mark compilable regions, had severe compile-time latency, never supported eager execution, and trailed the ecosystem on distributed primitives. TorchTPU is positioned against that history and against TorchAX, which the authors characterize as forcing PyTorch into JAX's way of working. The stated principles are that the runtime carries no JAX dependency, that eager mode is solid, that torch.compile is built on Torch Dynamo rather than lazy tensors, and that TPU collectives sit at the low level of PyTorch's Process Group API so DDP, FSDP and DTensor work.
The usability story is the PyTorch autoload mechanism: on a TPU machine with the TorchTPU wheel installed, importing torch picks up the backend. The post shows three snippets, a CUDA version using torch.device("cuda") and the nccl backend, a TPU version using "tpu" and tpu_dist, and an accelerator-agnostic version using torch.accelerator.current_accelerator() with a plain init_process_group. The obvious trade-off is acknowledged: a one-line device change gets a model running but not necessarily at peak TPU performance, because shapes, sharding and operator choice may be tuned for a GPU. This suggests porting teams should expect a second optimization pass after the first run.
Reported coverage at the time of writing includes eager and compiled modes plus a new DeferAndFuse mode; roughly 450 Aten ops, with forward and backward for all Core ATen ops the major architectures need; custom kernels authored via JAX/Pallas and the Helion DSL; and validated compatibility with Hugging Face Transformers v5, Diffusers and Kernels, with out-of-box execution on the 5,000 most downloaded Hugging Face models. Those span LLMs (Qwen, DeepSeek, Gemma, Llama, Mistral, GPT, OPT, BLOOM, Falcon), speech (Whisper, Wav2Vec2, HuBERT, MMS), vision (DinoV2, SigLIP, CLIP, Swin, ConvNeXt, ViT, ResNet, DETR) and text encoders (BERT family, RoBERTa, DeBERTa, T5, MT5, Flan-T5). DTensor sharding, multi-host inference through vLLM and SGLang, training through TorchTitan with Ray orchestration, and integration with TorchRec are all said to be validated.
On hardware, the post sketches TPU internals: TensorCores built around systolic-array MXUs of 256x256 multiply-accumulators, SparseCores for gathers, scatters, embeddings and collectives that can run in parallel, 288 GB of HBM at 8.6 TB/s on TPU 8i, inter-chip interconnect of 19.2 TBps on 8t and 8i, and 400 GBps data-center networking on 8t. Programming goes through the XLA stack at two levels, StableHLO for ML ops and Mosaic for custom kernels, with the TPU instruction set proprietary. A key structural difference from GPUs: a TensorCore behaves like a single-threaded processor with huge matrix units, so wide shapes matter more than launching many small parallel kernels. That helps explain why GPU-tuned model shapes may underperform on TPU.
Execution design has three eager modalities, settable by environment variable or context manager. Strict is the default op-by-op asynchronous dispatch akin to eager CUDA. Debug, via TORCH_TPU_LAUNCH_BLOCKING=1, makes execution synchronous for debugging and adds bounds checks. DeferAndFuse, via TORCH_TPU_DEFER_AND_FUSE=1, defers and fuses ops into larger chunks; the post says it can be 50% or more faster than Strict eager while lowering peak HBM use because fusion cuts temporary storage. Materialization happens when a value is needed, such as a .to("cpu") or .item() call; TorchTPU liveness-analyzes the deferred region, builds a compilation cache key for the resulting graph, looks it up, and compiles asynchronously on a miss. A repeated-ops heuristic triggers materialization early for patterns like stacked transformer blocks. The compilation cache has three tiers: in-process memory, machine-local shared memory across processes, and a persistent cache on remote storage such as Google Cloud Storage for reuse across hosts and restarts.
Precision is mapped to TPU hardware. BF16 feeds the matrix units with FP32 accumulation, while non-contracting work like activations, layernorm and softmax runs in IEEE FP32 on the vector unit. The default mode, medium, is single-pass BF16 multiply with FP32 accumulate. High uses a three-pass split for roughly 3x compute cost, and highest a six-pass scheme at roughly 6x; selecting either raises a TORCH_WARN_ONCE performance warning, and a thread-local torch.tpu.precision context manager scopes the setting to sensitive layers. Compared with CUDA, where the default is pure FP32 and fast math uses TF32 tensor cores, TPU defaults to the lower-precision path.
Two constraints stand out. A TPU device can only be opened by one user-space process, so TorchTPU uses one process per device rather than JAX's one process per host, and reinforcement-learning colocation cannot work out of the box while the team looks for workarounds. Separately, XLA and the PjRT runtime historically assume shapes known at compile time; bounded dynamism, where maximum bounds are fixed at compile time and actual sizes arrive at runtime, is being added to avoid recompilation on shape changes.
Why it matters: teams with PyTorch code and a reason to consider TPU capacity, whether cost, energy or supply, get a migration path that does not require rewriting into JAX or TensorFlow, and tooling vendors in inference and training (vLLM, SGLang, Ray, TorchTitan) already have validated integrations. Expect the first port to be easy and the second pass, retuning shapes, sharding and precision, to be where real performance work happens, an inference from the authors' own caveats. The one-process-per-device rule is the biggest architectural gotcha for anyone running mixed workloads on shared hardware.
Based on reporting from the original publisher. Visit the source for full context and later updates.
Publisher excerpt
TL;DR TorchTPU is an open source collaboration between Google and Meta that introduces a native and principled PyTorch backend for Google TPUs. It enables seamless out-of-the-box model execution and TPU... The post Introducing TorchTPU appeared first on PyTorch .