Hugging Face ❤️ TorchTPU: TPUs, Natively
Hugging Face published a post describing TorchTPU, a native PyTorch backend for Google's Tensor Processing Units, along with TPU support across its own libraries. Rather than routing PyTorch code through PyTorch/XLA or moving it to JAX, developers can now treat a TPU like any other device. Transformers, Diffusers and Kernels are said to run natively on TPU: pick a model from the Hub, send it to "tpu", and it runs.
TPUs are Google's in-house AI accelerators, designed over roughly the past decade for the matrix arithmetic that dominates neural network workloads. Google uses them to train and serve its own models, including Gemini, and they are sold through Google Cloud in configurations from a single chip up to pods of thousands of chips. Their traditional weakness has been software rather than silicon: the TPU toolchain grew up around TensorFlow and JAX, which left PyTorch users with extra work. This suggests the release is as much about closing that software gap as about raw performance.
The previous path was PyTorch/XLA, or torch_xla. It worked, but its tensors are lazy, meaning computation is deferred and code does not always execute when it reads as though it should. Small changes could force graph recompilation, and reasoning about failures was difficult. Hugging Face had built Optimum TPU on top of that to give its models inference and training support on TPUs. With TorchTPU providing the native integration, Optimum TPU is no longer required to run Hugging Face models with Transformers and Diffusers.
TorchTPU adds TPU as a PyTorch backend, so ordinary PyTorch code applies. It ships Pallas kernels tuned for TPUs, supports distributed training and integrates torch.compile for extra speed. Installation on a TPU VM is a single pip command, torch-tpu, pointed at PyTorch's CPU wheel index; that extra index matters because without it pip would fetch the much larger CUDA build, which is useless on a TPU.
For Transformers, the post frames version 5 around the goal of being the source of truth for model definitions so they run anywhere, and TPUs now count as anywhere. Loading onto a TPU means passing device_map="tpu" to from_pretrained. In the example, a Qwen/Qwen3.5-9B causal language model loads that way, the tokenizer encodes a prompt with return_tensors="pt", inputs are moved to "tpu", and generate is called for 64 new tokens. Models run in eager mode by default, with torch.compile available for better performance and distributed setups supported.
Diffusers treats TPU as a first-class device. For prototyping or debugging, eager mode needs only pipe.to("tpu"); for deployment, a single compile call enables compiled mode, and pipelines and code stay otherwise unchanged, with no TPU-specific fork to maintain. The sample uses StableDiffusion3Pipeline with stabilityai/stable-diffusion-3.5-medium in bfloat16, moves the pipe to tpu, and optionally compiles the transformer with backend="tpu", fullgraph=True and dynamic=False. The first invocation compiles, and later calls at the same image size reuse the graph.
For models too large for one chip, a TensorParallelConfig passed to from_pretrained shards the transformer across chips during loading, so the full weights never need to reside in one place. Many popular pipelines support this out of the box, and the rest can be configured with a tp_plan; the Diffusers TPU guide has the full recipe. The post shows examples run on a Cloud TPU v6e-4 with four-way tensor parallelism, bfloat16 and batch size 1.
The Kernels Hub carries ready-to-use TPU kernels, loaded through the Kernels library. TPU kernels are written in Pallas, which gives low-level hardware control, and kernels from the Tokamax library can also be brought in. One example loads a ragged-dot kernel from the Hub repo tengomucho/ragged-dot-tpu and registers it as the experts implementation for a Mixture-of-Experts model, Qwen/Qwen3-30B-A3B-Instruct-2507, via the transformers MoE integration. The reported gain scales with batch size: at batch 64, decoding is 21.5 times faster on Qwen3-30B-A3B and 12.8 times faster on Gemma 4 26B-A4B, measured on a TPU v6e-8 against the default implementation.
Hugging Face says it runs the transformers test suite on TPU to catch problems before users do, and is working toward a nightly TPU CI whose results appear on the transformers CI dashboard alongside other accelerators. Of the tests that run on TPU, 96.5% pass, which the post accepts as unfinished work. Each failure becomes a small repro script and an issue on the TorchTPU repository: 34 issues have been filed and 19 fixed. Running on TPU also surfaced bugs inside Transformers itself, including in StaticCache and the Qwen3 tensor-parallel plans, which were fixed upstream.
In practice, the value here is portability: the same model code can target GPU or TPU with a device change, which lowers the cost of trying TPUs for teams already standardized on PyTorch and the Hub. A likely trade-off is that eager mode keeps things simple but leaves performance on the table, while compiled mode brings the familiar compile-time cost and the risk that shape changes retrigger compilation because dynamic=False fixes the graph. Kernel-level gains like the reported MoE speedups also depend on batch size, so small-batch serving may see much less benefit. The 96.5% pass rate and the open issue count are a reasonable signal that the backend is usable but still maturing, and a TPU-dependent product team should expect to hit rough edges.
Why it matters: PyTorch developers and product teams who have avoided TPUs because of the XLA detour now have a supported path through the libraries they already use, which widens the hardware options for training and serving Hugging Face models. Teams on Google Cloud, in particular, can evaluate TPU capacity without maintaining a separate code path, though my inference is that production adoption will hinge on how quickly the remaining test failures and open TorchTPU issues are resolved. Kernel and compiled-mode work offer the largest performance upside for high-throughput workloads, while simple eager-mode ports are the lowest-effort starting point.
Based on reporting from the original publisher. Visit the source for full context and later updates.