AivexaNewsSearch
AI news for builders and product teamsChecked every hour

Google Open-Sources TPU Sync for KV Cache and Weight Transfer on TPUs

Collected Nov 12, 2026

Google Cloud has open-sourced TPU Sync, a native C++ library for moving KV cache blocks and model weights between TPU HBM, host DRAM, shared storage and the data center network. The source article by Daniel Dominguez ties the release to two workloads where accelerators spend time waiting on data: long-context and disaggregated LLM serving, and reinforcement learning loops that push updated weights from trainers to rollout samplers. The code is published under the Apache 2.0 license, and Google Cloud also announced the TPU 8tx and TPU 8i Max systems alongside it.

The headline numbers concern prefill/decode disaggregation, where the prefill phase that builds the KV cache runs on one set of hosts and decoding runs on another, so the cache has to cross the network. Google says median KV transfer latency for Qwen3 235B fell from several seconds to roughly 230 ms. Average time-to-first-token dropped by 60% for that model and by 90% for a 480B model at low concurrency. For prefix caching, Google reports up to 90% lower average TTFT on 64K-context workloads.

Those figures point at a familiar bottleneck. A 64K-context session can carry a KV cache measured in gigabytes, and every hop between machines adds latency before the first token appears. Cutting transfer time from seconds to a fraction of a second changes what deployment topologies are practical, since disaggregated serving and cross-node prefix reuse both live or die on cache movement.

Developers get Python wheels for JAX and PyTorch, exposed through a KV cache manager, a KV store and weight synchronizer APIs. TPU Sync plugs into vLLM-TorchTPU as the KV connector via vLLM's KVConnector interface, and serves as the disaggregation and hierarchical-cache backend for SGLang-TorchTPU. It also pairs with llm-d 1.0 on GKE for prefix-cache-aware routing. The weight-sync module for Pathways and multi-controller JAX is labeled experimental.

The GitHub repository, which absorbed the earlier TPU Raiden codebase, carries a clear caveat: APIs are changing rapidly, only development builds are on PyPI, and the project recommends pinning to its last-known-good commit. It is validated on TPU v7x and v6e, and the PyTorch wheel needs a matching torch_tpu build that is not on a public index, so PyTorch users will have to build or source the TPU backend themselves. The repository includes architecture documentation, C++ and Python APIs, and examples for single-host and multi-host disaggregated serving and host KV offloading. The storage tier and the RL weight synchronization remain experimental pending a stable release.

The GPU ecosystem has comparable pieces: NVIDIA Dynamo and its NIXL transfer library, LMCache for offloading KV cache to CPU memory, disk or remote stores, Mooncake as the KVCache-centric serving architecture behind Moonshot AI's Kimi, and vLLM's own disaggregated prefill connectors. TPU Sync differs mainly in being tied to PJRT and TPU tile layouts, and in covering RL weight transfer inside the same library. This suggests the practical advantage is closer to reduced glue code than a wholly new capability.

Early reactions in the article focused on data movement rather than the new chips. One developer, Anand Butani, said that cutting KV transfer latency 90% matters more than the chip names, because long-context serving mostly dies on moving cache between nodes. Another, posting as @lgoodbookd, argued that bandwidth alone does not solve the problem: 8i Max doubles HBM bandwidth to about 17.2TB/s, yet a single 64K session on a 235B MoE model still consumes roughly 12.6GB of KV cache. That commenter pointed to host-DRAM KV tiering at about 223GB/s per v7x host, concluding that bandwidth helps but capacity remains the wall. The reasoning is sound as a rule of thumb: bigger caches need somewhere to spill, and tiering is cheap only if moving data out and back is fast.

In practice, teams running JAX or PyTorch on TPU v7x or v6e with vLLM, SGLang or llm-d on GKE are the immediate audience, and the pin-to-a-commit advice is worth taking literally given the development-only builds. A likely trade-off is that the host offload path trades the high bandwidth of HBM for much larger DRAM capacity at a fraction of the throughput, so it helps most when sessions outnumber what HBM can hold at once.

Why it matters: developers serving long-context models on TPUs get a supported path to disaggregated prefill and hierarchical cache offload that previously had to be built in-house. Product teams should read the latency and TTFT gains as indicators, since the library is pre-stable, the PyTorch wheel needs a matching torch_tpu build, and storage tiering and RL weight sync are still experimental. The plausible effect, and this is inference, is a lower bar for costlier topologies like cross-node prefix reuse and separate prefill pools.

Read at InfoQ · AI, ML & Data Engineering

Based on reporting from the original publisher. Visit the source for full context and later updates.

Publisher excerpt

Google Cloud has open-sourced TPU Sync, a native C++ library that moves KV cache blocks and model weights between TPU HBM, host DRAM, shared storage, and the data center network. By Daniel Dominguez