AivexaNewsSearch
AI news for builders and product teamsChecked every hour

PyTorch 2.15 Release Blog

Collected Oct 28, 2026

PyTorch 2.15 is out, published by the PyTorch project, with 3,117 commits from 583 contributors since version 2.14. The headline change for distributed workloads is that the rewritten NCCL backend, previewed as opt-in nccl2 in 2.14, now becomes the default for every nccl process group. Users who hit problems can roll back with TORCH_DIST_USE_NCCL2=0 or by selecting the nccl-legacy backend name.

The new NCCL backend, ported from torchcomms, creates communicators eagerly alongside the process group. Reconfiguration can opt into a shrink/grow fast path on NCCL 2.29 or newer, letting surviving ranks skip a full bootstrap, and reconfigure for fault tolerance is enabled on ROCm with RCCL 2.30.7. The distributed API remains marked unstable, with PRs credited to Tristan Rice, Chinmay Kuchinad (AMD), Tushar Jain and Atharva Kshirsagar (Red Hat).

Operationally, the most interesting addition is a backend-agnostic flight recorder that writes a post-mortem to disk when a collective hangs. Watchdog health is published to the debug server, so a timeout on any single rank triggers dumps from every rank. The project's framing is that one stalled GPU can halt an entire training run and diagnosis has often meant reproducing the failure manually; the recorder aims to capture what each rank was doing at the moment of the hang. For teams running large jobs, that is a debugging affordance rather than a speed feature, and it is the kind of tooling that tends to matter more as cluster sizes grow.

torch.distributed also gains a Python transport API for one-sided data movement, with NIXL as the first backend. Transports connect to a peer through a store without needing a process group or collectives, and Transport.read_stream and write_stream enqueue transfers on a CUDA stream so they can be captured with torch.cuda.graph. This suggests one-sided transfers are being positioned as a building block for custom communication patterns and for graphs that bundle data movement with compute. The transport API is also marked unstable.

FSDP2 adds finer control over precision and capture. MixedPrecisionPolicy now accepts a param_dtype_override_fn, letting precision-sensitive parameters such as MoE router gates and norm weights stay in fp32 while the rest of the model trains in bf16, still with one all-gather per fully_shard. A full gradient-accumulation step can be captured in a single CUDA graph, and gradient precision can be set per parameter, with gradients accumulating directly in .grad and following each parameter's grad_dtype. That last change alters default behavior when reduce_dtype=None and appears under backwards-incompatible changes.

Pipeline parallelism gets a reworked memory profile. Receive buffers are allocated just before each receive rather than for every microbatch up front, and forward outputs are released once their sends complete. Schedules accept a max_outstanding_sends budget, FSDP unshards are staggered across each rank's startup bubble instead of all being issued at the start of a step, and each unique edge between pipeline stages gets a dedicated P2P communicator. FSDP reduce-grads can now overlap with subsequent compute. The stated result is that pipelining becomes cudagraphable and deterministic with an optimized memory profile, plus knobs for minimizing exposed communication. Credit goes to Chien-Chin Huang, Anshul Sinha, Andrew Or, Animesh Jain and Sanket Purandare.

On NVIDIA Blackwell, NVGEMM now folds the scalar dequantization factor of ModelOpt NVFP4 linear layers into the GEMM, removing a separate pointwise launch per layer during decode. A bounded, shape-aware candidate policy covers Blackwell NVFP4 decode shapes that nvMatmulHeuristics misses, kernels can use programmatic dependent launch so a Triton consumer overlaps with the tail of the producer GEMM, and NVGEMM moves onto CUTLASS Operators. In practice, the 4-bit decode gains come from fewer and better-overlapped kernel launches rather than a new arithmetic path.

Apple Silicon sees a broader set of native Metal kernels replace MPSGraph operators. A single-row softmax over a 128,256-entry vocabulary drops by 8x, upsample backward is up to 21.7x faster, and shared reduction kernels make norm full reductions more than 100x faster. For decode, F.linear now uses existing gemv kernels (1.23x geometric-mean speedup, up to 5x on small projections), and the new MPS grouped_mm has a gemv path giving 1.32–1.34x higher MoE decode throughput on an M3 Ultra. QR of a 512 x 512 matrix falls from about 7 seconds to 1.5 ms, and triangular solves are now blocked. MPS also gains torch.float8_e4m3fn support, _scaled_mm with tensorwise or rowwise scaling and a gemv fast path, and a public stream API modeled on CUDA's: torch.mps.Stream, current_stream, set_stream, the torch.mps.stream context manager and Tensor.record_stream.

Platform support widens on several fronts. On AMD, FlyDSL joins CuTeDSL, Triton and Helion as a kernel DSL; on gfx950, Inductor autotunes FlyDSL templates for GEMM, ragged F.grouped_mm and MXFP4/MXFP8 scaled_mm alongside Triton and ATen, and eager mode can override RMSNorm and topk. FlyDSL is optional and requires version 0.3.x. On gfx1250 (MI450), Inductor uses the Tensor Data Mover for dense mm/addmm, pointwise and reduction codegen plus FlexAttention forward and decode, and gfx1250 joins the nightly wheel architecture list. ROCm 10.0 wheels ship alongside the ROCm 7.14 wheels from 2.14; all ROCm wheels are now built on AMD's TheRock packages and install ROCm via a pip dependency rather than bundling the libraries.

For Intel GPUs, SYCL free-function kernel support replaces verbose functor-based implementations, with most existing SYCL kernels already migrated. torch.xpu.Event gains IPC support for cross-process sharing and torch.multiprocessing integration, and torch.xpu.MemPool supports use_on_oom and non_split allocation policies. Profiling adds kernel-level visibility into XPU Graph replay, cross-rank correlation for XCCL collectives and per-kernel hardware counters across all Intel GPUs in multi-GPU workloads.

CUDA-wise, the default wheel moves to CUDA 13.2 (13.2.2 toolkit), with CUDA 13.4 GA wheels alongside for Linux x86_64, Linux aarch64 and Windows; those add experimental support for NVIDIA Rubin GPUs. Python support runs from 3.11 through 3.15, including free-threaded 3.14t and 3.15t, and Dynamo adds Python 3.15 support. On the C++ side, torch::stable::sort joins the stable ABI, the Python interop shims gain dtype/device conversions and an is_tensor probe, more of c10/util moves into torch/headeronly with old paths kept as forwarding includes, and extensions error at compile time if TORCH_TARGET_VERSION is newer than their libtorch headers. load_inline can generate pybind11 modules safe to import on free-threaded Python.

Debugging gains CUDA graph kernel attribution: kernels replayed from a graph can be traced back to the code that captured them, with annotations recorded from CUPTI node-creation callbacks and written inline by export_chrome_trace, and record_py_stacks capturing the Python launch stack per node. The CUPTI monitor is renamed Cuspy and configured with CuspyConfig.

Breaking changes to note: Python 3.10 is dropped entirely (no wheels, no CI, no source builds), macOS 14 is no longer supported for MPS acceleration with macOS 15 required, C++ warnings from torch.ops and custom ops now propagate to Python, and FSDP2 separates sharded and unsharded gradient dtypes. A Q&A webinar is scheduled for Thursday, November 5, 2026 at 11 AM PT with Tristan Rice, Natalia Gimelshein, Huy Do and Jane Xu of Meta, moderated by Chris Gottbrath.

Why it matters: Anyone training across many NVIDIA GPUs gets a different communication backend by default, so teams should test 2.15 against their workloads and know the TORCH_DIST_USE_NCCL2=0 escape hatch before upgrading production jobs. The flight recorder and CUDA graph attribution are likely to shorten the debugging cycle for hangs and graph-captured kernels, which is my inference from the features described rather than a measured claim. Apple Silicon and AMD/Intel users gain meaningfully faster decode and kernel-authoring paths, while teams on Python 3.10 or macOS 14 must plan upgrades first.

Read at PyTorch

Based on reporting from the original publisher. Visit the source for full context and later updates.

Publisher excerpt

We are excited to announce the release of PyTorch® 2.15 (release notes)! The PyTorch 2.15 release features the following changes: Rewritten NCCL backend becomes the default, with a flight recorder... The post PyTorch 2.15 Release Blog appeared first on PyTorch .