AivexaNewsSearch
AI news for builders and product teamsChecked every hour

Accelerating PyTorch on AMD GPUs with FlyDSL

Collected Oct 21, 2026

FlyDSL, a kernel-authoring layer that exposes hardware layout decisions directly in Python and lowers through MLIR, is now reachable from ordinary PyTorch code on AMD Instinct MI350-series GPUs built on the gfx950 architecture. Applications do not need to call custom kernels. Dense GEMM, grouped GEMM, fused multi-head attention, MXFP8/MXFP4 scaled GEMM, RMSNorm and TopK are all available through existing operator APIs, and workloads that fall outside the supported shapes keep their previous PyTorch implementations. Attention support arrives through AOTriton 0.14.2b, the library behind PyTorch's ROCm F.scaled_dot_product_attention, where FlyDSL sits alongside Triton and AITER ASM as a selectable backend.

How the backend is wired in depends on the operation. In eager mode, eligible RMSNorm and TopK calls dispatch to FlyDSL automatically once the optional runtime is installed and enabled. For GEMMs under torch.compile, TorchInductor filters candidates, benchmarks FlyDSL against ATen and Triton where both are supported, caches the winner per workload, and runs whichever measured fastest. For attention, AOTriton consults its offline operator-tuning database to pick among the three kernel families for each workload. FlyDSL attention kernels are also shipped for RDNA4 (gfx1201).

Dense GEMM accepts FP16 and BF16 in all four A/B layout combinations (NN, NT, TN, TT), including eligible transpose views. Grouped GEMM handles FP16 and BF16 with ragged 2D A and grouped 3D B, so experts can process uneven or even empty token assignments. MXFP scaled GEMM takes FP8 E4M3 or packed FP4 E2M1 operands with one E8M0 scale per 32 values along the reduction dimension, accumulates in FP32, returns FP16 or BF16, and currently requires K to be a multiple of 128. It supports an optional 1-D bias of length N but not fast accumulation. Under torch.compile, Inductor requires unit stride along K for both operands, inserting a layout copy if needed; that canonicalization applies to scaled GEMM only. Attention covers FP16 and BF16 with head dimensions from 32 to 512, forward and backward, but not FP32. RMSNorm is forward-only; backward stays with the existing implementation. TopK supports FP32, contiguous input, largest-and-sorted output, with functional and out= variants.

The reported benchmarks were run on MI355X GPUs and measure steady-state operator execution, excluding compilation and autotuning, so they are not end-to-end latency figures. MXFP timings start from already-quantized operands, leaving quantization out of the timed region. On 15 BF16 NT dense-GEMM shapes, FlyDSL posts a 1.10x geometric mean over the faster of ATen and Triton at each shape: 10 wins, three ties and two losses, with gains concentrated in small and mid-sized problems such as 64 x 4096 x 4096 at 1.33x. A later all-layout regression check reported 1.12x aggregate. On 17 shapes per format, MXFP8 averages 1.58x and MXFP4 1.68x over ATen, with ranges of 1.31x to 2.23x and 1.31x to 2.96x respectively. Across 24 grouped-GEMM cases FlyDSL reaches 1.20x over Triton and 1.95x over ATen, winning all five ragged cases. RMSNorm ranges from 1.17x to 3.66x over ATen across 22 cases, and small-K TopK reaches 4.82x geometric mean over ATen on 10 cases with deterministic algorithms disabled. In end-to-end vLLM A/B tests, whole-request speedup reaches 1.13x for BF16, 2.04x for MXFP8 and 3.01x for MXFP4, depending on model and concurrency.

The claim worth scrutinizing is the low-precision one: the MXFP8 and MXFP4 figures are much larger than the BF16 result, which is what you would expect when a matmul spends most of its time moving data rather than doing arithmetic. Lower-precision operands shrink bytes fetched per multiply, so better tile scheduling and scale staging should pay off more. The catch is that quantization cost sits outside the timed region, so a real deployment only captures a fraction of that gain unless the quantized weights are produced once and reused at serving time.

The engineering story is about explicit control. Triton's block-tensor model lets authors describe program-level tiles while the compiler handles thread and wave mapping. FlyDSL instead exposes layout algebra: partitioning across blocks, waves, threads and values, picking tiled-copy and MFMA atoms, defining LDS swizzles and staging, and placing synchronization points. On dense GEMM, tuned configurations use a 256 x 256 output tile with an eight-wave workgroup (512 threads, Wave64), split into half-tile interleaved quadrants and executed with phase-shifted barriers so one four-wave group computes while the other loads. The MXFP path adds separate scale slots that advance in chunks spanning several K tiles, prefetching the next chunk at each boundary. Grouped GEMM avoids per-expert kernel launches by giving persistent workgroups a strided sequence of tiles, skipping empty experts and swizzling work across the chip's eight Accelerator Complex Dies. TopK uses a register-resident bitonic path for small fixed K and a four-pass radix selection for large K. As an interpretation, this control cuts both ways: per-architecture templates can win at specific shapes, but they must be maintained and retuned as new silicon arrives.

A caveat readers should carry: FlyDSL's register-based TopK can order equal-value ties differently from ATen, and it canonicalizes NaNs, so exact indices may diverge even with determinism on. The tuning database also means an upgrade can shift which kernel runs underneath you.

Why it matters: Teams running BF16 or low-precision inference on MI350 hardware can pick up speedups without changing application code, and the benchmark-and-cache behavior under torch.compile means gains arrive per workload rather than as a blanket switch. The largest numbers land on MXFP formats, which likely matters most to serving stacks already quantization-aware, while the modest BF16 dense-GEMM edge suggests the automatic selection matters more than any single kernel. Because support is confined to gfx950 and specific shapes and dtypes, other AMD parts and FP32 attention workloads stay on existing paths.

Read at PyTorch

Based on reporting from the original publisher. Visit the source for full context and later updates.

Publisher excerpt

PyTorch users on AMD MI350-series GPUs can now use FlyDSL through existing operator APIs for dense and grouped GEMM, fused multi-head attention (FMHA/Attention), MXFP8/MXFP4 scaled GEMM, RMSNorm, and TopK, without... The post Accelerating PyTorch on AMD GPUs with FlyDSL appeared first on PyTorch .