Extending Inductor for the IBM Spyre Accelerator
IBM's PyTorch integration for its Spyre accelerator, torch-spyre, has switched its torch.compile backend from sendnn — a compilation stack built from scratch — to backend=inductor, implemented by extending PyTorch's own Inductor. The result is a compiler roughly 7x smaller than the one it replaced, according to a PyTorch project post. Most of Inductor carried over untouched; the work concentrated in three areas where Spyre's design diverges from GPU assumptions. The team says it is now working with the upstream PyTorch project to formalize the extension points it relied on most, so other out-of-tree hardware backends can reuse them.
Spyre is IBM's clean-sheet AI accelerator for enterprise inference, designed around power efficiency: a 75-watt, PCIe-attached card that needs no custom power delivery or cooling. It is a dataflow machine built from 32 small cores connected in a ring, each with a local 2 MB scratchpad instead of a shared cache. Data moves in fixed 128-byte chunks called sticks — 64 fp16 values.
Architecturally, torch-spyre's compiler has two stages. The first lowers a torch.compile'd graph to an intermediate representation called SuperDSC using Inductor extensions kept in the torch-spyre repository. The second stage hands SuperDSC to DeepTools, a proprietary compiler that produces instructions for the Spyre hardware. The post covers only the first stage. Within the standard pipeline — Dynamo traces, AOTAutograd partitions, Inductor lowers the FX graph to LoopLevelIR and generates code — torch-spyre registers Spyre-specific decompositions (for example, mapping layernorm to primitive Spyre operations) and lowerings for those primitives. The bulk of its logic lives in CustomPreSchedulingPasses, where tiling, layout assignment, work division and scratchpad allocation are decided. That pass currently hooks Inductor's GraphLowering.codegen() via a monkey-patch rather than a first-class extension point; Inductor's own LoopLevelIR hook, _pre_fusion_custom_pass, runs too late because it fires only after LoopLevelIR becomes SchedulerNodes, foreclosing large-scale graph rewriting.
The first hardware divergence is sticks. GPUs expose individual-element addressing as the programming abstraction even when a cache line is fetched underneath; Spyre exposes the 128-byte aligned stick as the unit of movement and computation, with no path for moving a single element alone. Inductor's index arithmetic must therefore reason in sticks. Operations also constrain how inputs and outputs assemble into sticks — a pointwise op, for instance, requires a shared innermost stick dimension. When inputs disagree, the compiler inserts a restickify, a data-movement op that reorders elements and reforms sticks, the sole exception to stick immutability. Choosing stick dimensions across a graph to minimize data movement is a global optimization problem, and layout still affects wall-clock time: on a batched matmul with all tensors already resident in device memory, retiling inputs and outputs cut kernel time by 29–69% across batch sizes and shapes.
The second difference is memory hierarchy. Each core has a small fast local scratchpad backed only by global device memory, and nothing moves by itself — every transfer into scratchpad and into the compute engines comes from an explicit compiler-emitted instruction. There is no hardware-managed layer deciding what to keep close. If the compiler does not plan for a value to stay on-chip, it must write it back and re-fetch it later. Spyre workloads are frequently memory-bound rather than compute-bound, so those round trips often dominate runtime. This is a matter of degree relative to GPUs, which also have hierarchies, but with no automatic fallback, planning falls entirely to the compiler.
The third difference is core assignment. Spyre cores form a static dataflow machine with no runtime scheduler balancing work across units, so division of work must be fixed at compile time. The post frames Spyre kernels as naturally persistent: work division is fixed a priori, while each core's scratchpad can retain data across many rounds of computation.
Work division is handled by a three-pass planner. A mandatory pass enforces a hard per-core limit on addressable device memory, committing the minimum splits needed to bring oversized tensors under it. A cost-model pass for matrix multiplications enumerates plausible splits of the operation's three dimensions, estimates runtime for each, and picks the cheapest. A third pass covers everything else. In a worked example — fp16 matmul with a [8192, 32768] input, a [32768, 4096] weight and a [8192, 4096] output — the input's memory span exceeds a single core's limit, so the first pass commits a 2-way split along the reduction dimension, after which the cost pass picks a 16-way split along the output dimension. The planner prioritizes matmuls because they usually dominate end-to-end time.
Splitting work also splits data. A [1024, 2048] fp16 tensor is 4 MB, too large for one core's scratchpad, but row-split across four cores each slice is 1 MB. Scratchpad residency depends on every operation touching a buffer agreeing on that split, which nothing else guarantees.
The second theme, scratchpad planning, follows: intermediate tensors too large for a core either round-trip through device memory or are handled by tiling a sequence of ops and wrapping it in loops over tiles. The post notes that whether tiling belongs in PyTorch or in specialized languages such as Triton and Helion is an open question, and that torch.scan is expressive enough to encode tiled loops, if awkwardly. Torch-spyre builds for_each_tile on top of it, letting programmers control iteration count versus memory footprint, for example by tiling the reduction dimension of a split-K matmul. Once tiles are assigned to cores, the ScratchpadAllocator decides what stays resident and at what address, offering a choice of solvers from greedy up to CP-SAT and simulated annealing. The measured payoff: a single-core fp16 softmax lowered as max, sub, exp, sum and div drops from 32.51 µs to 23.67 µs median runtime, a 27% reduction, purely from on-chip residency decisions.
A subtler issue is that work division and scratchpad allocation can conflict. Torch-spyre's first implementation ran them separately: work division chose each op's core split independently, and allocation placed buffers afterward. When adjacent ops sharing a buffer pick different splits — a matmul optimizing compute-engine occupancy versus a neighboring reduction matching its own shape — the producer and consumer slice the buffer differently, a work-division mismatch. Historically the fix was to disqualify the buffer from scratchpad residency, forcing device-memory round trips in both directions. An interim approach derived work division for intermediate ops from the preceding matmul while still falling back to device memory when mismatches could not be avoided.
Spyre hardware supports core-to-core relayout, a shuffle between scratchpad slices that never touches device memory, now being enabled under the name lx_relayout. It is not free: the shuffle costs time and scratchpad space because both layouts coexist during the move, but it converts an all-or-nothing spill into a priced on-chip choice. Torch-spyre's newer co-optimizing allocator instead folds the standalone work-division pass into scratchpad planning and weighs shuffle cost from mismatched-but-optimal divisions against aligned-but-suboptimal ones. Two solver backends explore this: a pruned greedy path that narrows candidates with hardware heuristics and best-fit buffer packing, and an exhaustive OR-Tools CP-SAT path that considers candidate divisions and relayout choices together under hard constraints — buffer divisions matching every consumer directly or via a paid relayout, and everything fitting in scratchpad including both copies during a live relayout.
Why it matters: teams running enterprise inference on non-GPU silicon now have a working example of building a vendor backend by extending Inductor rather than starting from scratch, and IBM's numbers — a 7x smaller compiler, measured 27% to 69% wins from layout and residency decisions — indicate where the engineering effort actually goes. The upstream discussion of formalizing the pre-scheduling hook matters because torch-spyre presently relies on a monkey-patch; if that becomes a supported extension point, other accelerator vendors could avoid the same plumbing. In my reading, the harder lesson is architectural: on a dataflow part with explicit scratchpads and no runtime scheduler, tiling, layout, work division and allocation are one coupled global optimization, not a sequence of local choices.
Based on reporting from the original publisher. Visit the source for full context and later updates.
Publisher excerpt
TL;DR The PyTorch-based software stack for IBM’s Spyre accelerator recently moved its torch.compile backend from sendnn, a separate compilation stack built entirely from scratch, to backend=inductor, an open-source backend built... The post Extending Inductor for the IBM Spyre Accelerator appeared first on PyTorch .