Profiling TorchNeuron (Part 1): System-level profiling on AWS Trainium.

Hugging Face published the first part of a series on profiling TorchNeuron, the open source native PyTorch backend for AWS Trainium, an AI chip built for deep learning training and inference. The post is a hands-on walkthrough aimed at readers with no prior profiling experience, and covers how to measure inference performance systematically and find where time is spent. TorchNeuron supports standard PyTorch workflows on Trainium, including eager execution and torch.compile, with minimal code changes, and it integrates with the Neuron Kernel Interface (NKI) for custom operators.
The walkthrough covers four things: configuring torch.profiler with the Neuron backend to capture a clean trace, reading the key_averages() table to see which operations consume Neuron device time and how their Self CPU time compares, following a single operation through Neuron Explorer from framework dispatch to runtime scheduling to execution on a NeuronCore, and watching those signals change across three implementations of the same operation.
The Neuron backend plugs into PyTorch through the PrivateUse1 extension point, which lets an accelerator integrate without a dedicated dispatch key such as cuda. The torch_neuronx package registers Neuron as a native backend against upstream PyTorch.
Profiling uses two objects. NeuronConfig sets which Neuron data streams and NeuronCores to capture and where output is written, and is passed to torch.profiler.profile as experimental_config. NeuronProfiler wraps that config and supplies an export_trace callback that writes the framework trace, trace.json, alongside the Neuron system traces. In the example, the modes list selects the DEVICE and RUNTIME streams, capture_enabled_for_nc picks core 0, and correlate_device_time=True pulls device time into the key_averages() table so CPU and device time can be compared side by side.
Neuron Explorer is a browser-based viewer for the system trace, device timeline and per-engine breakdown, reached through an SSH tunnel or the Visual Studio Code extension. Output files pair a compiled Neuron executable (NEFF) with its hardware instruction trace (NTFF), matched by a shared numeric hash. A system or combined profile is opened by uploading the whole output directory; a device-only profile takes the .neff and .ntff files individually.
The test case is RMSNorm, chosen for its short, clear sequence of arithmetic and reduction steps. Three versions are profiled with identical inputs, profiler configuration and schedule: a naive version built from individual PyTorch operations (multiply, mean, add, reciprocal square root, multiply), the built-in torch.rms_norm() lowered to a single fused Neuron kernel, and the naive function compiled with torch.compile. Scripts to reproduce everything are in the torch-neuronx examples directory.
For the profiler call, two arguments matter on Neuron: include ProfilerActivity.PrivateUse1 to collect device activity, and pass NeuronConfig as experimental_config. The schedule uses wait=2, warmup=1 and active=1 across four prof.step() calls, so one warm iteration is recorded without startup overhead. Calling torch.neuron.synchronize() inside the loop is necessary, because otherwise the export can happen before the device finishes and the Neuron time column comes back empty.
In the naive run, sorting by self_privateuse1_time_total puts the three aten::mul calls at the top, since the function multiplies three times: x * x, then by the reciprocal square root, then by the weight. aten::mean reduces the last dimension, aten::add applies epsilon and triggers a host-to-device copy because the scalar starts on the CPU, and aten::rsqrt computes the reciprocal square root. Total Neuron device time for this version is 2.355 ms.
In Explorer, time runs left to right and each operation's lifecycle runs top to bottom: creation in the framework lane, scheduling in the runtime lane as nrta_execute_schedule, and execution in the hardware lane as nc_exec_running. A dependency chain viewer follows one op, such as aten::mul.Tensor, across all three lanes.
Swapping in torch.rms_norm(t, [512], weight) collapses the work into a single fused device operation, with total Neuron time of 403.3 μs. Compiling the naive function with @torch.compile(backend='neuron', fullgraph=True) produces one Torch-Compiled Region at 375.5 μs. TorchDynamo captures the Python bytecode as an FX graph, AOTAutograd lowers it to ATen operations, and the Neuron backend receives the whole graph, fuses it, optimizes memory access and compiles one NEFF that runs as a single hardware dispatch.
Both fused paths land at roughly one-sixth of the naive version's device time, and the compiled graph is comparable to the hand-fused built-in. The difference is where fusion comes from: a purpose-built operator versus the compiler discovering it from ordinary Python. This suggests the main gains come from removing dispatch gaps between kernels and avoiding materializing intermediates such as x_sq and mean_sq between separately dispatched operations, rather than from the arithmetic itself. A likely trade-off is that fusion depends on the compiler recognizing the pattern, which is why the post uses fullgraph=True; code that breaks the graph would lose that advantage.
Part 2 will move to device-level profiling: opening the NEFF, inspecting the TensorEngine, VectorEngine and DMA breakdown, and placing the kernel on the roofline to decide whether it is compute-bound or memory-bound.
Why it matters: teams running inference on Trainium now have a concrete recipe, key_averages() for numbers and Explorer for the visual trace, for spotting hotspots and testing whether fusion or torch.compile reduces them. The same loop, find a hotspot, verify it in a trace, try a fused op or the compiler, should work on larger models, so the practical skill here is diagnosing slowness rather than tuning a single normalization kernel.
Based on reporting from the original publisher. Visit the source for full context and later updates.