AivexaNewsSearch
AI news for builders and product teamsChecked every hour

Google Adds Cycle-Level Kernel Profiling to XProf

Collected Sep 30, 2026

Google has added a Kernel Profiling suite to XProf, its open-source profiler for TPU workloads, according to InfoQ. The addition lets developers see cycle-level details in custom Pallas kernels, which previously appeared as single opaque blocks in trace captures.

On TPU v7 (Ironwood), XProf samples hardware performance counters during runtime. In a tiled matmul example, Google found a memory stall and cut kernel time from 125.5 microseconds to 88 microseconds, about 30%, by adding triple buffering. The figure comes from a single Google-authored demo kernel, not a broad benchmark, so the post frames it as showing the workflow rather than gains to expect elsewhere.

According to Yogesh SY of Google's AI Infra team, the gap comes from how profilers treat custom compilation paths. Kernels made with Pallas, Mosaic, or Triton skip standard XLA passes, which can distort compile-time static cost models. Metrics such as "optimal FLOPs" and goodput efficiency may be inaccurate or not work at all. A static tool can flag an MXU instruction block as fully utilized while the unit sits idle waiting on HBM, because static analysis ignores time.

The suite works at three levels. Developers pass the flags --xla_enable_custom_call_region_trace=true and --xla_xprof_register_llo_debug_info=true for compiler inspection. The Graph Viewer then displays a "Custom Call Text" panel showing the lowered MLIR for each custom call. The Trace Viewer shows Low-Level Operations bundle data for static execution analysis, including machine instructions per clock cycle, with time-aligned tracks for the MXU, scalar and vector ALUs, vector fills, loads, spills, stores, and the cross-lane unit (XLU).

For runtime telemetry, XProf samples hardware counters periodically, with a 1 microsecond resolution floor driven by a host-level timer. A new external event-triggered mode removes that floor, capturing TPU trace instructions and boundary triggers including the entry and exit of custom call scopes, enabling sub-microsecond capture and improved attribution. Developers can configure up to 28 counters per core across up to four SparseCores, a 4x28 matrix. Collection is enabled through jax.profiler.ProfileOptions using tpu_enable_periodic_counter_sampling and tpu_tc_perf_counter_sampling_options with is_external_trigger:true; for periodic mode, interval_us replaces the trigger flag.

A new Perf Counters View lists over 16,000 raw counters in tabular form. Trace track height reflects the maximum raw counter value in an interval, not a normalized percentage: an increment of 100 cycles over a 500ns window on a roughly 2.0 GHz core equals 10% utilization of that unit. The post sets out a "hierarchy of trust" for metrics, counting values read directly from hardware registers, such as HBM utilization and TPO metrics, as ground truth for custom kernels, while XLA cost-model estimates need caution.

XProf is part of the OpenXLA project and integrates with JAX profiling. The post does not state a release status or version for the Kernel Profiling suite. Counter sampling is documented for TPU v7 (Ironwood), so teams on earlier TPU generations should not assume the same coverage. The limited 4x28 counter budget requires teams to pick counters for each investigation, and engineers should stop treating XLA-derived efficiency numbers as reliable for custom kernels.

Read at InfoQ · AI, ML & Data Engineering

Based on reporting from the original publisher. Visit the source for full context and later updates.

Publisher excerpt

Google has added a Kernel Profiling suite to XProf. This is its open-source profiler for TPU workloads. Now, developers can see cycle-level details in custom Pallas kernels. Before, these kernels appeared as single opaque blocks in trace captures. By Claudio Masolo