AivexaNewsSearch
AI news for builders and product teamsChecked every hour

Run Native DLA Inference in Python on NVIDIA Jetson AGX Orin Using cuDLA

Collected Nov 13, 2026

NVIDIA has published a developer walkthrough for running native Deep Learning Accelerator (DLA) inference from Python, enabled by a new cuda.bindings.cudla module introduced in cuda-bindings version 13.3.0 within the CUDA Python project. The guide targets Jetson AGX Orin running JetPack, where DLA is fixed-function hardware that executes common network operations such as convolutions and pooling while leaving GPU capacity free for other work. The same Python API, NVIDIA says, applies to RTX Spark, bringing DLA inference to Windows AI developer laptops.

The bindings are a thin, Cython-generated layer over the cuDLA C++ API, intended for hybrid-mode workflows in which CUDA streams coordinate DLA execution with GPU preprocessing and postprocessing. The module exposes thirteen functions, nine data types and eight enums. On Jetson, the underlying libcudla.so runtime ships with JetPack, so no separate cuDLA SDK or extra runtime libraries are needed. Installation is a pip install of cuda-bindings at 13.3.0 or later. Because JetPack's default Python may be older than the required 3.10 minimum, the post shows installing Python 3.12 from the deadsnakes PPA and building a virtual environment.

TensorRT is needed only to compile a trained model into a DLA loadable, the binary artifact DLA hardware executes. The example uses trtexec targeting TensorRT 11.4.1 with CUDA 13.4 Update 1 on JetPack 7.2.2 or later, producing an FP16 CHW16 loadable from an ONNX model with fixed input dimensions. Key flags are --useDLACore=0 to target the first DLA engine on Orin and --buildDLAStandalone to produce a raw loadable that does not need TensorRT at inference time. In TensorRT 11 precision is derived from the model's tensor types, so the older --fp16 and --int8 flags are no longer supported. NVIDIA notes that not every layer is DLA-compatible and that trimming incompatible head and tail layers from the ONNX graph is often necessary.

The runtime lifecycle spans nine steps. First, initialize CUDA and create a DLA device with cudla.Mode.CUDA_DLA, which selects hybrid mode so tasks are submitted through a CUDA stream. Standalone mode is not supported in this release and raises an unsupported-operation error. Second, load the loadable as a Python bytes object through module_load_from_memory, with no file path required. Third, query the module for input and output tensor counts and descriptors; each descriptor carries buffer size, shape (n, c, h, w), name, data format and stride, which lets allocation follow the model rather than hardcoded sizes.

The remaining steps: allocate GPU memory with CUDA runtime calls sized from each descriptor; register those buffers with cuDLA via mem_register, which returns a DLA-side handle the hardware uses; preprocess input into a NumPy array matching the descriptor layout and copy it asynchronously to the device on a non-blocking stream. Then build a cudla.Task, assigning Python lists to input_tensor and output_tensor, and call submit_task to enqueue the work on the stream. A stream synchronize waits for completion, after which the output is copied back to a NumPy array and a second synchronize confirms the transfer. Cleanup unregisters memory, unloads the module, destroys the device and frees allocations and the stream.

Error handling is idiomatic Python: all cuDLA calls raise cudla.CudlaError rather than returning codes, and the exception carries both a numeric status and a readable name, so code can branch on cudla.Status values ranging from invalid parameters to engine timeouts. NVIDIA recommends try/finally for cleanup because device handles are opaque C resources that will not release on garbage collection.

After synchronization, the output is a flat NumPy array of raw DLA bytes to reshape and interpret using the output descriptor's fields, then pass to NumPy, PyTorch or OpenCV in the same process. In the ResNet-50 example, the loadable reports one input (shape 1,64,56,56) and one output (shape 1,2048,7,7), with tensor names surfaced at load time. The workflow is identical across architectures; only tensor names, sizes and postprocessing change. Unit tests for enums, POD types and the API surface live under cuda_bindings/tests/cudla/ in the NVIDIA/cuda-python repo and run without DLA hardware.

This suggests the practical appeal is orchestration simplicity: preprocessing kernels, DLA inference and postprocessing can be enqueued in order on a single stream, so developers avoid a C++ runtime layer while still executing on dedicated silicon. The trade-offs are the fixed-function layer coverage and the one-time compilation step, plus the cleanup discipline required for opaque handles.

Why it matters: Developers building edge AI pipelines on Jetson can now manage DLA inference from Python, lowering the barrier to offloading supported layers from the GPU for power efficiency. Teams already using CUDA Python can add accelerator execution without new runtime dependencies beyond cuda-bindings, and the identical API on RTX Spark suggests, as an inference, a more portable path between embedded and laptop hardware. Multi-network applications with tight power budgets are the clearest beneficiaries.

Read at NVIDIA Developer Blog

Based on reporting from the original publisher. Visit the source for full context and later updates.

Publisher excerpt

AI applications at the edge often need to run multiple neural networks within limited compute and power budgets. NVIDIA Deep Learning Accelerator (DLA)...