AivexaNewsSearch
AI news for builders and product teamsChecked every hour

Deploying an HSTU Generative Recommender with NVIDIA Dynamo-Triton

Collected Sep 30, 2026

NVIDIA has published an end-to-end inference workflow for Hierarchical Sequential Transduction Unit (HSTU) generative recommender models, served through NVIDIA Dynamo-Triton (formerly Triton Inference Server) and available via the NVIDIA recsys-examples repository. The workflow combines HSTU models, PyTorch Ahead-of-Time Inductor (AOTI) compilation, FlexKV-backed KV caching, native C++ validation, an NV embedding cache, and Dynamo-Triton deployment.

The described pipeline starts from a PyTorch model exported with torch.export and AOTI, which compiles it into a package loadable by a native C++ runtime. Dynamo-Triton serves that package using its PyTorch backend with platform "torch_aoti". The workflow has five stages: building custom operators and runtime libraries, exporting the HSTU ranking model with PyTorch AOTI, starting the FlexKV-backed KV-cache service, validating artifacts with native C++ replay, and serving the KV-cache AOTI model with Dynamo-Triton. A KVCacheManager uses GPU memory and host storage; GPU cache is a paged table supporting lookup, allocation, append, and eviction, with an LRU-style policy for older users.

The benchmark uses the KuaiRand-1K ranking configuration on a single GPU with three-layer and eight-layer HSTU variants, hidden size 512, four attention heads, BF16 weights and KV cache, a maximum history sequence length of 8,192 (4,096 item plus action pairs), maximum candidate sequence length of 100, and six contextual features, at an effective sequence length of 8,298 tokens before alignment and 8,320 aligned.

On an NVIDIA RTX PRO 6000 Blackwell Workstation Edition GPU, Dynamo-Triton with PyTorch AOTI achieved best-case speedups of up to 4.47x for the three-layer model and 5.93x for the eight-layer model at dynamic batch size 8 under a 100% GPU KV-cache hit rate, relative to the same AOTI configuration without KV caching. At batch size 8, the three-layer model reached 0.423 ms latency per logical request with GPU KV-cache hits, and the eight-layer model reached 0.678 ms.

Read at NVIDIA Developer Blog

Based on reporting from the original publisher. Visit the source for full context and later updates.

Publisher excerpt

Generative recommender (GR) systems are emerging as a powerful new approach for large-scale personalization. Instead of treating recommendation as a set of...