Speed Up Image Generation with FLUX 3 Image Klein on NVIDIA Blackwell GPUs

NVIDIA has published deployment guidance for running FLUX 3 Image Klein, an image generation and editing model from Black Forest Labs, through TensorRT LLM VisualGen, a runtime dedicated to visual generation inside TensorRT LLM. The tutorial reports benchmarks on NVIDIA Blackwell hardware, including single-GPU and eight-GPU tests on B200 systems and a run on the NVIDIA RTX PRO 6000 Blackwell Server Edition. FLUX 3 Image Klein ships in a Base checkpoint and a distilled variant, aimed at different points on the latency-versus-fidelity curve, and NVIDIA says the model was trained on GB200 NVL72 systems.
Architecturally, Klein stitches together three components: Qwen3-VL-4B-Instruct for turning text into conditioning signals, a diffusion transformer that refines image latents, and an autoencoder that decodes latents into pixels and encodes reference images for editing. Text, a noisy target image and optional reference images are processed as separate streams and then merged through joint attention. That matters for benchmarking, because text-to-image and image-to-image workloads stress the stack differently, and the reported numbers treat them separately throughout.
On a single B200 at 2048x2048, NVFP4 weights alone delivered up to 1.62x faster text-to-image and 1.58x faster image-to-image generation than BF16. NVFP4 is a four-bit floating-point format with a small shared scaling block, which shrinks both weight memory and matrix-multiply cost; VisualGen accepts Black Forest Labs' prequantized NVFP4 diffusion transformer checkpoints by setting variant to nvfp4 in the pipeline configuration. With NVFP4 as the baseline, NVIDIA focused on denoising, which dominates generation time, and reported that BF16 vanilla attention consumed roughly 60 percent of diffusion transformer GPU kernel time at 2048x2048.
Three families of optimization follow. First, general matrix multiply kernels: VisualGen uses FlashInfer's PrimsTS FP8/NVFP4 kernels, tuned for FLUX 3 tensor shapes and enabled by default, with fused epilogues such as GEMM plus SwiGLU and fused query/key normalization and rotary position embeddings on the QKV projection. NVIDIA reports these pinned kernels were 1.02-1.10x faster for text-to-image on B200 than an autotuned TensorRT LLM NVFP4 GEMM implementation.
Second, attention. Two quantized paths are offered: cuDNN FP8 achieved 1.08x (Base) and 1.09x (distilled) speedups over BF16 attention, with LPIPS scores of 0.072 and 0.161; VC-Attention FP8, a TensorRT LLM variant of the VC-Attention method with a repair budget of 0.005 that re-corrects the tokens with the largest quantization error, reached 1.18x and 1.16x with LPIPS of 0.086 and 0.151. LPIPS here measures perceptual distance from the dense baseline output, so lower is closer. NVIDIA suggests VC-Attention FP8 for 2048x2048 quantized attention on B200. Separately, Sol-Attention introduces block sparsity controlled by tau; with tau equal to 1.0 it gave 1.24x faster denoising plus decoding for Base with 16 initial dense steps, or 1.30x with 12, and NVIDIA recommends 16 steps for fidelity or 12 for speed, noting that raising tau hurt quality with little extra speed.
Third, step caching. Cache-DiT reuses intermediate transformer outputs across denoising steps without retraining or changing weights, with a residual-difference threshold of 0.24 deciding when reuse is safe. It nearly halved Base latency at both 1024x1024 (1.97x) and 2048x2048 (1.94x), with LPIPS of 0.038 and 0.033 respectively, and helped the distilled model less (1.26x and 1.20x).
Combining compilation, NVFP4 with PrimsTS kernels, Sol-Attention and Cache-DiT cut 2048x2048 latency on one B200 from 22.36 to 6.46 seconds for text-to-image (3.46x) and from 54.64 to 12.28 seconds for image-to-image (4.45x). The 16-step distilled model reached up to 2.34x end to end. On RTX PRO 6000 Blackwell Server Edition, compilation plus NVFP4 with fused CuTe DSL kernels plus Cache-DiT produced 1.73-3.25x speedups.
Scaling follows the model's structure. Base uses classifier-free guidance, so VisualGen assigns the positive and negative branches to separate GPU groups (cfg_size 2) and splits each branch's sequence with Ulysses parallelism (ulysses_size 4); the distilled model drops guidance and uses Ulysses across all eight GPUs. On a DGX B200 at 2048x2048, median text-to-image latency fell from 14.90 to 3.00 seconds for Base and from 2.73 to 1.08 seconds for distilled. Denoising itself sped up 6.01x and 4.44x, but roughly half of the distilled model's eight-GPU time sits outside denoising, which caps end-to-end gains. At 1024x1024, adding CUDA Graphs on eight B200s reduced Base latency from 1.823 to 0.809 seconds (2.25x) and distilled from 0.791 to 0.366 seconds (2.16x), since per-GPU work shrinks and kernel-launch overhead becomes visible.
Practically, NVIDIA supplies a prebuilt container image, an offline example script that takes model, YAML config, prompt, size, seed and output path, and an OpenAI-compatible /v1/images/generations endpoint launched with trtllm-serve. Variants select nvfp4, row-wise fp8r, or default BF16; editing accepts reference images via repeated --image flags. Sample YAMLs wire together pipeline, compilation, attention and cache settings. The company notes these examples do not expand prompts, so any prompt upsampler must run beforehand, and reported latencies exclude upsampling. One operational detail worth flagging: the PrimsTS attention kernels that VC-Attention depends on are enabled with the TLLM_FMHA_LIBS=+prims_ts environment variable, so a missing variable would quietly change which optimizations are available. My reading is that the stacking behavior matters most for capacity planning, since caching and sparsity trade small perceptual drift for large latency cuts while multi-GPU scaling depends on whether guidance is in play.
Why it matters: Teams embedding diffusion models in editors or design tools can now see, in concrete seconds and perceptual scores, what each optimization buys before committing GPU budget. The results are Blackwell-specific and measured against a dense BF16 reference rather than human preference, so product teams should treat the LPIPS figures as a tuning dial rather than a quality verdict. Developers porting Klein will also need to decide up front between Base with guidance and the distilled model, because that choice changes the parallelism layout as well as the latency profile.
Based on reporting from the original publisher. Visit the source for full context and later updates.
Publisher excerpt
When deploying diffusion models in design tools, image editors, and visual production workflows, developers must account for inference latency. Generating an...