Accelerating Reinforcement Learning Weight Refit with the Reshard Collective

NVIDIA has introduced the Reshard collective, a new class of collective communication aimed at reinforcement learning weight refit, with an open-source reference implementation called NCCL M2N Reshard published in the NVIDIA/nccl-extensions GitHub repository. A reference integration also ships in the open-source NeMo RL framework. The work targets a specific bottleneck in disaggregated RL post-training, where inference and training run on separate GPU pools.
In that setup, training trajectories are generated on the fly by running inference with the current policy model. The policy is updated only on the training GPUs, so the inference side must periodically receive fresh weights. That transfer is the refit, and it has to happen at every training step. Two properties make it hard. Inference and training have different compute profiles, so the same weights are sharded differently on each side across data, context, tensor, expert and pipeline parallelism. And because every weight must be refreshed, the payload scales with model size.
Under BF16, Kimi K3 at 2.8 trillion parameters is roughly 5.6 TB of weights, and Qwen 3.8 2.4T A95B at 2.4 trillion parameters is about 4.8 TB. Conventional refit runs two stages: training GPUs all-gather their local shards into the full parameter, the full tensor is broadcast to inference GPUs, and each inference GPU keeps only its needed shard. Since both stages move roughly the model's full weight size, the logical payload is about double the weights, putting today's largest open-weight models near 10 TB per refit.
Reshard reframes refit as a DTensor-to-DTensor transfer. PyTorch's DTensor describes a tensor by its global shape, a device mesh of GPUs and placements saying how it is sharded or replicated. NVIDIA's insight is that refit is simply resharding a DTensor from the training layout into the inference layout, whatever parallelism configurations each side uses. Instead of building the whole tensor, each destination shard is assembled from the regions of source shards that actually overlap it. A worked example: an 8x8 matrix held row-sharded across two training GPUs (4x8 each) and needed column-sharded across four inference GPUs (8x2 each) is satisfied by each destination pulling a 4x2 region from each source. Only the theoretical minimum data crosses the network.
A Reshard call is specified just by the source and destination DTensor descriptions, so applications invoke one operation the way they would call all-reduce or all-gather, and the algorithm can improve without application changes. NCCL M2N Reshard handles the M-source-to-N-destination case, with training GPUs as the source group and inference GPUs as the destination group.
The implementation derives exact intersections between source and destination shards, then maps transfers onto the physical network. Replicas are grouped by NVLink domain and incoming contributions are spread as evenly as possible across the ranks in each domain, so each rank receives a distinct subset over the network. Ranks then swap subsets over NVLink to assemble complete shards, and the exchange pipelines with network transfer to overlap latency. Where replicas span multiple domains, one copy of each contribution is forwarded between them and the balanced local exchange repeats inside each domain. The contrast with direct point-to-point sends is clear in NVIDIA's TP4-to-TP2-by-DP2 example: naive sends push each shard to both data-parallel replicas, halving effective bandwidth per shard, while Reshard moves each shard once and fills the gap locally.
In NeMo RL, NVIDIA adopted a hybrid design. Feed-forward projection weights dominate the payload, around 95% of parameters in current LLMs, and those take the Reshard path; smaller weights such as QKV projections or mamba weights stay on gather-and-broadcast, which is broadly compatible. This keeps most of the savings without the edge-case coverage burden of resharding every layer type. The framework only needs a DTensor description per parameter for each side.
Benchmarks on NVIDIA DGX H100 (NVL8) and DGX GB200 (NVL72) systems across models including Qwen 3 235B and DeepSeek v3 671B show 2.7x to 5.7x speedups over conventional refit. In a refit-only microbenchmark, Reshard refit for Qwen3.8 2.4T BF16 took 4.3 seconds. Reshard is enabled with refit_transport: nccl_reshard under policy.generation.colocated, with colocated disabled, and NVIDIA points to its NeMo RL refit guide and NCCL Reshard configuration requirements for supported backends, layouts and precision modes.
My reading is that the DTensor abstraction is the more durable contribution. Once refit is expressed as layout-to-layout resharding, new parallelism schemes on either side become description changes rather than new communication code. The NVLink-domain grouping is a pragmatic dependency: the savings assume destination replicas share high-bandwidth local links, which is typical in a rack-scale NVL72 deployment but a weaker fit for scattered inference fleets. There is also an operational caveat around elasticity: sharding layouts can change at runtime as nodes join or leave during failure recovery, and NeMo RL currently destroys and recreates the communication group, with support for NCCL shrink and grow APIs planned for later.
Why it matters: RL post-training teams running separated inference and training clusters are the immediate audience, since refit overhead sits directly on the training step and grows with model size. The results suggest that for trillion-parameter models the transfer can drop from a multi-second-per-step tax toward single-digit seconds in the measured microbenchmark, and because the collective is open source, other frameworks can adopt the same pattern rather than reuse NVIDIA's code path. Inference infrastructure that lacks fast intra-domain links may see smaller gains than the reported figures.
Based on reporting from the original publisher. Visit the source for full context and later updates.
Publisher excerpt
During reinforcement learning (RL) post-training, training trajectories are generated on the fly by running inference with the current policy model. For...