NeMo-DCR: Nvidia Cuts a 1T-Parameter RL Weight Sync From 87 Minutes to 150 Seconds
Agentic RL at frontier scale has a plumbing problem nobody outside the labs talks about. Training and rollout run on different clusters, so after every policy update the new weights have to reach the rollout fleet before the next batch can start. Nvidia's NeMo-DCR paper (arXiv 2610.08430, HuggingFace 10 upvotes) measures the cost: moving a full 1T checkpoint between two AWS regions takes 87.5 minutes. At that rate the GPUs on the rollout side spend most of their life waiting for a file.
The observation that makes the fix possible is that BF16 training changes the stored value of only about 1% of weights per step. Prior systems exploited that sparsity but got placement wrong, rebuilt values arithmetically so the receiver's bits drifted from the trainer's, used cross-cluster collectives, or could not recover from a failure mid-transfer. NeMo-DCR sends only the changes and is bit-exact: the receiver ends up with the same parameter and buffer bits as a dense refit. Fixed affine mappings project changes from training shards into the checkpoint's canonical coordinates, compressible XOR masks carry the changes that survive that projection, overwrites carry the rest, retries fix partial writes, and a joint commit pins the policy to its baseline so the next delta has a known starting point. Payloads stream through object storage or a relay tree, no collective required.
Results: at 3% and 5% change rates, refits of 30B to 1T models run 12 to 40 times faster than a transport-only full-checkpoint reference. A 1T relay-tree refit at 3% takes 150 seconds. That turns cross-region agentic RL from a batch-per-hour regime into something you can iterate on.
Why file this under agents rather than infra: every RL-for-agents result from the past month assumes rollouts happen in sandboxes somewhere else, and the step count you can afford is set by how fast the policy gets there. Bit-exactness also matters for reproducibility. If the rollout weights are not identical to the trained weights, the trajectories you learn from are off-policy by an amount nobody is measuring.
Link: arxiv.org/abs/2610.08430
← Back to all articles
The observation that makes the fix possible is that BF16 training changes the stored value of only about 1% of weights per step. Prior systems exploited that sparsity but got placement wrong, rebuilt values arithmetically so the receiver's bits drifted from the trainer's, used cross-cluster collectives, or could not recover from a failure mid-transfer. NeMo-DCR sends only the changes and is bit-exact: the receiver ends up with the same parameter and buffer bits as a dense refit. Fixed affine mappings project changes from training shards into the checkpoint's canonical coordinates, compressible XOR masks carry the changes that survive that projection, overwrites carry the rest, retries fix partial writes, and a joint commit pins the policy to its baseline so the next delta has a known starting point. Payloads stream through object storage or a relay tree, no collective required.
Results: at 3% and 5% change rates, refits of 30B to 1T models run 12 to 40 times faster than a transport-only full-checkpoint reference. A 1T relay-tree refit at 3% takes 150 seconds. That turns cross-region agentic RL from a batch-per-hour regime into something you can iterate on.
Why file this under agents rather than infra: every RL-for-agents result from the past month assumes rollouts happen in sandboxes somewhere else, and the step count you can afford is set by how fast the policy gets there. Bit-exactness also matters for reproducibility. If the rollout weights are not identical to the trained weights, the trajectories you learn from are off-policy by an amount nobody is measuring.
Link: arxiv.org/abs/2610.08430
Comments