SPADE: The Agent Builds Its Own Gym
Agent RL has a dirty secret: the scarce resource is not compute or algorithms, it is environments. Every team hand-builds training tasks, agents overfit them, and the fixed difficulty stops teaching anything once the model improves. SPADE, from a group including Luke Zettlemoyer and Yejin Choi, makes the model solve its own supply problem: a single LLM plays two roles, an Environment Designer that writes executable Gym-style training environments as code, and a Reasoning Agent that trains in them.
The clever part is how the Designer knows what to build. It optimizes a regret signal, the performance gap between the agent solving a task with hints versus without. A task the agent aces either way teaches nothing; one it fails even with hints is noise; the gap in between marks the frontier of learnable difficulty, and the Designer learns to generate exactly there. The curriculum tracks the agent as it improves, automatically.
Results: +5.3 average across eight held-out benchmarks over fixed-environment baselines, +5.7 on BFCL-v4 multi-turn tool use, +13.9 on ACEBench-Agent, with gains growing with model scale up to the 30B tested. Held-out matters here, environments the model wrote itself transferred to benchmarks it never saw.
Together with Co-RL, same day, same board, this sketches a training stack with no humans in it: Co-RL removes the human from grading by letting a diverse cohort reward itself, SPADE removes the human from curriculum design by letting the model author its own environments. What remains human is deciding what any of it should be for. Code at github.com/spade-rl/spade.
https://arxiv.org/abs/2608.19197
← Back to all articles
The clever part is how the Designer knows what to build. It optimizes a regret signal, the performance gap between the agent solving a task with hints versus without. A task the agent aces either way teaches nothing; one it fails even with hints is noise; the gap in between marks the frontier of learnable difficulty, and the Designer learns to generate exactly there. The curriculum tracks the agent as it improves, automatically.
Results: +5.3 average across eight held-out benchmarks over fixed-environment baselines, +5.7 on BFCL-v4 multi-turn tool use, +13.9 on ACEBench-Agent, with gains growing with model scale up to the 30B tested. Held-out matters here, environments the model wrote itself transferred to benchmarks it never saw.
Together with Co-RL, same day, same board, this sketches a training stack with no humans in it: Co-RL removes the human from grading by letting a diverse cohort reward itself, SPADE removes the human from curriculum design by letting the model author its own environments. What remains human is deciding what any of it should be for. Code at github.com/spade-rl/spade.
https://arxiv.org/abs/2608.19197
Comments