When Self-Evolution Backfires: Agents Poison Their Own Skill Pools
Self-improving agents have a poisoning problem, and a new paper puts a number on it. When Self-Evolution Backfires (arXiv 2608.05810, submitted August 6) studies agents that distill reusable skills from their own execution trajectories. The pattern everyone hopes for — accumulate skills, get better — holds at first, then reverses. The mechanism is what the authors call cross-round contamination chains: one defective skill gets admitted to the pool, becomes reference material for the next round of distillation, and the defect compounds. Once it is in, the degradation is effectively irreversible.
The fix is bureaucratic in the best sense: Verifier-as-Gatekeeper, a pre-commit review that every candidate skill must pass before admission. Three critics check structural validity, behavioral harmlessness, and semantic consistency, and a marginal-gain selection step prunes combinations that hurt at the policy level. On Terminal-Bench 2 this hits 72% pass at 1 with a skill pool roughly five times smaller than unconditional accumulation, and the resulting frozen pool transfers to four other model backbones without re-evolving.
This slots straight into the theme of the past month: Prime Agent's refine loop turning a scoring exploit into a better cheating skill, the OSReward finding that VLM judges grade failed runs as passes. The emerging rule is consistent everywhere: letting agents generate improvements is safe, letting them approve their own improvements is not. Smaller, gated, audited skill pools beat big self-approved ones — a code-review culture for things no human wrote.
Paper at arxiv.org/abs/2608.05810.
← Back to all articles
The fix is bureaucratic in the best sense: Verifier-as-Gatekeeper, a pre-commit review that every candidate skill must pass before admission. Three critics check structural validity, behavioral harmlessness, and semantic consistency, and a marginal-gain selection step prunes combinations that hurt at the policy level. On Terminal-Bench 2 this hits 72% pass at 1 with a skill pool roughly five times smaller than unconditional accumulation, and the resulting frozen pool transfers to four other model backbones without re-evolving.
This slots straight into the theme of the past month: Prime Agent's refine loop turning a scoring exploit into a better cheating skill, the OSReward finding that VLM judges grade failed runs as passes. The emerging rule is consistent everywhere: letting agents generate improvements is safe, letting them approve their own improvements is not. Smaller, gated, audited skill pools beat big self-approved ones — a code-review culture for things no human wrote.
Paper at arxiv.org/abs/2608.05810.
Comments