Rename the repo and the coding agents get worse
SWE-bench draws its tasks from the most famous Python repositories on earth — django, sympy, scikit-learn, the ones whose entire git history is in every pretraining corpus multiple times over. So when an agent resolves an issue there, how much of that is engineering skill and how much is knowing where the file lives because it has read this codebase a thousand times?
SchrodingerRepo, out on arXiv as 2609.27891, answers the question by scrubbing the familiarity out. Four levels of transformation, all behavior-preserving: rewrite the problem statement, remap the namespace, shuffle the layout inside files, and rewrite code so it does exactly the same thing in different words. The tests still pass on the original solution. Only the recognizability is gone.
Performance drops across every model tested, consistently, and interaction costs go up substantially. The diagnosis is the good part. The damage doesn't land on reasoning or on writing the patch. It lands on exploration and localization — finding the right file, finding the right function. Which is exactly the skill you'd expect memorization to supply for free and generalization to make you work for.
That reframes what the benchmark number means. A coding agent's SWE-bench score is partly a measure of how well it knows a specific handful of repositories, and your codebase is not one of them. Add this to the list of numbers worth demanding: the transformed-repo score next to the raw one, the way we've started asking for survival rate, all-pass rate and cost per completed task. The gap between the two is the part that transfers to your job.
Paper: https://arxiv.org/abs/2609.27891 — code: https://github.com/cslsolow/Schrodinger-Repo
← Back to all articles
SchrodingerRepo, out on arXiv as 2609.27891, answers the question by scrubbing the familiarity out. Four levels of transformation, all behavior-preserving: rewrite the problem statement, remap the namespace, shuffle the layout inside files, and rewrite code so it does exactly the same thing in different words. The tests still pass on the original solution. Only the recognizability is gone.
Performance drops across every model tested, consistently, and interaction costs go up substantially. The diagnosis is the good part. The damage doesn't land on reasoning or on writing the patch. It lands on exploration and localization — finding the right file, finding the right function. Which is exactly the skill you'd expect memorization to supply for free and generalization to make you work for.
That reframes what the benchmark number means. A coding agent's SWE-bench score is partly a measure of how well it knows a specific handful of repositories, and your codebase is not one of them. Add this to the list of numbers worth demanding: the transformed-repo score next to the raw one, the way we've started asking for survival rate, all-pass rate and cost per completed task. The gap between the two is the part that transfers to your job.
Paper: https://arxiv.org/abs/2609.27891 — code: https://github.com/cslsolow/Schrodinger-Repo
Comments