A 19-Day Ban Tested Coding Benchmarks. Merge Rates Did Not Move
On a day when Google launched a model on a stack of coding benchmarks, an arXiv paper asked the question launch pages skip: does a higher score get more work merged? Its answer, from a natural experiment nobody planned, is not measurably. https://arxiv.org/abs/2611.23216 (submitted Nov 17, listed Nov 18)
The setup is clever. On June 12, 2026, a US export-control directive cut access to Claude Fable 5 for 19 days. Most users fell back to Opus 4.8, which is 11 points lower on SWE-Bench Pro. Same people, same projects, only the model changed. The authors tracked GitHub pull requests from 297 Fable 5 users before, during and after the cutoff, against 8,908 developers who stayed on Opus 4.8 the whole time, using a difference-in-differences model with developer and date fixed effects.
Fable 5 access raised the 7-day PR merge rate by 1.4 percentage points, with a 95% interval of minus 1.4 to plus 4.3. Not significant. Across five other 2026 model upgrades (1,208 developers, 15,442 PRs) the weighted mean was minus 0.9 points, interval minus 1.9 to plus 0.2, and all five point estimates were negative. PRs merged per week rose 2.4%, also within noise.
Two caveats before anyone throws out their benchmark. Merge rate is a narrow proxy: a better model might make bigger PRs, fewer bad ones, or save time that never shows up as a merge. And the intervals still allow a moderate effect. A neighboring paper, 2611.21784, shows how fragile these verdicts are: whether agent-written code survives depends on measurement choices, and a Devin result holds in 15 of 22 specifications but only 3 when errors are clustered by repo. https://arxiv.org/abs/2611.21784
The counterpoint is real teams. Mastra's open-source Factory opened and merged 238 PRs last week, next to 630 merged by humans (Show HN: https://news.ycombinator.com/item?id=50548660). The lesson is not that models stopped mattering. It is that past a certain level, what decides output is the factory around the model: review, CI, and who gets to merge.
← Back to all articles
The setup is clever. On June 12, 2026, a US export-control directive cut access to Claude Fable 5 for 19 days. Most users fell back to Opus 4.8, which is 11 points lower on SWE-Bench Pro. Same people, same projects, only the model changed. The authors tracked GitHub pull requests from 297 Fable 5 users before, during and after the cutoff, against 8,908 developers who stayed on Opus 4.8 the whole time, using a difference-in-differences model with developer and date fixed effects.
Fable 5 access raised the 7-day PR merge rate by 1.4 percentage points, with a 95% interval of minus 1.4 to plus 4.3. Not significant. Across five other 2026 model upgrades (1,208 developers, 15,442 PRs) the weighted mean was minus 0.9 points, interval minus 1.9 to plus 0.2, and all five point estimates were negative. PRs merged per week rose 2.4%, also within noise.
Two caveats before anyone throws out their benchmark. Merge rate is a narrow proxy: a better model might make bigger PRs, fewer bad ones, or save time that never shows up as a merge. And the intervals still allow a moderate effect. A neighboring paper, 2611.21784, shows how fragile these verdicts are: whether agent-written code survives depends on measurement choices, and a Devin result holds in 15 of 22 specifications but only 3 when errors are clustered by repo. https://arxiv.org/abs/2611.21784
The counterpoint is real teams. Mastra's open-source Factory opened and merged 238 PRs last week, next to 630 merged by humans (Show HN: https://news.ycombinator.com/item?id=50548660). The lesson is not that models stopped mattering. It is that past a certain level, what decides output is the factory around the model: review, CI, and who gets to merge.
Comments