On Real Company Code, the Best Agent Scores 38.8%
A benchmark called Real-SWE went up at https://withspecific.com/real-swe and the headline number is going to annoy a lot of people: the best model on the board resolves 38.8 percent of tasks. That's Fable 5.1. GPT-6 Astra is second at 33.8 percent, Gemini 3.8 Flash third at 31.2 percent. Eight frontier models tested as of September 2026, and not one of them cracks 40.
The reason the number is so much lower than the SWE-Bench figures you've been reading is the corpus. Real-SWE runs on private, real-world enterprise codebases, the kind with billing calculations and customer migrations in them, work the company describes as having business consequences and company-specific complexity. Not public GitHub repos with tidy issues attached. Tasks run in isolated sandboxes in Harbor format, with verifiers injected at grading time and derived from the test suites that already exist in those codebases. And critically, agents run in their native harnesses, which the team argues reflects how enterprise engineers actually work.
The private corpus is the whole design. Public benchmarks have a structural contamination problem that no amount of cleaning fixes, because the repos were in the training data. Code nobody can train on is code nobody has memorized, and the gap between 38.8 and the seventy-plus numbers on public boards is the closest thing anyone has to a measurement of how much of those numbers is recall.
Which makes the timing interesting, because [the SWE-Bench Pro Verified paper](https://clauday.com/article/d619f8b7-7b67-4ac7-9507-66e8828bd6c6) landed two days ago showing that exact family of benchmarks was leaking and that some models perform substantially worse than previously reported. And [Cognition's SWE-2](https://clauday.com/article/6a998ccf-b793-4da4-9f00-97b27af81532) shipped this week with a 92.8 on TB2.1. Both things can be true and probably are. The useful reading is that public benchmark scores and private-codebase scores are now measuring visibly different quantities, and if you're deciding whether to put an agent on your own repo, the 38.8 is the number shaped like your situation.
Caveats, because a leaderboard from a company selling into this market deserves them. The task set isn't public, which is the point and also means nobody can audit the difficulty distribution or reproduce a score. Eight models is a thin board. And native harnesses make the comparison more realistic and less clean at the same time, since you're now benchmarking model and scaffold together. Take the ordering as weaker evidence than the level. The level is the story: on code your model has never seen, with tests that actually matter, the frontier is currently getting a bit over a third of it right.
← Back to all articles
The reason the number is so much lower than the SWE-Bench figures you've been reading is the corpus. Real-SWE runs on private, real-world enterprise codebases, the kind with billing calculations and customer migrations in them, work the company describes as having business consequences and company-specific complexity. Not public GitHub repos with tidy issues attached. Tasks run in isolated sandboxes in Harbor format, with verifiers injected at grading time and derived from the test suites that already exist in those codebases. And critically, agents run in their native harnesses, which the team argues reflects how enterprise engineers actually work.
The private corpus is the whole design. Public benchmarks have a structural contamination problem that no amount of cleaning fixes, because the repos were in the training data. Code nobody can train on is code nobody has memorized, and the gap between 38.8 and the seventy-plus numbers on public boards is the closest thing anyone has to a measurement of how much of those numbers is recall.
Which makes the timing interesting, because [the SWE-Bench Pro Verified paper](https://clauday.com/article/d619f8b7-7b67-4ac7-9507-66e8828bd6c6) landed two days ago showing that exact family of benchmarks was leaking and that some models perform substantially worse than previously reported. And [Cognition's SWE-2](https://clauday.com/article/6a998ccf-b793-4da4-9f00-97b27af81532) shipped this week with a 92.8 on TB2.1. Both things can be true and probably are. The useful reading is that public benchmark scores and private-codebase scores are now measuring visibly different quantities, and if you're deciding whether to put an agent on your own repo, the 38.8 is the number shaped like your situation.
Caveats, because a leaderboard from a company selling into this market deserves them. The task set isn't public, which is the point and also means nobody can audit the difficulty distribution or reproduce a score. Eight models is a thin board. And native harnesses make the comparison more realistic and less clean at the same time, since you're now benchmarking model and scaffold together. Take the ordering as weaker evidence than the level. The level is the story: on code your model has never seen, with tests that actually matter, the frontier is currently getting a bit over a third of it right.
Comments