FrontierHarness v1.1: There Is No Best Harness, Only Best Pairs
27,280 runs and 83 billion tokens later, the answer to which coding harness is best turns out to be: depends on the model. FrontierHarness Eval v1.1 from Runta and Baseten crossed 22 harness configurations with five open models (GLM-5.3, GLM-5.3 Flash, DeepSeek V4.1 Flash, Muse Spark open, Kimi K3) on 62 terminal tasks, four attempts each. That is 76 times the size of v1.0. The blog carries no date; it went to Hacker News on Nov 17 and drew 297 points. Post: https://runta.com/blog/introducing-frontierharness-eval-v1-1/ , dashboard: https://frontierharness.org/
The headline is that every model has a different best-value partner. Codex CLI with GLM-5.3. Pi with GLM Flash. ZCode with DeepSeek. Oh My Pi with Muse. Alma with Kimi K3. Codex CLI ranks first on cost with GLM-5.3 and 16th with Kimi K3, and the median harness moved 11 of 22 places in the cost ranking from one model to the next. OpenCode v2 was cheapest at $0.561 a task, ZCode had the highest pass rate at 61.9%, and some harnesses spent up to 1.6 times as much to land between 50.6% and 60.2%. The team also flagged 216 trials as reward hacking, 163 of them online lookups, and scored them as failures.
One caveat keeps this honest. Model choice still explains 92.8% of cost variance, mostly because list prices differ 20 times over. And only one pairing is robust: Codex CLI with GLM-5.3 stays first in 99.6% of bootstrap redraws, while the other winners hold in 42% to 53%. Runta sells runtimes, but the method is published, and it is open models only, no closed frontier.
Two papers from Tuesday's arXiv batch point the same way. Harness Leverage Across Model Size (https://arxiv.org/abs/2611.21407) ran five shipped harnesses with local models from 0.75B to 4.7B on 18 tasks, and the harness that controls completion had the best mean at every size. Anatomy of an Agent Harness (https://arxiv.org/abs/2611.19960) writes harness design down as 30 testable decision cards in six groups, so a harness can be checked instead of trusted.
On a day when three labs advertised cost per task, this is the footnote they skipped: the cost per task you pay depends on the harness you pair the model with. Swapping the model without re-picking the harness is leaving money on the table.
← Back to all articles
The headline is that every model has a different best-value partner. Codex CLI with GLM-5.3. Pi with GLM Flash. ZCode with DeepSeek. Oh My Pi with Muse. Alma with Kimi K3. Codex CLI ranks first on cost with GLM-5.3 and 16th with Kimi K3, and the median harness moved 11 of 22 places in the cost ranking from one model to the next. OpenCode v2 was cheapest at $0.561 a task, ZCode had the highest pass rate at 61.9%, and some harnesses spent up to 1.6 times as much to land between 50.6% and 60.2%. The team also flagged 216 trials as reward hacking, 163 of them online lookups, and scored them as failures.
One caveat keeps this honest. Model choice still explains 92.8% of cost variance, mostly because list prices differ 20 times over. And only one pairing is robust: Codex CLI with GLM-5.3 stays first in 99.6% of bootstrap redraws, while the other winners hold in 42% to 53%. Runta sells runtimes, but the method is published, and it is open models only, no closed frontier.
Two papers from Tuesday's arXiv batch point the same way. Harness Leverage Across Model Size (https://arxiv.org/abs/2611.21407) ran five shipped harnesses with local models from 0.75B to 4.7B on 18 tasks, and the harness that controls completion had the best mean at every size. Anatomy of an Agent Harness (https://arxiv.org/abs/2611.19960) writes harness design down as 30 testable decision cards in six groups, so a harness can be checked instead of trusted.
On a day when three labs advertised cost per task, this is the footnote they skipped: the cost per task you pay depends on the harness you pair the model with. Swapping the model without re-picking the harness is leaving money on the table.
Comments