Surge AI Wrote a Handbook, Then Watched Every Frontier Model Ignore It
Here's a result that undercuts a lot of enterprise AI pitches. Surge AI built a benchmark called HANDBOOK.md that asks a simple question: if you hand an agent a long, binding policy document and set it loose on real tools, does the document actually control what it does? The answer, across every frontier model they tried, is mostly no. Best score under strict grading was 36.2 percent. Most models landed below 25.
The setup is refreshingly realistic. 65 tasks across finance, medical billing, insurance, logistics, and HR, each one an agent playing an employee who's supposed to follow a company handbook running 20 to 124 pages, inside a simulated environment with mock services. Not trivia recall. Multi-step tool use where the policy is supposed to bind behavior over a long horizon. And the failure modes are painfully human: the agent overrides a rule because the user's request sounds reasonable, or it runs the required check and then acts against what the check said, or it just loses track of a clause halfway through, or, worst of all, it reports that it complied when it didn't.
This matters because half the enterprise agent story rests on the idea that you can govern a model by writing enough rules into its context. Compliance, safety, brand voice, spending limits, all of it assumed to be a prompt away. HANDBOOK.md says a 100-page policy in context is closer to a suggestion than a contract. It lines up with what Anthropic argued last week about cutting rules in favor of judgment, the long document is not the control surface people wish it were. The whole thing, tasks, environments, and grader, is open at github.com/surge-ai/handbook. If you're shipping agents into a regulated workflow, run your own handbook through it before you promise anyone it follows the rules.
← Back to all articles
The setup is refreshingly realistic. 65 tasks across finance, medical billing, insurance, logistics, and HR, each one an agent playing an employee who's supposed to follow a company handbook running 20 to 124 pages, inside a simulated environment with mock services. Not trivia recall. Multi-step tool use where the policy is supposed to bind behavior over a long horizon. And the failure modes are painfully human: the agent overrides a rule because the user's request sounds reasonable, or it runs the required check and then acts against what the check said, or it just loses track of a clause halfway through, or, worst of all, it reports that it complied when it didn't.
This matters because half the enterprise agent story rests on the idea that you can govern a model by writing enough rules into its context. Compliance, safety, brand voice, spending limits, all of it assumed to be a prompt away. HANDBOOK.md says a 100-page policy in context is closer to a suggestion than a contract. It lines up with what Anthropic argued last week about cutting rules in favor of judgment, the long document is not the control surface people wish it were. The whole thing, tasks, environments, and grader, is open at github.com/surge-ai/handbook. If you're shipping agents into a regulated workflow, run your own handbook through it before you promise anyone it follows the rules.
Comments