EvoSafeHarness Says Your Agent's Guardrails Should Be Custom, Not Universal
Top of HuggingFace's daily papers board today is EvoSafeHarness, arXiv 2609.05903, from Nanxi Li, Yingzi Ma, Yulong Cao, Edward Suh, Bo Li, Dawn Song and Chaowei Xiao. The premise is a sentence everyone shipping agents should sit with: existing harnesses are usually designed once by experts and applied across heterogeneous models and domains. One safety config, every model, every deployment. That is obviously wrong and almost universally practiced.
What they build instead is a synthesis loop that jointly optimizes natural-language policies and executable code logic, fitted to a specific model, a specific domain and a specific adversary. Policies and code together is the right call, because prose policies drift and pure code cannot express intent. The numbers hold up: DecodingTrust-Agent attack success rate drops from 45.6 percent to 10.0 percent with minimal utility loss, and on AgentDojo they get 82.8 percent utility at 0 percent attack success, roughly double the utility of comparable methods at that security level. Under adaptive attacks with limited refinement budget, average attack success stays under 20 percent. https://arxiv.org/abs/2609.05903
The finding underneath the benchmark table is the useful one for practitioners. Enforcement is deployment-dependent. Different models need different enforcement strength for the same policy, because a model that reliably refuses one phrasing folds to another. Different domains need different policy shapes. The implication is that a security vendor selling one hardened harness to every customer is selling something that is over-restrictive for half of them and porous for the other half.
Pair this with Ecdysis, which trended out of the same week arguing harness optimization should be batched and generalization-aware. Both papers are saying the harness is a fitted artifact, not a fixed one. That is a meaningfully different engineering posture from how most teams treat their agent scaffolding, which is as a config file somebody wrote in March.
Related reading: https://clauday.com/article/f907d596-5631-4680-96ab-972f3d852877
← Back to all articles
What they build instead is a synthesis loop that jointly optimizes natural-language policies and executable code logic, fitted to a specific model, a specific domain and a specific adversary. Policies and code together is the right call, because prose policies drift and pure code cannot express intent. The numbers hold up: DecodingTrust-Agent attack success rate drops from 45.6 percent to 10.0 percent with minimal utility loss, and on AgentDojo they get 82.8 percent utility at 0 percent attack success, roughly double the utility of comparable methods at that security level. Under adaptive attacks with limited refinement budget, average attack success stays under 20 percent. https://arxiv.org/abs/2609.05903
The finding underneath the benchmark table is the useful one for practitioners. Enforcement is deployment-dependent. Different models need different enforcement strength for the same policy, because a model that reliably refuses one phrasing folds to another. Different domains need different policy shapes. The implication is that a security vendor selling one hardened harness to every customer is selling something that is over-restrictive for half of them and porous for the other half.
Pair this with Ecdysis, which trended out of the same week arguing harness optimization should be batched and generalization-aware. Both papers are saying the harness is a fitted artifact, not a fixed one. That is a meaningfully different engineering posture from how most teams treat their agent scaffolding, which is as a config file somebody wrote in March.
Related reading: https://clauday.com/article/f907d596-5631-4680-96ab-972f3d852877
Comments