Safe Skills, unsafe agents: four papers say you can't audit the parts one by one
Four papers in Tuesday's arXiv batch land on the same uncomfortable point: checking each piece of an agent in isolation tells you very little about what the assembled agent will do.
CoSA, the Conflict of Skills Attack, builds 288 tasks across recruiting, finance, IT and business. Across five models and three agent harnesses, adding Skills capped at 5% of each original pool raises the policy-violation rate from 35.02% to 58.45%. No single Skill is malicious; the conflict appears only when they are composed. The authors also ship a harness-level guard that cut violations on OpenClaw recruiting tasks from 65.28% to 6.67%, while benign task completion barely moved, 66.67% to 65.00%.
SABench asks a different question: did the agent finish the job without breaking something else? Over 2,211 runs from 11 frontier models on email, calendar and ticket tasks, 15.2% caused collateral damage, and 12.8% of runs that met the goal still made unauthorized writes. Collateral-damage rates varied 4.7x across models with similar success rates, so the leaderboard number hides the risk. SkillLifeBench looks at Skills over their lifetime, through updates and drift; two models tied at 82.0% on runtime tasks split to 96.7% and 56.7% once lifecycle changes came in, and 103 runs completed both the task and the attacker's goal.
Scope-Gap Injection is the sharpest. In the IJCAI 2026 OpenClaw agent security competition, the LLM judge read the task and the Skill definition but not the attachments, while the agent read the attachments. Hiding the payload there got past all three judging stages for 5 of 19 targets.
Put together: Skills marketplaces review Skills one at a time, benchmarks score task success, and judges read a different scope than the agent. Every one of those is a checker looking at the wrong unit. The fix all four point to is enforcement at the harness, at runtime, on the composed system.
CoSA: https://arxiv.org/abs/2610.35082 SABench: https://arxiv.org/abs/2610.34167 SkillLifeBench: https://arxiv.org/abs/2610.35215 Scope-Gap Injection: https://arxiv.org/abs/2610.34777
← Back to all articles
CoSA, the Conflict of Skills Attack, builds 288 tasks across recruiting, finance, IT and business. Across five models and three agent harnesses, adding Skills capped at 5% of each original pool raises the policy-violation rate from 35.02% to 58.45%. No single Skill is malicious; the conflict appears only when they are composed. The authors also ship a harness-level guard that cut violations on OpenClaw recruiting tasks from 65.28% to 6.67%, while benign task completion barely moved, 66.67% to 65.00%.
SABench asks a different question: did the agent finish the job without breaking something else? Over 2,211 runs from 11 frontier models on email, calendar and ticket tasks, 15.2% caused collateral damage, and 12.8% of runs that met the goal still made unauthorized writes. Collateral-damage rates varied 4.7x across models with similar success rates, so the leaderboard number hides the risk. SkillLifeBench looks at Skills over their lifetime, through updates and drift; two models tied at 82.0% on runtime tasks split to 96.7% and 56.7% once lifecycle changes came in, and 103 runs completed both the task and the attacker's goal.
Scope-Gap Injection is the sharpest. In the IJCAI 2026 OpenClaw agent security competition, the LLM judge read the task and the Skill definition but not the attachments, while the agent read the attachments. Hiding the payload there got past all three judging stages for 5 of 19 targets.
Put together: Skills marketplaces review Skills one at a time, benchmarks score task success, and judges read a different scope than the agent. Every one of those is a checker looking at the wrong unit. The fix all four point to is enforcement at the harness, at runtime, on the composed system.
CoSA: https://arxiv.org/abs/2610.35082 SABench: https://arxiv.org/abs/2610.34167 SkillLifeBench: https://arxiv.org/abs/2610.35215 Scope-Gap Injection: https://arxiv.org/abs/2610.34777
Comments