NVIDIA SkillEvaluator Asks Whether Your Agent Skill Actually Helps
Agent skills are multiplying faster than anyone is checking them. On Sunday's GitHub Trending, three skill bundles sat on the board at once, from trading strategies to a red-team pack, and none came with evidence that an agent does better with them installed. NVIDIA's SkillEvaluator, trending at 205 stars a day (about 1,600 total), is built for exactly that question. Repo: https://github.com/NVIDIA/SkillEvaluator , docs: https://docs.nvidia.com/skills/skillevaluator/
It grades a skill folder, written to the Agent Skills spec, in tiers. Tier 1 runs static quality, security and PII checks with no API key. Tier 2 looks for semantic overlap with other skills. Tier 3 is the one that matters: run the agent on a dataset with the skill and without it, and report the difference. NVIDIA calls that number Skill Lift, with pass@k shown as a reliability side note. pip install skillevaluator gets you the CLI, skillevaluator ci init opens a setup PR that checks every changed skill on each pull request, and you can even install it as a skill and ask your agent whether your skill is any good. It is Apache-2.0 and part of NVIDIA's Verified Skills pipeline.
Version 0.9.0 shipped Nov 13, before this window, so treat this as a trending pickup, not a launch. The release is mostly about trusting the grader. Datasets can be generated from skill instructions, author intent or real traces, but expectations only get promoted after explicit human review. You can reuse a no-skill baseline, compare skill versions over time, and cap token, judge-call and dollar budgets, with partial results kept when a limit stops the run. A tier3 validate-grader command qualifies the verifier itself, and every judge request and reply is kept for audit.
The shift is small in wording and big in practice. The old question was whether a skill is well written. The new one is whether the agent scores higher with it than without it, measured the same way every time. That is the verifier-is-the-product idea applied to the fastest-growing, least-tested layer of the agent stack. A skill marketplace without a lift number is an app store without reviews.
← Back to all articles
It grades a skill folder, written to the Agent Skills spec, in tiers. Tier 1 runs static quality, security and PII checks with no API key. Tier 2 looks for semantic overlap with other skills. Tier 3 is the one that matters: run the agent on a dataset with the skill and without it, and report the difference. NVIDIA calls that number Skill Lift, with pass@k shown as a reliability side note. pip install skillevaluator gets you the CLI, skillevaluator ci init opens a setup PR that checks every changed skill on each pull request, and you can even install it as a skill and ask your agent whether your skill is any good. It is Apache-2.0 and part of NVIDIA's Verified Skills pipeline.
Version 0.9.0 shipped Nov 13, before this window, so treat this as a trending pickup, not a launch. The release is mostly about trusting the grader. Datasets can be generated from skill instructions, author intent or real traces, but expectations only get promoted after explicit human review. You can reuse a no-skill baseline, compare skill versions over time, and cap token, judge-call and dollar budgets, with partial results kept when a limit stops the run. A tier3 validate-grader command qualifies the verifier itself, and every judge request and reply is kept for audit.
The shift is small in wording and big in practice. The old question was whether a skill is well written. The new one is whether the agent scores higher with it than without it, measured the same way every time. That is the verifier-is-the-product idea applied to the fastest-growing, least-tested layer of the agent stack. A skill marketplace without a lift number is an app store without reviews.
Comments