AiSOC v21 admits its evals couldn't say which model answered, and its headline number halves
Here is a release note more agent products should write. AiSOC, an open-source AI security operations center, shipped v21.0.0 on November 1 and is trending on GitHub at about 436 stars a day, roughly 6,200 total.
The headline change is provenance. Before this release, every published verdict-eval row had an empty list of resolved models: 72 LLM calls were placed and answered, and nobody could say which model answered them. Now the extractor reads the model from the gateway header first, which it treats as verified, before falling back to the model field in the response body, which it treats as self-reported. Every scorecard carries a provenance status: verified, self-reported, unattributed, serving mismatch or no calls.
The second change is more painful. A grouping fix stopped counting unrelated alerts as grouped, and the measured alert-reduction figure dropped from 42.6% to 24.5%. That's the project's own key metric, and it nearly halved because the old number was counting the wrong thing.
Why it matters beyond one repo: agent systems route through gateways, fallbacks and model aliases, and most eval dashboards quietly assume they know which model produced each answer. AiSOC found out it didn't, and published both the gap and the corrected number. If your agent evals don't record the serving model from the gateway side, you may have the same gap.
Release: https://github.com/beenuar/AiSOC/releases/tag/v21.0.0
← Back to all articles
The headline change is provenance. Before this release, every published verdict-eval row had an empty list of resolved models: 72 LLM calls were placed and answered, and nobody could say which model answered them. Now the extractor reads the model from the gateway header first, which it treats as verified, before falling back to the model field in the response body, which it treats as self-reported. Every scorecard carries a provenance status: verified, self-reported, unattributed, serving mismatch or no calls.
The second change is more painful. A grouping fix stopped counting unrelated alerts as grouped, and the measured alert-reduction figure dropped from 42.6% to 24.5%. That's the project's own key metric, and it nearly halved because the old number was counting the wrong thing.
Why it matters beyond one repo: agent systems route through gateways, fallbacks and model aliases, and most eval dashboards quietly assume they know which model produced each answer. AiSOC found out it didn't, and published both the gap and the corrected number. If your agent evals don't record the serving model from the gateway side, you may have the same gap.
Release: https://github.com/beenuar/AiSOC/releases/tag/v21.0.0
Comments