Your LLM Judge Drifts, and Its Reasons Are Confabulated
Two more papers from the weekend batch go after the judge itself. Clean Engineering, Unstable Measurement (https://arxiv.org/abs/2609.04198) ran a preregistered audit of black-box LLM judges on shared endpoints and found the readings are unstable β the assumption that the same model name gives you the same judge tomorrow simply fails. Legibility Is Not Interpretability (https://arxiv.org/abs/2609.04194) is worse news: the reasoning steps a judge rates as important diverge from the steps that actually drive the outcome, which undercuts judge-based process supervision at the root.
Put together, that's two silent cracks under most agent eval stacks. If your pipeline scores outputs with a judge-as-a-service endpoint, your metric drifts underneath you between runs, and the judge's explanation of what mattered is confabulated. Every team that A/B-tested a harness change against last month's judge baseline has some numbers to quietly re-run.
This is the research wing of the same movement we covered as a deep dive β verification becoming the product (https://clauday.com/article/0637f1ca-e93f-4c63-b4a2-97b3632bb6b0). Artificial Analysis hides its test sets, coding benchmarks add reviewer-grade constraints, and now the judge is getting audited and failing. The practical floor, today: pin judge versions where you can, re-baseline on a schedule, and never cite a judge's rationale as ground truth about the model being judged. The judge is a measurement instrument, and these two papers say it's an uncalibrated one.
← Back to all articles
Put together, that's two silent cracks under most agent eval stacks. If your pipeline scores outputs with a judge-as-a-service endpoint, your metric drifts underneath you between runs, and the judge's explanation of what mattered is confabulated. Every team that A/B-tested a harness change against last month's judge baseline has some numbers to quietly re-run.
This is the research wing of the same movement we covered as a deep dive β verification becoming the product (https://clauday.com/article/0637f1ca-e93f-4c63-b4a2-97b3632bb6b0). Artificial Analysis hides its test sets, coding benchmarks add reviewer-grade constraints, and now the judge is getting audited and failing. The practical floor, today: pin judge versions where you can, re-baseline on a schedule, and never cite a judge's rationale as ground truth about the model being judged. The judge is a measurement instrument, and these two papers say it's an uncalibrated one.
Comments