RLSVR: A Spy Game Instead of a Reward Model
Top paper on Hugging Face daily papers with 143 upvotes: From RLVR to RLSVR, Task Transformation Induces Self-Verifiable Rewards for Open-Ended LLM Self-Improvement. The problem is familiar. RL with verifiable rewards works great on math and code because you can check the answer. Open-ended tasks like writing and summarization have no checker, so everyone falls back on reward models or LLM judges, which drift, get gamed, and cost inference money.
The trick: don't judge the output, transform the task so the environment scores itself. Their instantiation is SpyRL, an information-asymmetric self-play game, think of the party game Spyfall. A spy model gets a slightly different brief, and players must produce output good enough that a teammate can spot who's off. Output quality becomes identity-recognition accuracy, which is exactly checkable, no judge needed. Across summarization, creative writing and even mathematical reasoning, SpyRL beats existing self-improvement methods.
Why it matters: the judge bottleneck is the wall every self-improvement pipeline hits. The student can't outgrow the grader, and LLM graders are expensive and hackable. Turning quality into a game outcome sidesteps the grader entirely. It also rhymes with what OpenAI's Astra math drop showed the same weekend from the formal-proof side: the models that improve fastest are the ones whose work can be checked mechanically. The current bottleneck of self-improvement isn't intelligence, it's verifiability.
Paper: https://arxiv.org/abs/2607.23802
← Back to all articles
The trick: don't judge the output, transform the task so the environment scores itself. Their instantiation is SpyRL, an information-asymmetric self-play game, think of the party game Spyfall. A spy model gets a slightly different brief, and players must produce output good enough that a teammate can spot who's off. Output quality becomes identity-recognition accuracy, which is exactly checkable, no judge needed. Across summarization, creative writing and even mathematical reasoning, SpyRL beats existing self-improvement methods.
Why it matters: the judge bottleneck is the wall every self-improvement pipeline hits. The student can't outgrow the grader, and LLM graders are expensive and hackable. Turning quality into a game outcome sidesteps the grader entirely. It also rhymes with what OpenAI's Astra math drop showed the same weekend from the formal-proof side: the models that improve fastest are the ones whose work can be checked mechanically. The current bottleneck of self-improvement isn't intelligence, it's verifiability.
Paper: https://arxiv.org/abs/2607.23802
Comments