GUI Agent Skills That Rewrite Themselves Mid-Run, No Training
EvoSkill-GUI comes out of ZJU-REAL at Zhejiang University and takes on the thing that actually breaks computer-use agents in production, which is not reasoning. It is that the interface moved. A pop-up appeared, a panel loaded late, a button shifted, and a skill that worked yesterday now drives into a wall. Paper is Reflect, Revise, Reuse: Training-Free Skill Evolution for GUI Agents, arXiv 2609.17653, at https://arxiv.org/abs/2609.17653.
Their move is to stop treating a skill as a frozen script. A skill here is a package containing an executable plan, recovery rules, and a record of past failure cases, and it can be revised from execution feedback at deployment time with no additional training. The loop is in the title. Reflect on what went wrong, isolate the diagnosis to the step that failed rather than blaming the whole trajectory, revise the skill in place, reuse it.
The gains are 16.2 percent on MobileWorld, 6.0 percent on AndroidWorld and 10.5 percent on OSWorld at maximum, and the more interesting result is that an evolved skill library transfers, so fixing a pop-up handler for one task improves related tasks that never hit that failure. That is the difference between a patch and a memory.
Training-free is the part that decides whether anyone uses this. Everything that improves through gradients requires you to own the model, collect the data, and run the job, which means about four organizations can do it and everyone else waits for a release. Everything that improves through a writable skill library runs on top of an API you are already paying for.
Environment knowledge going stale is now the recurring failure across every computer-use paper this month. This is the first one where the agent is allowed to edit its own notes about the world instead of waiting for a human to notice.
← Back to all articles
Their move is to stop treating a skill as a frozen script. A skill here is a package containing an executable plan, recovery rules, and a record of past failure cases, and it can be revised from execution feedback at deployment time with no additional training. The loop is in the title. Reflect on what went wrong, isolate the diagnosis to the step that failed rather than blaming the whole trajectory, revise the skill in place, reuse it.
The gains are 16.2 percent on MobileWorld, 6.0 percent on AndroidWorld and 10.5 percent on OSWorld at maximum, and the more interesting result is that an evolved skill library transfers, so fixing a pop-up handler for one task improves related tasks that never hit that failure. That is the difference between a patch and a memory.
Training-free is the part that decides whether anyone uses this. Everything that improves through gradients requires you to own the model, collect the data, and run the job, which means about four organizations can do it and everyone else waits for a release. Everything that improves through a writable skill library runs on top of an API you are already paying for.
Environment knowledge going stale is now the recurring failure across every computer-use paper this month. This is the first one where the agent is allowed to edit its own notes about the world instead of waiting for a human to notice.
Comments