PalmClaw Says Phone Agents Should Stop Looking at the Screen
PalmClaw hit 1.12k upvotes on HuggingFace papers, which is an absurd number for an agent framework paper. The idea is simple enough to explain in one line: stop making phone agents tap buttons.
Every mobile agent you've seen works by screenshotting the screen, figuring out where the button is, and tapping it. It's how humans use phones, so it seems obviously right. It's also miserably slow and fragile, because you're driving a machine through the interface built for eyes and thumbs. PalmClaw throws that out and exposes the phone's actual capabilities as device tools with structured operations and defined execution boundaries. The agent doesn't look for the send button. It calls send.
The numbers make the argument better than the abstract does. 97.1% task success versus 87.1% for the strongest GUI-based competitor. 11.5% relative improvement over the best baseline. And the one that should stop you: 94.9% reduction in average completion time. Not 5% faster. Twenty times faster. That's the cost of pretending to be a thumb, quantified.
It runs on the device, it's open source under AGPLv3, setup is two steps and about two minutes, and the experiments used DeepSeek-V4-Flash through an API rather than a local model. Authors are from Hong Kong Polytechnic University and Hangzhou Diagens Biotechnology.
The reason this matters beyond phones: GUI agents are the default assumption everywhere right now, including desktop computer use. The whole field decided that the way to automate software is to teach a model to be a person clicking. PalmClaw's result suggests that's a transitional hack, not the destination. Wherever a real API or tool interface exists, going through the pixels is a 20x tax you're paying for nothing.
https://github.com/ModalityDance/PalmClaw
← Back to all articles
Every mobile agent you've seen works by screenshotting the screen, figuring out where the button is, and tapping it. It's how humans use phones, so it seems obviously right. It's also miserably slow and fragile, because you're driving a machine through the interface built for eyes and thumbs. PalmClaw throws that out and exposes the phone's actual capabilities as device tools with structured operations and defined execution boundaries. The agent doesn't look for the send button. It calls send.
The numbers make the argument better than the abstract does. 97.1% task success versus 87.1% for the strongest GUI-based competitor. 11.5% relative improvement over the best baseline. And the one that should stop you: 94.9% reduction in average completion time. Not 5% faster. Twenty times faster. That's the cost of pretending to be a thumb, quantified.
It runs on the device, it's open source under AGPLv3, setup is two steps and about two minutes, and the experiments used DeepSeek-V4-Flash through an API rather than a local model. Authors are from Hong Kong Polytechnic University and Hangzhou Diagens Biotechnology.
The reason this matters beyond phones: GUI agents are the default assumption everywhere right now, including desktop computer use. The whole field decided that the way to automate software is to teach a model to be a person clicking. PalmClaw's result suggests that's a transitional hack, not the destination. Wherever a real API or tool interface exists, going through the pixels is a 20x tax you're paying for nothing.
https://github.com/ModalityDance/PalmClaw
Comments