MerchantBench: A Year of Shopkeeping Breaks Every Agent
The top agent paper on Hugging Face Daily Papers this round (83 upvotes) is MerchantBench (arxiv.org/abs/2607.28956), and its result deserves the attention: run LLM agents as e-commerce sellers for a simulated 365 days, and the best configuration ends the year with 27.3% of the final net assets that human participants achieve. Not 27% worse. 27% of.
The setup is the most serious long-horizon simulation yet: order-level, grounded in 98,843 real product records from 1688 (Alibaba's wholesale platform), with 26 tools. The agent handles product sourcing, listing and pricing, cash-flow management β and, the clever part, mixed-latency feedback, where some consequences arrive immediately and some take simulated weeks. Eight LLMs across two agent frameworks, 48 full-year runs. Everything fails the same way: agents execute individual tasks fine, then lose the thread when today's decision constrains next month's options and feedback arrives out of order.
This is Vending-Bench's grown-up sibling, and it lands the same week the Saul experiment showed GPT-5.6 Sol burning $350 and 320 million tokens to earn zero revenue running a real business. The pattern across both: what is missing is not capability but coherence β behaving purposefully across compounding consequences. Benchmarks that only measure bounded tasks with immediate success criteria will keep telling us agents are ready for jobs they are not ready for. A 365-day simulation with cash flow is much closer to what "doing a job" actually means, and the honest number it produces is 27.3%.
← Back to all articles
The setup is the most serious long-horizon simulation yet: order-level, grounded in 98,843 real product records from 1688 (Alibaba's wholesale platform), with 26 tools. The agent handles product sourcing, listing and pricing, cash-flow management β and, the clever part, mixed-latency feedback, where some consequences arrive immediately and some take simulated weeks. Eight LLMs across two agent frameworks, 48 full-year runs. Everything fails the same way: agents execute individual tasks fine, then lose the thread when today's decision constrains next month's options and feedback arrives out of order.
This is Vending-Bench's grown-up sibling, and it lands the same week the Saul experiment showed GPT-5.6 Sol burning $350 and 320 million tokens to earn zero revenue running a real business. The pattern across both: what is missing is not capability but coherence β behaving purposefully across compounding consequences. Benchmarks that only measure bounded tasks with immediate success criteria will keep telling us agents are ready for jobs they are not ready for. A 365-day simulation with cash flow is much closer to what "doing a job" actually means, and the honest number it produces is 27.3%.
Comments