Two papers on making agents cheaper: rebuild the server, shrink the model
Agents don't talk to models the way chatbots do. They come back again and again with a growing context, call tools, wait, and come back. Two papers on Thursday's Hugging Face board attack the cost of that from opposite ends.
Evolving Infra, from MIT's Han lab, rebuilds the serving layer for agent traffic: https://arxiv.org/abs/2611.12304, code at https://github.com/mit-han-lab/EvolvingInfra. Today's prefill/decode disaggregated serving splits machines into fixed roles and evicts KV caches using coarse signals, so it throws away caches an agent is about to reuse and leaves one side idle while the other queues. Evolving Infra decouples compute from KV-cache management so a node can flip between prefill and decode, uses an online planner to match the split to the workload mix, and lets an agent-aware KV manager use the agent's own context-management signals to decide what to keep. On coding, math and computer-use tasks with GLM-5.2 and Kimi K2.6 on up to 256 GB300 GPUs, the authors report mean end-to-end latency and P95 time to first token down by up to 4.1x and 14x versus NVIDIA's Dynamo.
SteerDistill attacks the model instead: https://arxiv.org/abs/2611.13410, code at https://github.com/air-embodied-brain/SteerDistill. Quantize an agent to 4 bits and it fails in a specific way: it overlooks task requirements or environment conditions. So a full-precision teacher watches the 4-bit student fail, steers it to revise those exact turns, checks whether the fix worked in the environment, and the student is then trained to act without the teacher. Across five agent benchmarks and three models, the authors report W4A16 agents 1.67 to 2.18 points above the best quantized baseline and within one point of BF16. On tau2-Bench it beats QeRL by 1.84 points using 67% fewer training GPU hours.
Both papers are the authors' numbers on their own setups, and "up to 14x" is a best case, not an average. But the direction is clear. The cost of an agent is now an infrastructure question as much as a model question, and the people who understand agent traffic patterns, how contexts grow and get reused, will run them for a fraction of what a chatbot-shaped stack costs.
← Back to all articles
Evolving Infra, from MIT's Han lab, rebuilds the serving layer for agent traffic: https://arxiv.org/abs/2611.12304, code at https://github.com/mit-han-lab/EvolvingInfra. Today's prefill/decode disaggregated serving splits machines into fixed roles and evicts KV caches using coarse signals, so it throws away caches an agent is about to reuse and leaves one side idle while the other queues. Evolving Infra decouples compute from KV-cache management so a node can flip between prefill and decode, uses an online planner to match the split to the workload mix, and lets an agent-aware KV manager use the agent's own context-management signals to decide what to keep. On coding, math and computer-use tasks with GLM-5.2 and Kimi K2.6 on up to 256 GB300 GPUs, the authors report mean end-to-end latency and P95 time to first token down by up to 4.1x and 14x versus NVIDIA's Dynamo.
SteerDistill attacks the model instead: https://arxiv.org/abs/2611.13410, code at https://github.com/air-embodied-brain/SteerDistill. Quantize an agent to 4 bits and it fails in a specific way: it overlooks task requirements or environment conditions. So a full-precision teacher watches the 4-bit student fail, steers it to revise those exact turns, checks whether the fix worked in the environment, and the student is then trained to act without the teacher. Across five agent benchmarks and three models, the authors report W4A16 agents 1.67 to 2.18 points above the best quantized baseline and within one point of BF16. On tau2-Bench it beats QeRL by 1.84 points using 67% fewer training GPU hours.
Both papers are the authors' numbers on their own setups, and "up to 14x" is a best case, not an average. But the direction is clear. The cost of an agent is now an infrastructure question as much as a model question, and the people who understand agent traffic patterns, how contexts grow and get reused, will run them for a fraction of what a chatbot-shaped stack costs.
Comments