Amazon swapped Sonnet for a model that can't talk, and cut the bill 98%
The top story on Hacker News in the past day was not a model launch. It was two Amazon engineers explaining how they pulled Claude Sonnet 4.6 out of one of the world's largest internal ticketing systems and put in Strands Decider E4B, a model that cannot generate a single sentence. Their claim: it labels tickets just as well at 98% less cost, saving "millions of dollars of token costs in a year".
A decision model gets a state and a question with fixed options, and returns a probability for each option plus a confidence score. Nothing else. Ticket labeling fits perfectly: the ticket is the state, the 20 label categories are the options. Sonnet had been running a long system prompt on every ticket and writing JSON. E4B, the Gemma 4 build of Decider, runs on one A10G in a SageMaker ml.g5.xlarge at about 640 ms a request. That is roughly 5,600 tickets an hour per instance, $250 per million tickets at full use, about $500 per million with burst headroom. The team puts it at about 44x cheaper than Sonnet. Average Lambda duration fell 74% the day Sonnet was switched off.
The useful part is the scar tissue. Decider's window is 4,096 tokens, and the production label descriptions alone did not fit, so each of the 20 labels was rewritten as a one- or two-sentence criterion plus a "none" option, and long tickets drop their oldest comments. Then a surprise: reordering JSON keys in the same input moved output probabilities by up to 0.19. They now sort keys on every request. And the rollout used a three-way switch, Sonnet only, both in parallel on real traffic, Decider only. Judging every disagreement on live traffic told them more than any offline eval, and rolling back was a 45-second config change.
Keep the caveats in view. This is the Strands team's own blog, about a launch we covered on Oct 2, and nobody outside Amazon has checked the "just as well". Sonnet 4.6 is a nine-month-old model, not today's Sonnet. The authors also state the limits plainly: it works because the choice is closed, the catalog is known, volume is huge and latency mattered. If labels need explanations, the catalog changes weekly, or the useful context won't fit in a few thousand tokens, keep the LLM.
The bigger lesson: inside most agent pipelines there are steps that are really multiple-choice questions dressed up as text generation. Paying frontier prices for those is like hiring a novelist to tick boxes.
Source: https://strandsagents.com/blog/amazon-saves-millions-replacing-sonnet-with-strands-decider
← Back to all articles
A decision model gets a state and a question with fixed options, and returns a probability for each option plus a confidence score. Nothing else. Ticket labeling fits perfectly: the ticket is the state, the 20 label categories are the options. Sonnet had been running a long system prompt on every ticket and writing JSON. E4B, the Gemma 4 build of Decider, runs on one A10G in a SageMaker ml.g5.xlarge at about 640 ms a request. That is roughly 5,600 tickets an hour per instance, $250 per million tickets at full use, about $500 per million with burst headroom. The team puts it at about 44x cheaper than Sonnet. Average Lambda duration fell 74% the day Sonnet was switched off.
The useful part is the scar tissue. Decider's window is 4,096 tokens, and the production label descriptions alone did not fit, so each of the 20 labels was rewritten as a one- or two-sentence criterion plus a "none" option, and long tickets drop their oldest comments. Then a surprise: reordering JSON keys in the same input moved output probabilities by up to 0.19. They now sort keys on every request. And the rollout used a three-way switch, Sonnet only, both in parallel on real traffic, Decider only. Judging every disagreement on live traffic told them more than any offline eval, and rolling back was a 45-second config change.
Keep the caveats in view. This is the Strands team's own blog, about a launch we covered on Oct 2, and nobody outside Amazon has checked the "just as well". Sonnet 4.6 is a nine-month-old model, not today's Sonnet. The authors also state the limits plainly: it works because the choice is closed, the catalog is known, volume is huge and latency mattered. If labels need explanations, the catalog changes weekly, or the useful context won't fit in a few thousand tokens, keep the LLM.
The bigger lesson: inside most agent pipelines there are steps that are really multiple-choice questions dressed up as text generation. Paying frontier prices for those is like hiring a novelist to tick boxes.
Source: https://strandsagents.com/blog/amazon-saves-millions-replacing-sonnet-with-strands-decider
Comments