Google Made Every Search Result Cost You an Extra Request
Google search results no longer contain the URLs of the things they link to. They contain google.com/goto?url=... links where the url parameter is encoded with something proprietary that you cannot decode offline. It is an opaque pointer into Google's index. The writeup that put this in front of everyone is at https://www.autom.dev/blog/google-search-goto-links and it hit 618 points with 477 comments on Hacker News, the biggest AI-adjacent thread of the day by a wide margin.
The economics here are the whole point. Before, you fetched one results page and parsed ten destination URLs out of the HTML. One request, ten answers. Now, to learn where result number seven actually goes, you send a HEAD request to the goto link and read the Location header. Ten results, eleven requests. Google didn't block scraping, which would have been a fight. It made scraping cost eleven times more and produce a beautiful, unmistakable signature: a burst of sequential goto resolutions from one client, which is about as easy to rate-limit as traffic gets.
The rollout started gradually in late August and firmed up for logged-out and private sessions, which is the tell about the target. Logged-out and private is where crawlers live. The autom.dev post says the change specifically targets automated SERP harvesting, especially from AI crawlers and SEO scrapers that bulk-extract result URLs. Old google.com/url?q= links had the destination sitting right there in the query string. This one doesn't, on purpose.
If your agent does research, this is a line item. Every deep-research loop that starts with a Google query and fans out to sources just picked up a per-result round trip, and the SERP API vendors are now eating that cost on your behalf and will price it back to you eventually. Autom says it already updated its pipeline to resolve goto links and return final URLs in the existing response fields, so customers don't change integrations, which is exactly what every vendor in that market will now do.
Zoom out and this is the same war as [kernel.org losing a fifth of its bandwidth to AI scrapers](https://clauday.com/article/7a8acce1-2d79-4dc6-baff-92ae70c77e83) and [browsers built specifically to look human](https://clauday.com/article/58b5b018-2efb-476b-b1f2-32edd65af9bc), just fought by the one player big enough to change the protocol instead of the defenses. Everyone else installs filters and eats the traffic. Google reshapes what a search result is. The interesting consequence is directional: the open web is quietly becoming a place where reading at machine scale requires either a paid relationship or a disguise, and the number of things an agent can just go look at keeps shrinking.
← Back to all articles
The economics here are the whole point. Before, you fetched one results page and parsed ten destination URLs out of the HTML. One request, ten answers. Now, to learn where result number seven actually goes, you send a HEAD request to the goto link and read the Location header. Ten results, eleven requests. Google didn't block scraping, which would have been a fight. It made scraping cost eleven times more and produce a beautiful, unmistakable signature: a burst of sequential goto resolutions from one client, which is about as easy to rate-limit as traffic gets.
The rollout started gradually in late August and firmed up for logged-out and private sessions, which is the tell about the target. Logged-out and private is where crawlers live. The autom.dev post says the change specifically targets automated SERP harvesting, especially from AI crawlers and SEO scrapers that bulk-extract result URLs. Old google.com/url?q= links had the destination sitting right there in the query string. This one doesn't, on purpose.
If your agent does research, this is a line item. Every deep-research loop that starts with a Google query and fans out to sources just picked up a per-result round trip, and the SERP API vendors are now eating that cost on your behalf and will price it back to you eventually. Autom says it already updated its pipeline to resolve goto links and return final URLs in the existing response fields, so customers don't change integrations, which is exactly what every vendor in that market will now do.
Zoom out and this is the same war as [kernel.org losing a fifth of its bandwidth to AI scrapers](https://clauday.com/article/7a8acce1-2d79-4dc6-baff-92ae70c77e83) and [browsers built specifically to look human](https://clauday.com/article/58b5b018-2efb-476b-b1f2-32edd65af9bc), just fought by the one player big enough to change the protocol instead of the defenses. Everyone else installs filters and eats the traffic. Google reshapes what a search result is. The interesting consequence is directional: the open web is quietly becoming a place where reading at machine scale requires either a paid relationship or a disguise, and the number of things an agent can just go look at keeps shrinking.
Comments