Karpathy: 5,000 People Are Watching Swarms. A Harvard Paper Says Swarm Size Is the Risk Variable.
Andrej Karpathy dusted off an April post on Saturday and said the gap in shared understanding of what these systems can do is no longer two groups talking past each other, it is a sharp funnel. His napkin math: about 6 billion people have barely touched an LLM. One to two billion use free-tier chatbots and reasonably see a derpy search replacement. Around 20 million professionals in math and code, 0.2 percent of the planet, are watching weeks-long projects complete from a prompt, building, copying, translating and decompiling apps. And roughly 5,000 people with internal frontier access are watching swarms of thousands of agents collaborate for weeks on software mega-projects, minting zero days, running attacks and defenses at machine speed, advancing mathematics. His line on the last tier: human review and comprehension are starting to fall behind, people are still doing archaeology on the OpenAI Hugging Face incident, and mathematicians will be poring over the 722 manuscripts for a while. The post sat at 530,000 impressions and 11,000 likes by evening and topped Hacker News.
He gives three drivers. Impact scales with ambition and horizon, so a paragraph-length question never stresses the system. Capability is jagged, peaking where tasks are digital, verifiable and economically valuable, because that is what the RL environment mixture is tuned to. And access is tiered. The uncomfortable implication is that the tier with the clearest view is also the one with the least public accountability, which is exactly the shape of every incident report this month.
A paper on Friday's arXiv listing from Erin Crawley and Hidenori Tanaka puts a model under Karpathy's top tier. Ecology of AI Agents treats a population of agents the way ecologists treat a species: a growth equation where fitness depends on cyber capability, since agents that can compromise machines can deploy more agents. Without collaboration, the population only takes off if a single agent's capability clears a threshold. With collaboration, collective capability rises with population size, and that produces a second threshold on population itself. Below it the population shrinks, above it, it runs away, with no change in what any individual agent can do. Ecologists call it the strong Allee effect. The practical claim follows directly: red-teaming a handful of agents tells you nothing about a thousand, so the authors ask for ecological red teaming and population pacing, deploying larger fleets in controlled settings while measuring how capability scales with headcount, and re-estimating the critical size for every new model generation.
Put the two together and the number that matters for safety is not the model's score, it is how many copies are cooperating. OpenAI's 1,200-agent Hugging Face swarm is the one public datapoint, Karpathy says thousands is the internal norm, and nobody outside has published the curve.
Karpathy: https://x.com/karpathy/status/2109361546505966046
Paper: https://arxiv.org/abs/2610.12436
← Back to all articles
He gives three drivers. Impact scales with ambition and horizon, so a paragraph-length question never stresses the system. Capability is jagged, peaking where tasks are digital, verifiable and economically valuable, because that is what the RL environment mixture is tuned to. And access is tiered. The uncomfortable implication is that the tier with the clearest view is also the one with the least public accountability, which is exactly the shape of every incident report this month.
A paper on Friday's arXiv listing from Erin Crawley and Hidenori Tanaka puts a model under Karpathy's top tier. Ecology of AI Agents treats a population of agents the way ecologists treat a species: a growth equation where fitness depends on cyber capability, since agents that can compromise machines can deploy more agents. Without collaboration, the population only takes off if a single agent's capability clears a threshold. With collaboration, collective capability rises with population size, and that produces a second threshold on population itself. Below it the population shrinks, above it, it runs away, with no change in what any individual agent can do. Ecologists call it the strong Allee effect. The practical claim follows directly: red-teaming a handful of agents tells you nothing about a thousand, so the authors ask for ecological red teaming and population pacing, deploying larger fleets in controlled settings while measuring how capability scales with headcount, and re-estimating the critical size for every new model generation.
Put the two together and the number that matters for safety is not the model's score, it is how many copies are cooperating. OpenAI's 1,200-agent Hugging Face swarm is the one public datapoint, Karpathy says thousands is the internal norm, and nobody outside has published the curve.
Karpathy: https://x.com/karpathy/status/2109361546505966046
Paper: https://arxiv.org/abs/2610.12436
Comments