AirLLM Squeezes Kimi K3, a 2.8T Model, Into 3.72GB of VRAM
AirLLM has been on GitHub trending three days straight, over 1,700 stars added today alone, 28K total. The trigger is v3.1.0, released July 29: it runs Kimi K3, the 2.8-trillion-parameter open-weights flagship, with peak VRAM usage of 3.72GB. That is not a typo. A model roughly a thousand times larger than the memory it occupies, running on hardware most gamers already own.
The trick is layer streaming. Instead of holding the model in memory, AirLLM loads one layer at a time from disk, computes, discards, loads the next. Honest caveat, and the project is upfront about it: this is slow. Tokens come out at a crawl, and nobody should serve users this way. What it buys you is access, not throughput.
Access turns out to matter, and the star count says so. Until now, touching a frontier-scale open model meant renting a GPU cluster or trusting someone's API. With this, a student with a 4GB card can load the actual K3 weights, probe them, run evals, extract layers, study routing. The open-weights wave of the past two weeks, Kimi K3, DeepSeek V4 Flash, Qwen3.8-Max promising weights next week, keeps producing artifacts that only a handful of labs can run. Tools like this are how the other several million people get their hands on them.
Repo at github.com/lyogavin/airllm.
← Back to all articles
The trick is layer streaming. Instead of holding the model in memory, AirLLM loads one layer at a time from disk, computes, discards, loads the next. Honest caveat, and the project is upfront about it: this is slow. Tokens come out at a crawl, and nobody should serve users this way. What it buys you is access, not throughput.
Access turns out to matter, and the star count says so. Until now, touching a frontier-scale open model meant renting a GPU cluster or trusting someone's API. With this, a student with a 4GB card can load the actual K3 weights, probe them, run evals, extract layers, study routing. The open-weights wave of the past two weeks, Kimi K3, DeepSeek V4 Flash, Qwen3.8-Max promising weights next week, keeps producing artifacts that only a handful of labs can run. Tools like this are how the other several million people get their hands on them.
Repo at github.com/lyogavin/airllm.
Comments