llama-modchip squeezes a 284B model into 64GB of RAM, no GPU
A 284-billion-parameter model on a desktop with no GPU and 64GB of RAM. That's the pitch of llama-modchip, a set of llama.cpp patches and quantization tools that took 315 points on Hacker News on Sunday. The repo itself dates from Oct 29 and the last release, v2.4.0, came out Nov 4, so this is a discovery more than a launch. It's Apache 2.0.
The headline trick: standard 1-bit builds of DeepSeek-V4-Flash (284.33B total, 13.17B active) take 76.87 GiB, more than the roughly 62.5 GiB a 64GB Linux box can actually use. llama-modchip's transcoder keeps embeddings, shared experts and attention projections at higher precision and squeezes the cold routed experts harder, landing at 1.71 bits per weight and 56.56 GiB on disk. Peak memory at 256K context is 57.61 GiB, with zero swap. By default all six routed experts still run per layer, with no expert dropping.
Speed, all author-reported on a 16-core Zen 5: 44.4 tokens per second prefill and 9.1 to 14.7 tokens per second generation for the 284B model, thanks to cache-tiled AVX-512 kernels that the author says add 77% to prefill and 53% to decode. A 30B Nemotron at 3.75 bits runs 25 to 60 tokens per second on a 16-32GB laptop. It serves an OpenAI- and Anthropic-compatible endpoint, so a coding agent can point at it directly.
The catch is the one that matters for agents. 1.71 bits per weight is an extreme quant, and the README shows speed and memory but no agent-task quality numbers. A model that fits but loses the thread on step 30 of a tool loop is a demo, not a backend. Ask for SWE-style numbers before you swap out your API key.
Still, 14 tokens a second from a frontier-class open model on hardware you already own is a real line for private and offline agent work.
Source: https://github.com/synchopate/llama-modchip
← Back to all articles
The headline trick: standard 1-bit builds of DeepSeek-V4-Flash (284.33B total, 13.17B active) take 76.87 GiB, more than the roughly 62.5 GiB a 64GB Linux box can actually use. llama-modchip's transcoder keeps embeddings, shared experts and attention projections at higher precision and squeezes the cold routed experts harder, landing at 1.71 bits per weight and 56.56 GiB on disk. Peak memory at 256K context is 57.61 GiB, with zero swap. By default all six routed experts still run per layer, with no expert dropping.
Speed, all author-reported on a 16-core Zen 5: 44.4 tokens per second prefill and 9.1 to 14.7 tokens per second generation for the 284B model, thanks to cache-tiled AVX-512 kernels that the author says add 77% to prefill and 53% to decode. A 30B Nemotron at 3.75 bits runs 25 to 60 tokens per second on a 16-32GB laptop. It serves an OpenAI- and Anthropic-compatible endpoint, so a coding agent can point at it directly.
The catch is the one that matters for agents. 1.71 bits per weight is an extreme quant, and the README shows speed and memory but no agent-task quality numbers. A model that fits but loses the thread on step 30 of a tool loop is a demo, not a backend. Ask for SWE-style numbers before you swap out your API key.
Still, 14 tokens a second from a frontier-class open model on hardware you already own is a real line for private and offline agent work.
Source: https://github.com/synchopate/llama-modchip
Comments