oMLX Fixes the One Thing That Made Local Coding Agents Painful
Running a coding agent against a local model on a Mac has always had one ugly failure mode. The agent fires off dozens of requests where the prompt prefix keeps shifting slightly, and every existing MLX server throws away the KV cache the moment that happens, so it recomputes the entire context from scratch. On a long context that means thirty to ninety seconds before the first token. Do that on every step of an agent loop and the model isn't slow, it's unusable.
oMLX is a menu-bar app for Apple Silicon that goes straight at this. Its core trick is paged KV caching with a tiered store, hot blocks in RAM and cold blocks persisted to SSD, so when a prefix you've seen before comes back it's restored from disk instead of recomputed. Users are reporting time-to-first-token dropping from that 30-to-90-second range down to one to three seconds on long contexts. Under the hood it's a FastAPI process wrapping Apple's MLX, with an engine pool that holds several models in memory, block-based paged cache with prefix sharing and copy-on-write, and OpenAI plus Anthropic API compatibility so Claude Code, OpenClaw and Cursor can point at it directly.
It's Apache 2.0 and open source, installable as a .dmg, via Homebrew, or from source, and it also serves vision-language, OCR, and embedding models through the same batching stack. The reason this matters beyond convenience is the cost thread we keep coming back to: as the frontier gets expensive to default to, the question becomes how good a local setup you can actually run, and the bottleneck was never the model, it was the serving layer throwing away work. oMLX is the serving fix. Repo at github.com/jundot/omlx.
← Back to all articles
oMLX is a menu-bar app for Apple Silicon that goes straight at this. Its core trick is paged KV caching with a tiered store, hot blocks in RAM and cold blocks persisted to SSD, so when a prefix you've seen before comes back it's restored from disk instead of recomputed. Users are reporting time-to-first-token dropping from that 30-to-90-second range down to one to three seconds on long contexts. Under the hood it's a FastAPI process wrapping Apple's MLX, with an engine pool that holds several models in memory, block-based paged cache with prefix sharing and copy-on-write, and OpenAI plus Anthropic API compatibility so Claude Code, OpenClaw and Cursor can point at it directly.
It's Apache 2.0 and open source, installable as a .dmg, via Homebrew, or from source, and it also serves vision-language, OCR, and embedding models through the same batching stack. The reason this matters beyond convenience is the cost thread we keep coming back to: as the frontier gets expensive to default to, the question becomes how good a local setup you can actually run, and the bottleneck was never the model, it was the serving layer throwing away work. oMLX is the serving fix. Repo at github.com/jundot/omlx.
Comments