GigaToken Makes Tokenization 1000x Faster, Which Nobody Thought Was a Problem
Twenty-four gigabytes per second. That is GigaToken doing GPT-2 tokenization on an AMD EPYC, roughly 989 times faster than HuggingFace's tokenizers and 681 times faster than tiktoken. Marcel Roed wrote it in Rust with SIMD and cache-hierarchy work, MIT licensed, and it landed on Hacker News at 274 points in four hours.
Tokenization was supposed to be a solved rounding error. It is the boring step between text and the model, and everyone assumed the GPU was the bottleneck so the CPU side did not matter. Three orders of magnitude says otherwise. When you are tokenizing a multi-trillion-token pretraining corpus, or re-tokenizing a codebase on every agent turn, or serving a retrieval pipeline that chews through documents to decide what to put in context, the CPU work is real and it has been quietly taxing everyone.
The agent angle is context management. An agent loop does not tokenize once, it tokenizes constantly, because every decision about what fits in the window requires counting. Trimming history, budgeting tool outputs, deciding whether a file gets summarized or pasted, scoring retrieval candidates. All of that is token counting on the critical path of a request, and today most harnesses either eat the latency or cheat with character-count heuristics that are wrong in ways that bite you at the boundary.
It ships HuggingFace and tiktoken compatibility modes, so this is a drop-in, not a rewrite. That is the part that decides whether an optimization like this actually propagates. The lesson here is broader than tokenizers: the AI stack was built fast, and a lot of it is a first draft nobody profiled. Roed profiled one piece and found a thousand-fold. There are more of those sitting around.
https://github.com/marcelroed/gigatoken
← Back to all articles
Tokenization was supposed to be a solved rounding error. It is the boring step between text and the model, and everyone assumed the GPU was the bottleneck so the CPU side did not matter. Three orders of magnitude says otherwise. When you are tokenizing a multi-trillion-token pretraining corpus, or re-tokenizing a codebase on every agent turn, or serving a retrieval pipeline that chews through documents to decide what to put in context, the CPU work is real and it has been quietly taxing everyone.
The agent angle is context management. An agent loop does not tokenize once, it tokenizes constantly, because every decision about what fits in the window requires counting. Trimming history, budgeting tool outputs, deciding whether a file gets summarized or pasted, scoring retrieval candidates. All of that is token counting on the critical path of a request, and today most harnesses either eat the latency or cheat with character-count heuristics that are wrong in ways that bite you at the boundary.
It ships HuggingFace and tiktoken compatibility modes, so this is a drop-in, not a rewrite. That is the part that decides whether an optimization like this actually propagates. The lesson here is broader than tokenizers: the AI stack was built fast, and a lot of it is a first draft nobody profiled. Roed profiled one piece and found a thousand-fold. There are more of those sitting around.
https://github.com/marcelroed/gigatoken
Comments