DeepInfer: DeepSeek open-sources the engine that serves its own traffic, and the headline feature is boring on purpose
DeepSeek pushed the first public commit of DeepInfer on October 20, and it became the top Hacker News story of the week at 2,298 points and 881 comments. The repo passed 17,700 stars within two days, Apache 2.0. The README calls it the high-performance inference framework that powers DeepSeek's large-scale production services, originally built mostly on vLLM, and it ships with DeepSeek-V4.1-Flash-1017 as the default checkpoint. Link: https://github.com/deepseek-ai/DeepInfer
What is inside: attention data parallel and context parallel, MoE expert parallel with load balancing, a speculative path that drafts up to five tokens and keeps them based on a confidence schedule, structured outputs, and support for both NVIDIA and Huawei Ascend. What is not inside yet: prefill and decode disaggregation and data-parallel load balancing, the pieces that make the biggest difference at real scale. So this is the engine, not the whole factory.
The feature worth noticing is batch-invariant inference with deterministic sampling, on NVIDIA for now. Normally the same prompt can give a slightly different answer depending on what else is in the batch, because the math gets ordered differently. That sounds like a rounding error until you run an agent for 200 steps and try to work out why run 14 failed and run 15 did not. Every agent eval this year has hit the same wall: you cannot debug what you cannot replay. A frontier lab shipping reproducible serving as a default, in the open, removes one excuse.
Ascend support next to NVIDIA is the political subtext. DeepSeek is getting banned from US federal devices the same week it hands everyone, including the US, the code it uses to serve its own models. The open-source play keeps working because it is generous in the parts that cost DeepSeek nothing to give away and keeps the parts that make the factory run.
← Back to all articles
What is inside: attention data parallel and context parallel, MoE expert parallel with load balancing, a speculative path that drafts up to five tokens and keeps them based on a confidence schedule, structured outputs, and support for both NVIDIA and Huawei Ascend. What is not inside yet: prefill and decode disaggregation and data-parallel load balancing, the pieces that make the biggest difference at real scale. So this is the engine, not the whole factory.
The feature worth noticing is batch-invariant inference with deterministic sampling, on NVIDIA for now. Normally the same prompt can give a slightly different answer depending on what else is in the batch, because the math gets ordered differently. That sounds like a rounding error until you run an agent for 200 steps and try to work out why run 14 failed and run 15 did not. Every agent eval this year has hit the same wall: you cannot debug what you cannot replay. A frontier lab shipping reproducible serving as a default, in the open, removes one excuse.
Ascend support next to NVIDIA is the political subtext. DeepSeek is getting banned from US federal devices the same week it hands everyone, including the US, the code it uses to serve its own models. The open-source play keeps working because it is generous in the parts that cost DeepSeek nothing to give away and keeps the parts that make the factory run.
Comments