DeepInfer: High-performance inference framework for DeepSeek models — now open source

What's new

• DeepInfer open-source: DeepSeek's production inference framework — the first direct look at their internal infrastructure
• DSpark: speculative decoding with confidence estimates — generates draft tokens and verifies them
• Bounded suffix replay: reduces decoder prefill work
• Async scheduling: overlaps CPU batch scheduling with device execution (enabled by default)
• Batch-invariant inference: consistent results across different batch sizes (NVIDIA only)
• Deterministic sampling: identical outputs for identical inputs (NVIDIA only)
• Structured outputs: constrained by regex, JSON schema, or EBNF grammar

Why it matters

For teams self-hosting DeepSeek models, this provides a foundation matching cloud API quality without external dependency. The first open-source look at what powers DeepSeek in production.

How to try it

Repository: github.com/deepseek-ai/DeepInfer. Hardware: NVIDIA SM100 or Huawei Ascend 950 NPUs. Supported model: DeepSeek-V4.1-Flash.