GigaToken: ~1000x Faster Language Model Tokenization

What happened

Marcel Roed on July 22, 2026 published GigaToken on GitHub — an open-source LLM tokenizer with extreme performance, achieved through SIMD optimizations, branch minimization, and pretoken mapping caching.

Context and impact

Tokenization is a fundamental step in LLM training and inference. GigaToken is compatible with Hugging Face Tokenizers and tiktoken APIs, so it can be adopted without changing existing pipelines. For companies training on large corpora, it could represent significant compute savings.

Details

• Speed: 500-1000x faster than Hugging Face Tokenizers
• Speed: ~100x faster than OpenAI tiktoken
• Benchmark: 5,564 Mtok/s on AMD EPYC 9565 72-Core
• All of Common Crawl (130 trillion tokens): completed in ~6.5 hours
• Compatibility: Hugging Face Tokenizers API and tiktoken API
• License: MIT, available on GitHub
• HN score: 330 points