Kimi K3, and what we can still learn from the pelican benchmark

Main idea

Simon Willison reviews Kimi K3 from Moonshot AI (2.8T parameters) and reflects on the evolution of his "pelican benchmark" — an SVG drawing test of a pelican on a bicycle. He argues the pelican test has lost its correlation with real performance but remains useful as a "hello world" for new models.

Context

Willison uses the pelican test as a quick check with every new model — tracking both price and SVG visual quality. The post responds directly to Kimi K3's release and its extremely long reasoning (13,241 tokens per image, costing ~25 cents).

Why it matters

He warns that no simple benchmark — not even his own pelican — can capture what matters most in 2026: agentic tool-calling. For developers comparing models, this is an important methodological stance.

Details / arguments

• Kimi K3 costs $3/M input tokens — not a cheap Chinese model, priced like Anthropic Sonnet
• One pelican SVG run = 13,241 reasoning tokens = ~25 cents
• Pelican test previously correlated with performance — that correlation has weakened
• Critical missing dimension: benchmark does not assess tool-calling
• Kimi K3 weights release July 27 — until then no independent model inspection