The new GPT-5.6 family: Luna, Terra, Sol — OpenAI Challenged SWE-Bench Pro Reliability After Underperforming

Main idea

Willison analyzes three new GPT-5.6 models (Luna, Terra, Sol) and highlights a concerning pattern: OpenAI publicly challenged the reliability of SWE-Bench Pro precisely when GPT-5.6 underperformed on it relative to competitors. OpenAI claims ~30% of the benchmark's tasks are 'problematic.' Willison doesn't allege direct manipulation but names a pattern worth watching.

Context

Willison has long focused on critical evaluation of AI benchmarks and is one of the most trusted independent AI commentators. This post was published on the day GPT-5.6 launched (July 9, 2026). He also notes technically interesting new API features — programmatic tool calling, multi-agent orchestration, prompt cache breakpoints.

Why it matters

The pattern of 'underperform on a benchmark → challenge the benchmark' is critically important for developers and ML engineers who rely on SWE-Bench Pro for model selection. If ~30% of tasks are genuinely flawed, entire comparison leaderboards need reinterpretation — including historical comparisons.

Details / arguments

• GPT-5.6 Luna ($1/$6), Terra ($2.50/$15), Sol ($5/$30) per million tokens
• New API: programmatic tool calling, multi-agent orchestration, prompt cache breakpoints
• OpenAI claims ~30% of SWE-Bench Pro tasks are 'problematic'
• Sol scored 53.6 on Agents' Last Exam vs. 40.5 for Fable 5 — strong result on agentic benchmark
• 1M token context window, February 2026 knowledge cutoff