Databricks benchmarks coding agents on its own multi-million line codebase: GLM-5.2 on the Pareto frontier

What happened

Databricks published results from internal coding agent benchmarks on July 8, 2026, evaluating agents on real tasks their engineers performed on the company's codebase. The codebase contains millions of lines in Python, Go, TypeScript, Scala, and other languages.

Context and impact

Public benchmarks like SWE-Bench are often over-tuned, so Databricks took actual engineering tasks and created test suites for them. The practical message: buyers should compare coding agents by cost per completed task, not just token pricing. GLM-5.2 on the Pareto frontier shows Chinese open-source models are a strong argument for cost-conscious enterprises.

Details

• Benchmark: real engineering tasks from Databricks codebase (Python, Go, TypeScript, Scala)
• Pareto frontier: OpenAI, Anthropic, and GLM-5.2
• GLM-5.2 is strong and cheaper — a challenge for US models
• Measures: ability to solve end-to-end tasks on real codebase
• Matei Zaharia (Databricks co-founder): custom benchmarks needed for company-specific coding tasks
• Hacker News top AI story July 9 (score 108)