Cactus Hybrid: Gemma 4 Post-Trained to Know When It Is Wrong - 0.814 AUROC, 15-35% Cloud Escalation

What happened

The Cactus.compute team post-trained Google Gemma 4 2B with a 68K-parameter probe layer that reads intermediate hidden states to predict the probability of a wrong answer (p(wrong)) for each output.

Context and impact

The result enables hybrid on-device/cloud routing: simple queries run locally, while uncertain answers escalate to a larger cloud model. Only 15-35% of queries need escalation while matching frontier model performance overall. Weights are MIT-licensed.

Details

• 68K-parameter probe: minimal size, reads hidden states in real time
• AUROC: 0.814 (probe) vs. 0.549 (standard token entropy)
• 15-35% escalation rate: the rest handled by local Gemma 4 2B
• MIT license; weights available on HuggingFace
• Show HN: 152 points, 113 comments at time of publication