Large language models can predict the results of social science experiments

What happened

Researchers published a Nature study in which an LLM simulated responses from representative US population samples across 70 preregistered survey experiments involving 469 experimental effects and 119,330 participants, predicting experimental outcomes.

Context and impact

Findings suggest LLMs can serve as low-cost pre-screening tools for social experiment design, accelerating hypothesis selection before expensive field studies. They also raise ethical questions: models reproduce biases embedded in training data.

Details

• Prediction correlation: r = .83 for survey experiments
• For large multi-treatment experiments: LLM r = .34 vs. experts r = .26
• Scale: 70 preregistered experiments, 469 effects, 119,330 participants
• Practical use: hypothesis pre-screening, A/B test piloting, research acceleration
• Risk: LLMs reproduce cultural and social biases from training data