Research
Synthetic Panels: Can an LLM Predict Consumer Response?
Language models will answer any survey question you put to them. The question is whether those answers correlate with what a real panel would have said, and where the correlation breaks.
Mohit Luthra
Draft
~12 min
This post is in progress. The outline below is what it will cover — the full version lands shortly.
Testing language models against real survey panels on brand preference tasks.
What this post will cover
- Building a synthetic panel with demographic conditioning
- Isotonic calibration against a held-out real panel
- The honesty mechanic: refusing to calibrate when calibration does not improve