Synthetic Decision Panel Report

Match persona sources to the decision

Lee, Dongryeol; Łajewska, Weronika; Perelli, Leonardo; Mansour, Saab (2026). Data-Driven Personas for Survey Simulation: Insights into Simulation Alignment Across Data-Access Regimes. arXiv preprint; accepted at REALM EMNLP 2026. Submitted: 5 October; revised: 6 October 2026.

Research finding

Across five survey datasets, eight persona sources and three models, unrelated-source personas rarely beat simple demographic context; correct group assignment and relevant history helped.

Behavioral history and persona continuity

Pourtaheri, Khashayar; Zareei, Ahmad (2026). Behavioral History Outperforms Descriptions of the Person for LLM Synthetic Personas. arXiv preprint. Submitted: 2 October 2026.

Research finding

In an 845-person two-wave study, past answers improved individual prediction more than demographic, personality and cognitive descriptions, while substantial individual signal remained missing.

Evaluate outcome types and inference cost separately

Huang, Jin; Ferreras Garrucho, Diego; Xie, Yutong; Yuan, Walter M.; Mei, Qiaozhu; Lian, Chen; Hazell, Jonathon (2026). HouseholdBench: Evaluating Large Language Models as Predictors of Household Economic Behavior. arXiv preprint. Submitted: 6 October 2026.

Research finding

A 32-task benchmark compared 13 models with persistence and boosted-tree baselines. Fine-tuning plus prediction aggregation helped a small open-weight model on selected tasks; gains were not universal.

Make calibration assumptions inspectable

Gonzalez-Bonorino, Augusto; Biriukova, Kseniia; Capra, Monica (2026). Synthetic Cultural Agents from Aggregate Anchors. arXiv working paper. Submitted: 5 October 2026.

Research finding

Group-specific adapters retained declared aggregate preference signals, but transfer to new wording and agreement with human distributions were distinct and sometimes divergent.

Explicit context for synthetic personas

Argyle, L. P.; Busby, E. C.; Fulda, N.; Gubler, J. R.; Rytting, C.; Wingate, D. (2023). Out of One, Many: Using Language Models to Simulate Human Samples. Political Analysis, 31(3), 337–351. Online: 21 February 2023.

Research finding

Conditioned GPT-3 samples reproduced some patterns in studied US survey populations.

Keep simulation separate from population evidence

Bisbee, J.; Clinton, J. D.; Dorff, C.; Kenkel, B.; Larson, J. M. (2024). Synthetic Replacements for Human Survey Data? The Perils of Large Language Models. Political Analysis, 32(4), 401–416. Online: 17 May 2024.

Research finding

Synthetic survey responses showed compressed variation and sensitivity to prompts and time.

Panel size: 10 profiles, plus 2 control repeats

Guest, G.; Bunce, A.; Johnson, L. (2006). How Many Interviews Are Enough? An Experiment with Data Saturation and Variability. Field Methods, 18(1), 59–82. February 2006. · Hennink, M.; Kaiser, B. N. (2022). Sample sizes for saturation in qualitative research: A systematic review of empirical tests. Social Science & Medicine, 292, 114523. January 2022.

Research finding

In qualitative research with purposive samples, new themes stopped appearing within the first twelve interviews; a systematic review of 23 studies found saturation at 9–17 interviews for homogeneous populations and narrow objectives.

Profiles chosen by quota on what drives the decision

Guest, G.; Bunce, A.; Johnson, L. (2006). How Many Interviews Are Enough? An Experiment with Data Saturation and Variability. Field Methods, 18(1), 59–82. February 2006.

Research finding

Purposive samples are the most common form of non-probabilistic sampling in qualitative research: participants are chosen for what they can reveal, not drawn at random.

Each profile sees the options in its own order

Krosnick, J. A.; Alwin, D. F. (1987). An Evaluation of a Cognitive Theory of Response-Order Effects in Survey Measurement. Public Opinion Quarterly, 51(2), 201–219. 1987.

Research finding

In a split-ballot survey experiment, the order in which answer choices were presented changed the answers (a primacy effect).

The decision rule is fixed before the panel runs

Nosek, B. A.; Ebersole, C. R.; DeHaven, A. C.; Mellor, D. T. (2018). The preregistration revolution. Proceedings of the National Academy of Sciences, 115(11), 2600–2606. March 2018.

Research finding

Defining the questions and the analysis plan before seeing the outcomes separates tested predictions from explanations fitted after the fact.

Forced choice of the best and the weakest option

Finn, A.; Louviere, J. J. (1992). Determining the Appropriate Response to Evidence of Public Concern: The Case of Food Safety. Journal of Public Policy & Marketing, 11(2), 12–25. September 1992.

Research finding

The study that introduced best–worst choice into marketing research: respondents name the most and the least important item in a set, which yields a relative order rather than separate ratings.

Objections are asked before scores

Lord, C. G.; Lepper, M. R.; Preston, E. (1984). Considering the opposite: A corrective strategy for social judgment. Journal of Personality and Social Psychology, 47(6), 1231–1243. December 1984.

Research finding

In two experiments, asking people to consider the opposite possibility corrected biased judgment better than asking them to be fair.

Score weights, winning margin and model mix

The method's own rule

The method sets the weights and the margin per type of decision and fixes them before each run: the criterion that matters most for that type of decision counts double, and a difference below the margin is treated as a tie.

Control product, differences instead of levels, agreement between models

The method's own rule

A product choice of the method, not validated on people: an everyday control product is asked the same way as the tested options, the panel is never shown a list of obstacles to rate, an option is placed against another only as a paired difference, and a difference on which the AI models point in different directions is not reported.