Evaluating LLM-Generated Preference Distributions
This paper asks whether Large Language Models can serve as stable, implicit world priors for simulation and decision support when real-world data are unavailable. The authors introduce a framework to evaluate full LLM-generated probability distributions over discrete choice sets—airports, restaurants, and consumer products—across nine open-weight models, three families, and repeated sampling with controlled prompt perturbations. Their central finding is that model scale and family introduce far larger distributional shifts than prompt wording, with modes stabilizing rapidly but encoding structurally different priors across models. This directly challenges the assumptions underlying recent silicon-sampling work that treats LLMs as interchangeable stand-ins for human respondents.
The paper delivers a methodologically careful empirical characterization of LLM distributional behavior. Its core contribution—the demonstration that scale discordance exceeds prompt sensitivity—is important and well-supported for distant scale pairs, though less resolved for adjacent large checkpoints where confidence intervals overlap with the largest prompt effects. The reliance on verbalized elicitation is pragmatic but introduces a known fidelity limitation that the authors appropriately flag.
The modes-first convergence pattern is robustly documented across all nine models and three domains: high-probability top-$k$ outcomes stabilize within a few samples while residual variability concentrates in the tail, with tail $\mathrm{JS}$ divergence roughly double mode divergence post-convergence ($\mathrm{JS}_{\text{tail}}$ 0.124–0.132 versus $\mathrm{JS}_{\text{mode}}$ 0.058–0.059). The extensive cross-family validation and multi-pronged robustness checks—temperature ablation, greedy decoding, permutation analysis, and choice-set size variation—substantially strengthen the claim that this behavior reflects intrinsic model properties rather than artifacts of the elicitation protocol.
Sample size is a limiting factor: with only twenty personas, bootstrap intervals for cross-scale divergences are wide enough that the 70B-versus-405B gap ($\mathrm{JS}=0.36\pm 0.04$) overlaps with the age-perturbation effect ($\mathrm{JS}=0.33\pm 0.05$), so the claim that model choice dominates prompt design is strongest for distant scales and weakened for adjacent large checkpoints. The use of verbalized elicitation—despite known inferiority to logit-based readouts for probability fidelity—may introduce unmeasured systematic distortions, and single-context point estimates for Qwen and Mistral are not directly comparable to Llama's persona-averaged statistics. Finally, the restriction to open-weight models and the absence of any ground-truth human benchmark leave open whether these implicit priors correspond to real preference structures.
The empirical evidence supports the main claims about convergence and scale discordance, and the comparisons to related work are fair and accurate. The authors correctly distinguish their distribution-level analysis from prior evaluations of single-output accuracy or confidence calibration, and they accurately cite Santurkar et al. (2023) as reporting that misalignment between LLM opinion distributions and US demographic groups persists even under explicit demographic steering. The chosen metrics—$\mathrm{JS}$ divergence, Jaccard overlap, and Spearman correlation—are well-suited to the discrete setting. The study does not, however, collect ground-truth human preferences, so it cannot adjudicate which model prior is more valid, only that they differ structurally.
Reproducibility is generally strong. The paper specifies exact model checkpoints (e.g., Llama-3.3-70B-Instruct-Turbo), decoding parameters ($T=0.2$, $R_{\max}=20$), API provider (Together AI), and provides full prompt templates, parsing pipelines, and normalization procedures in the appendices. The authors state that code and processed data will be made publicly available. The main barriers to independent reproduction are the computational cost of querying the largest checkpoints and the complexity of the free-form JSON parsing pipeline, which requires post-hoc rectification and code-point normalization that could deviate across implementations unless the released code is used verbatim.
Large Language Models (LLMs) are increasingly used as probabilistic generators for simulation, synthetic data generation, and decision support in settings where real-world data are unavailable. Yet, the structure and reliability of the distributions they produce remain understudied. Here, we systematically analyze LLM-generated distributions of preferences for air travel, restaurants, and consumer products. Encouragingly, all models considered in our analysis exhibit self-coherence, with the most probable outcomes stabilizing rapidly under repeated sampling. At the same time, we observe substantial discordance across both model families and scales, with little consensus even among their most probable outcomes. These patterns hold across nine open-weight models, three choice domains, and show robustness under temperature changes, greedy decoding, and perturbations of prompt and ordering. Our findings indicate that outcomes are influenced more by the choice of model than by the wording of the prompt, challenging the common assumption that sufficiently capable LLMs produce similar preference distributions when used as stand-ins for survey respondents.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.