Abstract Priorities, Concrete Rewards: How Stated Preferences Relate to What People Reward in AI Responses
This paper investigates why user-stated preferences for AI responses often fail to predict their actual ratings. Using the public PRISM dataset, the authors show that the gap between stated and revealed preferences largely disappears when attributes are measured via aggregate human perceptions rather than simple text proxies, yet concrete features like length dominate ratings regardless of what users claim to value. The findings suggest that personalization systems should treat preference elicitation format as a design parameter matched to the decision at hand, not as a direct window into stable, latent utility functions.
The paper is a carefully executed secondary analysis that makes a genuine conceptual advance: it reframes the stated–revealed preference gap in LLM evaluation as a measurement-dependence problem rather than a simple failure of user insight.
The core moderation analyses are well-specified, cross-validated, and transparently reported, and the authors commendably distinguish population-level stability from individual-level recoverability. Nevertheless, the observational design, private pre-analysis plan, and proliferation of exploratory comparisons limit the strength of causal and confirmatory claims, while the tiny predictive gains raise questions about near-term practical utility.
The three-way comparison of attribute representations—designed text proxies, lens composites, and aggregate attribute profiles—is the study's strongest contribution, clearly demonstrating that operationalization determines whether stated priorities appear aligned with behavior.
The coding of spontaneous concrete feedback is also compelling: participants who explicitly requested shorter or longer answers in earlier conversations showed corresponding shifts in their length weights, establishing an ecologically valid link between open-ended user language and a concrete feature. Additionally, the variance-component analysis honestly documents a large person-level standard deviation for length ($0.17$) alongside near-zero split-half reliability, separating the existence of heterogeneity from its practical exploitability.
The biggest limitation is the observational design: perceptions and reasons were collected after the full conversation and referred only to the chosen response, so their tight coupling may reflect post-choice reconstruction rather than independent drivers of preference.
Because aggregate attribute profiles are built from other raters' post-conversation reports of chosen responses, they are vulnerable to selection and common-quality confounding even after echo correction. Furthermore, the pre-analysis plan was not publicly registered, and a large share of the reported analyses—including the lens composites, concrete feedback coding, and prediction contest—were exploratory, increasing the risk of capitalization on chance despite robustness checks.
The evidence strongly supports the central claim that alignment with stated priorities is sensitive to how attributes are measured: text proxies were uncorrelated with priorities, while aggregate profiles showed robust, attribute-specific moderation even after removing common perceived quality. Comparisons to related work are accurate: the authors' characterization of PRISM-X and LiteraryTaste matches the papers' own abstracts.
However, the practically negligible improvement in held-out pairwise accuracy from personal information ($0.13$ percentage points) versus a stronger pooled model ($1.11$ points) suggests that the existence of stable individual differences, which the paper establishes, does not yet translate into actionable personalization gains under the tested estimators.
Reproducibility is currently impaired by the absence of open artifacts: the code and data are only available upon request and will be released after review, leaving independent verification impossible for now.
The paper compensates with unusually detailed methodological description—feature lexicons, echo-correction formulas, and nested cross-validation are spelled out—but many critical hyperparameters (e.g., gradient-boosting depth, winsorizing thresholds) are not reported, and the complex participant-bootstrap pipeline for prediction would be hard to replicate from text alone. The relatively small amount of data per participant (median six conversations) means that even with perfect code, independent reproduction would likely recover the same poorly estimated individual slopes.
AI assistants built on large language models (LLMs) are trained on human feedback and increasingly personalized to users' stated preferences. Yet how closely these preferences reflect what people reward in responses remains unclear. We compare stated and revealed preferences in the public PRISM dataset, in which 1,396 participants reported how much seven response attributes mattered to them and then rated 29,347 responses from 21 LLMs. Among responses to the same prompt, features the survey never asked about, such as response length, formatting and refusal language, were at least as strongly associated with ratings as text proxies of the surveyed attributes. Whether stated priorities matched ratings depended on how attributes were measured: participants who prioritized an attribute favoured responses from models that others perceived as strong on it, but not responses that scored higher on text proxies for it. Written feedback was more telling: participants who asked for shorter answers, even in earlier conversations, gave length less weight in their ratings, and those asking for longer answers gave it more. Their relative priorities over the surveyed attributes, however, did not predict this weight. Reasons given for choices reflected perceived attributes more than measured features. Participants also differed stably in how much they weighted several concrete features, especially length, but with a median of 6 conversations per participant, these weights were estimated poorly. The largest gain from personal information in predicting held-out choices was 0.13 percentage points, compared with 1.11 for a stronger pooled model. Stated priorities thus track perceived attributes of AI responses but say little about the concrete features that ratings reward, a distinction that matters for preference elicitation and AI personalization.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.