Who Is Sycophantic to Whom? Confident LLMs, Human Deference, and the Collapse of Trust Calibration
The paper asks who adapts to whom when a confident LLM is wrong. In a within-subjects experiment, 32 participants learned science topics from GPT-5-mini assistants whose briefings were covertly falsified at 0%, 25%, 50%, 75% or 100%. The study finds that learners absorbed roughly two-thirds of stated falsehoods regardless of overall assistant reliability, while their confidence and trust ratings failed to track accuracy. By reframing trustworthiness as a property of the human-AI pair and introducing a Human Deference Index, the work argues that sycophancy evaluation must measure human-side deference as well as model-side compliance.
The paper makes a conceptually important contribution by inverting the standard sycophancy question and providing strong evidence that human deference, not model compliance, drives misinformation uptake in confident tutoring scenarios. The experimental design is careful---Latin-square assignment, transcript-coded exposure, and baseline-corrected absorption metrics---but the evidence is drawn from a small, homogeneous sample under extreme artificial conditions. Claims about the 'calculator fallacy' and the collapse of trust calibration are well-supported within these bounds, though generalisation to naturalistic settings remains uncertain.
The within-subjects manipulation cleanly separates assistant reliability from participant traits, and the transcript coding strengthens causal interpretation by distinguishing exposure from acceptance. The core finding that per-claim acceptance is insensitive to overall reliability is robust, and the dissociation between trust ratings and objective performance is striking. The Human Deference Index is a sensible methodological adaptation of model-side sycophancy metrics to the human direction.
The sample is small and selected for high AI exposure: 84% used AI chat assistants daily, which likely inflates baseline deference. The assistant behaviour was deliberately extreme---never hedging, never conceding, and systematically falsifying at rates far above deployment reality---making the absorption rates a proof-of-concept rather than a naturalistic estimate. The prohibition on outside sources artificially created the 'circular verification' the authors lament; participants may have behaved differently if independent checks were permitted. The use of immediate multiple-choice recognition tests with the exact planted falsehood as an option inflates measured absorption relative to free recall or delayed testing. Finally, transcript coding relied on an LLM-based coder without reported human inter-rater reliability, and the new Human Deference Index lacks external validation.
The evidence supports the core claims about human-side deference and metacognitive collapse. The $\mathrm{AUROC}_2$ for confidence discrimination fell from 0.70 to chance (0.52--0.53) at the two highest falsification levels, and confidence in absorbed falsehoods ($M = 8.43$) nearly matched that for correct facts ($M = 8.90$). Comparisons to model-side sycophancy benchmarks are fair and well-contextualised, though the authors occasionally overstate the novelty of 'borrowed confidence' given prior work on cognitive offloading. The odds ratio of 12.5 for choosing planted falsehoods when falsified versus correctly taught is strong, but the 95% CI [5.2, 30.2] is wide, reflecting the small sample.
Reproduction is partially enabled but blocked by several factors. The system prompt and fact pairs are provided in appendices, and the authors promise code and data, though the repository URL was not populated at submission. The model used, GPT-5-mini, is specified but its public accessibility is unclear. Critical hyperparameters (temperature, top-$p$, system prompt versioning) are not reported. The Latin-square design is clearly described, but the LLM-based transcript coding protocol would need human validation to ensure reproducibility of the key exposure-conditioned acceptance measure.
Research on sycophancy asks whether large language models (LLMs) bend towards what their users want to hear. We ask the reverse question: when an LLM is confident, consistent, and wrong, do users bend towards it? In a within-subjects experiment, 32 participants learned up to five science topics by chatting with five assistants built on the same model (GPT-5-mini). Unknown to participants, each assistant's briefing had 0, 25, 50, 75, or 100% of its facts replaced by plausible falsehoods, and every assistant spoke with full confidence. A surprise quiz after each topic offered the correct fact, the planted falsehood, and two distractors, with a confidence rating for every answer. Across 104 quizzes and 1,040 answers, scores fell from 7.4 to 2.9 out of 10, and the planted option was chosen on 60% of falsified items against 11% of correctly taught ones (odds ratio 12.5). Transcript coding showed that whenever the assistant stated a falsehood, it was adopted about 65% of the time at every falsification level. Absorbed falsehoods were held with high confidence (8.4/10), the modeled confidence advantage of right over wrong answers vanished, and the discrimination of confidence for correctness fell from an AUROC of 0.70 to chance, while post-task trust ratings barely moved. Participants tended to return to the chat more when the assistant was worse, but returning went with more absorption, not less, and 30% of all turns pasted quiz questions back to the source being checked. The assistant itself was almost never sycophantic and never conceded under pushback. We introduce a baseline-corrected Human Deference Index to quantify this human-side sycophancy and argue that trustworthiness is a property of the human-AI pair, that sycophancy evaluations should measure both directions, and that learning interfaces must make reliability legible claim by claim.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.