More Isn't Always Better: Balancing Decision Accuracy and Conformity Pressures in Multi-AI Advice

The paper's core empirical finding—that decision accuracy saturates at three advisors—holds robustly across multiple tasks and represents a practical design insight for multi-AI systems. The use of a Rashomon set (decision trees sampled from a random forest with equivalent $70\%$ accuracy but diverse predictions) provides excellent experimental control over advisor competence while naturally generating consensus and disagreement patterns. The gradient of consensus effects is particularly compelling: unanimous agreement drives overreliance (Agreement Fraction $0.97$ in consensus vs. $0.73$ in divergence), single dissent reduces conformity pressure, and $3\text{--}2$ splits create decision paralysis. The methodology explicitly distinguishing informational from normative conformity through survey items $Q_3$ and $Q_6$ adds theoretical rigor.

The study suffers from significant external validity limitations. Participants were Japanese crowdworkers (mean age $44.5$), and the authors acknowledge that 'conformity varies by culture,' yet make broad design claims without cross-cultural validation. More critically, the 'AI advisors' were decision trees from random forests presented through an LLM-generated explanation layer—not actual LLM agents. This abstraction misses critical phenomena in real multi-LLM consultation where models may have correlated errors, shared training data, or conversational dynamics. The human-likeness manipulation (Study 2) showed null effects on accuracy despite successful manipulation checks, suggesting either insufficient power ($n=88$ across three tasks) or that the chosen anthropomorphic cues (static photos, names) lack ecological validity compared to voice or interactive persona. The authors' claim that 'panel consensus can elicit informational conformity' relies on correlational evidence (Table 6) where $R \approx 0.42$ for pressure-reliance relationships—directionality remains unclear.

The paper adequately positions itself against prior work, contrasting its discrete-panel approach with Lu et al.'s sequential second-opinion paradigm and Song et al.'s (2024) finding that informational conformity may not arise with AI panels. However, the comparison to Song et al. is somewhat strained—the current study uses accuracy-grounded tasks where correctness is verifiable, while Song et al. examined opinion change on societal issues without ground truth. The claim that 'simple statistical aggregation usually outperforms humans' deliberative selection' is supported by classic literature (Soll and Larrick, 2009), yet the paper does not empirically test whether participants would perform better with an explicit majority-vote aggregation compared to the sequential advice presentation used. The relationship to Asch conformity studies is well-articulated, though the translation from perceptual line-judgment tasks to predictive inference merits deeper theoretical justification.

Reproducibility is moderately supported but incomplete. The authors specify using GPT-4o via the OpenAI API for explanation generation with detailed prompts (Figure 3), and the Rashomon set construction from random forests is algorithmically precise ($70\%$ accuracy selection). However, the paper makes no mention of code, data, or stimuli availability—neither the decision tree models, the specific feature attributions, nor the explanation generation prompts are publicly archived. The attention-check methodology (selecting features highlighted by AI) is described but not validated for efficacy. Hyperparameters for the random forest (number of trees, feature sampling) are unspecified. Without access to the exact tree structures or the GPT-4o prompt templates, independent reproduction of the advice stimuli would be impossible. The study protocol appears rigorous (three attention checks, comprehension quizzes), yet the lack of open materials significantly impedes replication efforts.