PerSo: Investigating Facets of LLMs' Social Intelligence in Real-World Persuasion Scenarios
PerSo introduces a benchmark for evaluating LLM social intelligence through real-world persuasion dialogues from Reddit's ChangeMyView. It tests cognitive intelligence by forecasting a persuadee's beliefs, emotions, and intents, and behavioral intelligence by selecting the more persuasive response. Unlike prior work that often relies on synthetic scenarios or isolates these two aspects, PerSo jointly evaluates both on naturally occurring interactions, providing a more ecologically valid testbed.
PerSo offers a conceptually well-motivated benchmark distinguishing cognitive from behavioral social intelligence and rooting both in real persuasion data. However, empirical claims are weakened by a critically small human reference sample and LLM-in-the-loop task generation that risks construct circularity. The paper asserts that the strongest model achieves "76% accuracy, only 1 percentage point above the best human participant at 75%," yet this comparison rests on just three participants, and the authors themselves caution that "Our human reference results are based on only three participants, which limits the statistical reliability of any comparison between human and model performance."
The theoretical framing and task design are solid. The forecasting formulation—predicting mental states before seeing the reply—is well justified as harder than recognition and aligns with prior work on conversational emotion forecasting. The non-LLM baselines confirm the tasks are not trivial: the SBERT similarity baseline "achieves 21.00%, 23.25%, and 11.75% on belief, emotion, and intent respectively, all below the 25% chance level," and surface-feature classifiers barely beat a length heuristic. The consistency of emotion as the hardest dimension for both models and annotators adds credibility: "emotion is consistently the most difficult dimension for all models... at or below chance."
The most serious limitation is the human evaluation size. With only three annotators, the best-human ceiling is an unreliable point estimate. As the authors acknowledge, "Our human reference results are based on only three participants, which limits the statistical reliability of any comparison between human and model performance." Additionally, the behavioral task is heavily confounded by reply length: "a length heuristic that always selects the longer reply (65.5%)" outperforms most small models, and while filtering extreme ratios modestly reduces scores, the confound is not eliminated. Generalizability is also narrow, covering only single-turn English dialogues from a specific subreddit community.
The literature comparison in Table 1 is accurate: external sources confirm SOTOPIA uses simulated scenarios and FaNToM uses LLM-generated conversations. For instance, SOTOPIA presents "an open-ended environment to simulate complex social interactions between artificial agents," while FaNToM is built by automatically generating "full conversations using the large language model (LLM) InstructGPT davinci-003." The scale-performance trend is mostly supported, though DeepSeek-V4-Pro's regression on the behavioral task shows scale is not monotonic. A notable gap is the absence of widely used models such as GPT-4 or Llama, which limits the paper's relevance.
Reproducibility is hampered by the lack of a public code or data release statement in the paper. While hyperparameters are reported—"we set the temperature to 0 and top-k to 1 across all experiments"—critical details such as random seeds, specific API versions, and inference frameworks are missing. The dataset construction pipeline depends on a proprietary model (Gemini 2.5 Pro), which may not be accessible to others at the same version. Although "The complete prompt templates are provided in Appendix A.9," independent replication of the exact evaluation items remains uncertain.
Understanding the social intelligence of large language models (LLMs) is increasingly important for applications involving human-AI interaction and multi-agent environments. Social intelligence comprises two aspects: cognitive intelligence, the ability to infer others' mental states, and behavioral intelligence, the ability to select appropriate social actions based on these inferences to achieve social goals. Persuasion provides a natural testbed in which both aspects are simultaneously at play, yet prior benchmarks typically evaluate them in isolation using synthetic scenarios. We introduce PerSo, a benchmark that evaluates both aspects on real-world persuasion dialogues from Reddit's ChangeMyView community, using an LLM-in-the-loop pipeline to construct the evaluation tasks, validated through human annotation. Our evaluation of diverse LLMs shows that performance generally scales with model size but varies across dimensions. The strongest LLM slightly surpasses the best observed human performance on the behavioral task, while the best human participant remains ahead on the cognitive task.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.