Beyond Final Accuracy: Auditing Communication in LLM Multi-Agent Systems
Multi-agent LLM systems are usually judged by end-task accuracy, which conflates genuine information sharing with extra reasoning or fortunate architecture. This paper introduces Independent–Communicate–Revise (ICR), a controlled audit that freezes each agent’s initial trajectory and measures answer revision conditional on both agents’ initial correctness. By separating correction from preservation and comparing against a no-message control, the work shows that aggregate accuracy can hide large, policy-dependent trade-offs between fixing errors and corrupting correct answers.
The paper makes a timely, well-motivated argument that aggregate accuracy is a poor proxy for communication quality in LLM multi-agent systems. Its protocol—disentangling communication-driven revision from independent reasoning via fixed trajectories, correctness-conditioned metrics, and a no-message baseline—is conceptually clean and carefully executed, though the empirical scope is narrow and several key causal comparisons remain unresolved. "Similar aggregate accuracy can conceal substantially different revision behaviors" (Abstract). "ICR therefore recenters evaluation on selective revision, providing a unified framework for examining how message content and receiver policies jointly produce benefits and harms" (Section 5).
The strongest element is the controlled revision protocol itself. By caching independent trajectories and reusing them across all communication conditions, the authors isolate the effect of the message from architecture and generation variance. "Agents first solve each question independently. Their initial trajectories are then fixed and reused across communication conditions, with each ordered sender–receiver pair evaluated separately" (Section 3.1). The no-message baseline is a sharp design choice that credibly separates communication gains from mere additional reasoning, directly addressing the concern that "intrinsic self-correction can fail to improve reasoning and may degrade initially correct responses" (Huang et al., 2024, as cited in Section 2).
The empirical claims outrun the evidence in several places. The study relies on a single small model family (Qwen3-4B and 8B) and only four main reasoning benchmarks, with very few mixed-correctness questions in some strata; on GPQA-D, for example, the CR and PR denominators are just 116 events each, and the both-incorrect recovery rate is near zero across all conditions. "Recovery when both agents are initially wrong also remains rare in our experiments" (Section 1). Moreover, the authors acknowledge that "ICR standardizes the revision experiment and audit targets while retaining channel-specific interfaces and computation. Its conclusions therefore concern the evaluated channel–policy combinations, rather than an intrinsic ranking of communication representations" (Section 3.3), yet some framing still implies broader lessons about latent versus textual communication that the data do not fully support. The estimated full-set accuracy also rests on strong extrapolation assumptions: "The Qwen3-4B estimates generally extrapolate observed all-three-correct accuracy to unobserved directions; the Qwen3-8B estimates instead assume unchanged correctness on all directions outside the mixed-correctness population" (Appendix G).
The evidence clearly supports the central claim that final accuracy masks divergent correction–preservation trade-offs: "On GSM8K, estimated full-set accuracy ranges from 94.68% to 94.79% across the evaluated revision conditions, a spread of only 0.11 percentage points. Among the communicating conditions, however, PR ranges from 46.74% to 73.91%, a gap of 27.17 points" (Section 4.2). However, the comparisons between textual and latent channels are not fully apples-to-apples. The latent implementations are adaptations rather than reproductions of their native pipelines: "These implementations differ in both the sender information represented and its delivery... The audit therefore evaluates these message–interface combinations under specified receiver policies, rather than reproducing the methods’ complete native agent workflows" (Appendix B). In addition, the random-delivery reference intended to test whether Answer Only outperforms a content-independent reduction in message exposure yields inconclusive results: "All four intervals include zero, leaving the CR differences at matched preservation unresolved" (Section 4.3), suggesting the message-content analysis is not yet decisive.
The authors provide an anonymous code repository and extensive generation settings, but independent reproduction faces practical obstacles. "Anonymous source code and supporting materials for reproducing the reported experiments are available at https://anonymous.4open.science/r/AgentICR-AA82/" (Reproducibility statement). Yet critical dataset provenance is incomplete: "MedQA uses all entries in a fixed local file containing 300 questions; the original subset-selection procedure and source-question identifiers are unavailable" (Appendix C), and GPQA-D relied on a local conversion with an unavailable option-permutation procedure. The scoring pipeline for HumanEval+ uses a custom execution harness rather than the official EvalPlus one (Appendix C), and the authors note that "exact source snapshots for every historical execution were not archived, so a single commit does not fully specify all evaluated runs" (Appendix A.4). Together with deterministic GPU algorithms not being enforced, these gaps mean that exact reproduction of the reported numbers is not guaranteed.
Multi-agent communication aims to help agents benefit from one another's information. Yet improvements in system performance leave a fundamental ambiguity: do they reflect effective communication, a favorable agent architecture, or simply additional reasoning? Because communication methods are commonly evaluated within the systems they were designed for, these factors are difficult to disentangle. Final accuracy further merges corrected errors and corrupted answers into a single outcome, obscuring how communication changes decisions. We introduce Independent--Communicate--Revise (ICR), a controlled framework that evaluates communication as answer revision following independent reasoning. ICR fixes initial reasoning trajectories, measures correction and preservation conditional on both agents' initial correctness, and uses a no-message revision control to quantify gains beyond additional reasoning. Across four reasoning benchmarks, our audit of textual and latent communication reveals that similar aggregate accuracy can conceal substantially different revision behaviors. Compared with transmitting answers alone, full reasoning increases correction while reducing preservation on all four benchmarks, so richer messages amplify beneficial and harmful influence alike. Receiver-policy comparisons on MedQA and GPQA-D further show that a structured verification policy shifts every channel toward greater preservation and lower correction, while its effect on selectivity varies across channels and tasks. These findings challenge treating communication quality as an intrinsic property of a channel. ICR therefore recenters evaluation on selective revision, providing a unified framework for examining how message content and receiver policies jointly produce benefits and harms.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.