Understanding Issues, Causes and Solutions in Open-Source LLM-based Multi-Agent Systems
The paper mines 22,848 closed GitHub issues from 21 open-source LLM-based multi-agent systems, filtering down to 944 MAS-related reports to catalog what goes wrong in practice. Through qualitative coding, the authors map issues, causes, and solutions, finding that orchestration and workflow design dominate practitioner pain points. It matters because it replaces anecdotal MAS debugging lore with an empirical taxonomy grounded in actual maintenance traces.
The paper delivers a descriptively rich taxonomy of practitioner-reported issues, causes, and solutions in open-source LLM-based multi-agent systems. However, its reliability is weakened by a filtering pipeline that relies on a single local LLM evaluation followed by first-author manual curation without independent double-coding for the full set, and by the fact that extracted solutions are treated as effective without outcome validation. The taxonomies are plausible and useful as heuristic catalogs, but should be read as observational mappings rather than rigorous causal findings.
The authors collect a large initial corpus and apply transparent inclusion criteria to select 21 active, popular projects, which lends breadth to the sample. The resulting taxonomies are granular and well-illustrated with direct practitioner quotes, and the issue-cause-solution mappings offer a structured lens for framework developers. Notably, the prominence of Orchestration & Execution Issue and Optimize Workflow aligns with known coordination bottlenecks in multi-agent systems, giving the taxonomies face validity.
The study conflates issue frequency with importance by treating all issues equally regardless of project scale, so 77 CrewAI issues and 6 ChatDev issues carry the same analytic weight despite vastly different user bases. Category boundaries are also blurry: Context Loss appears under both Orchestration & Execution Issue and Memory & State Management Issue with overlapping definitions, suggesting the taxonomy may not be mutually exclusive. Moreover, analyzing only closed issues systematically excludes unresolved and potentially severe open problems, biasing findings toward more tractable failures and away from deep architectural defects.
The evidence supports descriptive claims about commonly discussed pain points, but the paper never establishes a counterfactual—there is no comparison against single-agent LLM systems or traditional non-LLM multi-agent codebases—so it remains unclear which problems are distinctive to LLM-based MAS. The authors assert a lack of prior empirical studies in this space, yet they do not systematically map their taxonomy onto existing software engineering defect classifications to demonstrate novelty. Consequently, many categories (e.g., Dependency Conflict, Parameter Configuration Error) read as generic software engineering issues recast in agent terminology.
Reproduction is hampered by missing artifacts: the DeepSeek-R1 prompts, temperature settings, and Ollama configuration are not disclosed, and the MS Excel dataset is referenced but not linked or archived in the paper text. The formal extraction was performed by a single author with post-hoc review rather than independent parallel coding, and inter-rater reliability is reported only for a 500-issue pilot, not for the final 944-issue taxonomies. Because GitHub issue trackers evolve continuously, the lack of versioned issue dumps or permanent identifiers for the analyzed issues makes independent replication nearly impossible.
With the advancement of LLM-based multi-agent systems (MAS), an increasing number of opensource projects are adopting multi-agent architectures as the foundation of their core functionality. Although research and practice on MAS have attracted considerable attention, limited studies have explored the challenges faced by practitioners of open-source LLM-based MAS, the causes of these challenges, and potential solutions. To address this gap,we conducted an empirical study to understand the issues that practitioners encounter when developing and using open-source LLM-based MAS, the possible causes of these issues, and potential solutions. We collected 22,848 closed issues from 21 open-source LLM-basedMASand applied a mixed automated and manual filtering approach to reduce the dataset to 944 issues related to LLM-based MAS.We then analyzed these issues to understand the frequent issues encountered by practitioners, their underlying causes, and potential solutions. Our study results show that (1) Orchestration & Execution Issue is the most common issue faced by practitioners, (2) Workflow Problem, Tool Integration Problem, and Memory Problem are identified as the most frequent causes of the issues, and (3) Optimize Workflow is the predominant solution to the issues. Based on the study results, we derive empirically grounded implications for practitioners and researchers aimed at improving orchestration, tool integration, and memory mechanisms in LLM-based MAS.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.