Beyond Accuracy: Disentangling Decomposition from Typed-State Quality Assurance in Multi-Agent LLM Pipelines
Multi-agent LLM pipelines are widely assumed to outperform single-call setups, yet it is unclear whether gains come from typed inter-agent contracts or merely from decomposition into specialized prompts. This paper conducts an 18-condition controlled experiment on requirements classification and MoSCoW prioritization using three small language models on the PROMISE NFR+ dataset. The core finding is that most accuracy gains are associated with decomposition and task-specific prompts rather than with Pydantic-validated typed states, though the typed contract shows other quality-assurance properties such as earlier fault containment.
A carefully controlled associative experiment makes a valuable but narrow point: for this task and model panel, F1 gains are driven by decomposition and specialized prompts ($\Delta_{\text{p}}$ up to $+0.075$), whereas typed states add only marginal and directionally inconsistent accuracy changes ($\Delta_{\text{s}} \in [-0.003,+0.013]$). The authors are admirably transparent about confounds, yet the title's 'disentangling' overstates what is fundamentally an associative upper-bound analysis rather than a causal isolation of the typed contract.
The experimental methodology is rigorous and commendably cautious. The authors employ a full 18-condition grid (3 models $\times$ 3 architectures $\times$ 2 languages), paired McNemar and Wilcoxon tests with Bonferroni correction, class-stratified bootstrap for Macro-F1, paired percentile bootstrap for Fleiss' $\kappa$, and an explicit two-call ablation that holds prompts constant while stripping the typed contract. The fault-injection campaign and the openly archived replication package on Zenodo (prompts, manifests, raw predictions, analysis scripts, tests) set a strong reproducibility standard for empirical software-engineering research.
The title promises 'disentangling,' but the authors forthrightly admit their design does not isolate mechanisms. The $\Delta_{\text{s}}$ contrast bundles validation timing, LangGraph orchestration, and confidence-aware prompting, yet confidence routing fired for only 3 of 3,750 pipeline predictions, leaving it essentially unevaluated on the main grid. Generalisation is thin: one requirements dataset, three 7--8B SLMs, and a Portuguese condition that jointly varies machine-translated text and translated prompts. The statistically significant MoSCoW consistency increment ($\Delta\kappa$) is explicitly not attributed to the typed contract by the authors, who instead point to 'execution-order and run-to-run effects that a single-run design cannot separate from the contract.'
The primary evidence is internally consistent: the direct baseline-vs-two-call comparison corroborates that decomposition yields most of the pipeline's F1 advantage. The fault-injection campaign provides clean, if synthetic, evidence on validation timing, though the authors rightly note it 'characterises where each architecture detects a fault and what that costs, not how often such faults occur.' The supplementary gpt-4.1-mini run reproduces the small-model pattern ($\Delta_{\text{s}}$ near zero), but the caveat that some conditions used a slightly different agent implementation weakens that reinforcement. Comparisons to related work are fair but brief, correctly positioning the paper against systematic reviews that note unresolved architectural attribution.
Reproduction is well-supported by manifests recording code commit, dataset MD5, prompt versions, and LangGraph version, plus raw predictions for all 36 executions. However, determinism is compromised: the fixed seed governs dataset sampling only and is not forwarded to the Ollama server, so determinism 'rests on temperature 0,' which is not bitwise deterministic. The only repeated execution used a newer Ollama release (0.35.1 vs 0.24.0), mixing run-to-run and server-version effects. Per-call JSONL traces were retained only for the two-call supplement runs, limiting deep audit of individual failures in the main grid. An independent team would need local GPU inference and exact Ollama tags to replicate.
Multi-agent LLM pipelines are often credited with gains in software engineering tasks, but it is rarely clear whether a gain stems from the typed contract between agents or from the specialized prompts that decomposition enables. We study this attribution problem in Requirements Engineering, comparing three architectures for classifying requirements as functional or non-functional and then prioritizing them with MoSCoW: a single-call baseline; a two-call variant that reuses the pipeline's specialized prompts but passes an unvalidated intermediate; and a pipeline with a Pydantic-validated typed state. The 18 conditions cover PROMISE NFR+ (625 requirements), three locally hosted small language models, and two languages. The F1 gain over the baseline is associated with the two-call, specialized-prompt variant (up to +0.075), whereas the typed state adds a small difference (-0.003 to +0.013) that is significant in one of six comparisons. Beyond accuracy, the typed pipeline contains out-of-range model outputs at the agent boundary, while the unvalidated variant fails later and drops the requirement, and it shows a small, reproducible increase in inter-model MoSCoW consistency whose cause our design cannot isolate. The study is associative: the contrasts bundle prompts, call count, and staged context, and the results come from three 7--8B models on one dataset. The implementation and replication package are archived on Zenodo (https://doi.org/10.5281/zenodo.23380444).
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.