Pay for the Fault, Not the Flow: Label-Free In-Flow Multi-Agent Workflow Optimization
InFlowOp attacks the costly cycle of building and fixing multi-agent LLM workflows by pricing every decision in a single label-free cost. The framework first constructs a workflow at the right granularity via bidirectional cost-driven coalescing, then detects and repairs faults locally during execution without reference answers. The authors also introduce Braid, a benchmark designed to demand multi-agent coordination, and report accuracy gains up to +11.97% over a single agent across eight domains and six backbones.
The paper proposes a clever two-stage, label-free framework for multi-agent workflow optimization. The conceptual split between bidirectional coalescing and run-time local fault repair is elegant, but the empirical story is front-loaded: "Coalesce reaches 22.21%, +6.72% above greedy search and +4.83% above the single-agent baseline. In-flow tempering adds a further +1.59%, and scoring each agent on the evolving profile improves +1.33% more." As the authors themselves observe, "the two (3)-(4) optimizing in-flow earn less than the one deciding the workflow before it runs." (Section 5.2) Combined with an unvalidated rubric estimator and an internally constructed benchmark, the case for "paying for the fault" is weaker than the architecture suggests.
The strongest contributions are the bidirectional cost-driven coalescing algorithm and an unusually honest ablation showing that naive multi-agent decomposition can actively harm performance. As Table 2 reveals, "a workflow constructed greedily obtains 15.49%, falling below the single-agent's 17.38%. Therefore, dividing a task is not by itself what earns the gain." The label-free fault-detection mechanism is also genuinely novel: "where construction matches an atom's demands against an agent's card, in-flow detection matches each declared condition against the work actually realized, the observed input I_i and output O_i of s_i." (Section 3.2) This contract-based approach localizes errors without labels, replay, or retraining.
The empirical gains are heavily concentrated in the construction stage. Table 2 shows that Coalesce alone yields 22.21%, while the full method reaches only 25.13%; the in-flow stage contributes a modest +2.92 percentage points in total. This undermines the central framing that the system pays only for the fault, since cost-aware construction does most of the work. The rubric semantic estimator that powers the entire cost matrix is treated as a primitive without standalone accuracy validation or sensitivity analysis, yet every downstream decision—granularity, assignment, blame, and tempering—depends on it. The Braid benchmark is internally constructed and validated solely by the authors' own quality-control criteria, leaving open whether its tasks truly require multi-agent coordination or merely reward Coalesce's preferred decomposition patterns.
Furthermore, the theoretically motivated global-localization ablation degrades accuracy by "−1.63% on Qwen3.5-4B and −1.41% on Qwen3.5-9B," which the authors take as evidence that tempering beyond the local closure "costs more than it recovers." (Section 5.3) But since the objective minimizes $\mathrm{Cost}(W)$ rather than raw accuracy, reporting accuracy drops without corresponding cost figures is inconsistent with the paper's own framework.
The authors fairly characterize the evaluation landscape, noting that "the tasks these systems are measured on are largely ones a single competent agent already handles." (Section 1) Their Table 1 covers six backbones across five domains, which is more thorough than typical workflow papers. Yet the baselines are limited to a single greedy-decomposition workflow and a single agent with full tools, omitting topology-search competitors such as Zhou et al.'s Mass framework. Without this comparison, it remains unclear whether the gains stem from cost-aware decomposition or simply from searching a richer space than the greedy baseline. No statistical significance tests, confidence intervals, or variance estimates are reported for any accuracy comparison.
Critical hyperparameters—including the competence threshold $\bar{C}$, the unmet-condition threshold $\bar{u}$, and the relative weights $\beta$ and $\gamma$ in Eq. 2—receive no discussion of how they are set or how sensitive results are to them. The paper advertises a project page but does not explicitly release source code, the rubric semantic estimator implementation, or the Braid benchmark data within the manuscript. Algorithm 1 reveals the core formula: "$C(s,a) \leftarrow -\log \mathrm{match}(s,a); \rho(s) \leftarrow \mathbb{1}\big[\min_a C(s,a) > \bar{C}\big]$," yet the match function itself remains opaque. (Algorithm 1) Since all evaluated models are proprietary API endpoints and the sandbox details in Appendix E are high-level, independent reconstruction of the full pipeline is currently blocked.
Large language models (LLMs) increasingly construct multi-agent workflows that decompose a complex task and assign specialist agents from a pool. However, building such a workflow well remains challenging: how finely to divide the task, which agent to trust with each subtask, and when to create a new specialist are all critical decisions a workflow constructor needs to settle up front. Thus, whether each subtask succeeds remains unknown until the workflow runs. Yet, improving a workflow is costly. Locating a fault usually requires a reference answer, a graded outcome, or a trained assessor, and the fix is applied to the whole workflow through re-execution, re-search, or retraining. We propose InFlowOp, which prices every decision in one label-free cost that weighs how well an agent's competence meets what a subtask demands against how much that agent takes to run. Before execution, InFlowOp bidirectionally determines the granularity of task decomposition and agent assignment following from the cost rather than from a fixed template. During execution, InFlowOp corrects a fault with the cheapest move via the same cost that serves the workflow both as it is built and as it runs. Facing the workflow-level evaluation challenge, we introduce Braid, a benchmark whose tasks require multi-agent coordination beyond single-agent capability. Across various domains and backbones, InFlowOp outperforms single agent baselines by up to $+11.97\%$, achieving $+9.64\%$ with in-flow optimization. Our project page: https://xhguo7.github.io/InFlowOp/.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.