It Takes Workflows to Evolve Better Workflows
Multi-agent workflows can solve complex tasks that stump a single LLM, but current methods typically train only the workflow generator while leaving execution agents frozen despite their shared influence on outcomes. This paper proposes FloWright, a paradigm that treats the workflow itself as a shared harness to optimize any role—generator, executor, or skill—using a hierarchical, structure-aware reward derived from execution traces, without extra models or labels. It also introduces DataWright, a rule-based data hardening procedure that converts single-agent datasets into workflow-level benchmarks. On 44 evaluation arms across documents, slides, charts, code, math, and finance, the authors show that co-evolving multiple roles yields larger gains than evolving one alone.
FloWright offers a conceptually clean solution to a real problem: because workflow outcomes depend jointly on every agent, training only the generator wastes improvement potential. The harness framework elegantly unifies evaluation, training, and test-time distillation into one signal, and the paper shows that co-evolving roles produces monotonically larger gains than single-role training. However, the empirical ceiling remains modest—absolute accuracies stay in the 30–50% range—and the most critical technical detail, how root-cause failure nodes are identified from traces, is stated rather than explained.
The strongest contribution is the preliminary evidence that standard benchmarks are ill-suited for multi-agent evaluation. The authors survey 20 workflow-generation methods and find that 85 of 121 reported evaluation datasets are single-question math, short-answer QA, or function-level coding—tasks a single agent can handle in one pass. Their controlled comparison on MMLongBench-Doc and DocFinQA shows that an untuned workflow actually loses to a single agent on the original data, but after DataWright hardening the same workflow surges past the single agent by nearly 40 points, confirming that the benchmark design, not the workflow, was the limiting factor. Inside FloWright, the staged reward ladder is also well validated: adding the validity term contributes the largest single increment, and structure-aware credit adds a further +0.69%.
The paper's central claim that credit assignment requires 'no additional models, labels, or executions' hinges on identifying root-cause failure nodes $F_j^\rho$ from the execution trace, yet the method for this identification is never specified beyond the formula $c_j^\rho = -|F_j^\rho|/|V_j^\rho|$. This opacity matters because misattribution would directly corrupt the reward for every co-evolving role.
A second concern is the narrow training footprint: FloWright is trained on only four datasets under one hardening strategy at level $\ell=3$, then tested on 44 arms spanning seven domains; the out-of-domain transfer may therefore reflect shared hardening templates more than robust generalization. Finally, the abstract cites a top gain of +7.41%, but Table 1 reports at most +7.32% in-distribution for the same 4B multi-agent mode, and only +5.03% overall, a small but distracting inconsistency.
The evidence supports the core co-evolution claim: validity reward and credit terms each incrementally improve accuracy, and co-evolution strictly dominates single-role evolution across all regimes. The authors also fairly represent the literature on benchmark inadequacy. Cemri et al. find that 'their performance gains on popular benchmarks are often minimal,' and Choi et al. conclude that 'Majority Voting alone accounts for most of the performance gains typically attributed to MAD,' both of which justify DataWright. Where the paper falls short is in direct empirical comparison to prior multi-agent training methods; rather than benchmarking against, e.g., MALT or MAPoRL, the related work is limited to qualitative discussion, making it hard to assess whether FloWright's gains are novel or simply what any multi-agent RL pipeline would achieve on harder data.
Reproducibility is mixed. DataWright's hardening rules are spelled out in Algorithm 1 and should be replicable, but the quality-control step relies on GPT-5.4 to flag wrong reference answers, and the evaluation pipeline depends on GPT-5.6-luna as an LLM judge and GPT-5.4 as a distillation teacher—closed models that readers cannot access. The reward weights $\{w_f, w_v, w_e, w_a, w_c\}$ are 'configured per role' with values relegated to Appendix F.2, and the project page is mentioned but no training code or data splits are explicitly released in the provided text. Most critically, the procedure that maps an execution trace to root-cause failure nodes $F_j^\rho$ is not specified at a level that would allow independent implementation.
Tackling complex real-world tasks can exceed the capabilities of a single large language model (LLM), motivating the use of multi-agent workflows that coordinate specialized agents to work together on these tasks. Recent methods train LLMs to construct better workflows from execution outcomes, but they optimize only the workflow generator, while the other agents that build or execute each workflow remain fixed even though every outcome depends on all of them. However, extending training beyond the generator is challenging: the agents are coupled, and a workflow's outcome is a single sparse score that cannot tell which agent causes a failure. We propose FloWright, which leverages the workflow as a harness to optimize workflows. By introducing a hierarchical, structure-aware reward paradigm, FloWright enables one role to self-evolve and two or more roles to co-evolve, with no additional models, labels, or executions. Considering the limitation that workflows are commonly trained and evaluated on data that a single agent can already handle, we further propose DataWright, an adaptive data hardening approach that converts existing datasets into workflow-level tasks with increased difficulty. Across document, slide, chart, code, math, and finance tasks, small open models trained with FloWright achieve improved performance by up to $+7.41\%$, with co-evolving ($+5.03\%$) more roles gaining more than optimizing one of them alone ($+2.83\%$). Our project page: https://xhguo7.github.io/FloWright/.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.