Regulated institutions must produce technical evidence for LLM deployments under the EU AI Act, yet existing benchmark suites are engineer-facing command-line tools while accountable reviewers are non-technical managers. VERA is an open-source, self-hosted dashboard that closes this workflow gap by letting reviewers select a model, run public benchmarks, and read requirement-level scores—with decomposable evidence and run-level energy consumption—in a browser. Its core contribution is wrapping raw measurement into a review and filing process designed for institutional approval.

Trace2World tackles the bottleneck of building faithful GUI training environments from sparse real-app traces. It separates global semantic reconstruction—resolving page identities, state fields, and conditional effects across traces—from browser implementation, then validates by independently replaying every source workflow. On 200 workflows across ten apps, this contract reaches 91.1% end-to-end fidelity versus 59.8% for the strongest task-local baseline and retains 94.0% of simulated training gains when transferred to real devices.

Coding agents typically retrieve skills before execution but rarely choose at runtime whether to invoke coarse macro skills or fine-grained primitives. This paper reframes skill use as a runtime control problem, introducing Adaptive Skill Granularity: a decision-tree router selects micro, macro, or hybrid execution, and a finite-state executor downgrades from macro to micro after observable failures. The framework matters because it shifts skill libraries from static documents into controllable, verifiable execution abstractions that can trade off efficiency against robustness.

Feature flags become technical debt once fully rolled out, yet manually cleaning them at scale is prohibitively expensive. This paper describes a multi-agent LLM system deployed at DoorDash that automates stale flag removal by querying live rollout data via MCP, coordinating parallel cleanup agents in isolated git worktrees, and producing merge-ready pull requests. Evaluated on 50 stale flags across hundreds of repositories, the system produced usable PRs for 90% of cases at an average cost of $4.79 and 13.8 minutes per cleanup, offering one of the first quantitative industrial benchmarks for LLM-driven feature flag cleanup.

The paper defines superintelligent software engineering as the discipline of building software with agents that exceed human reviewers, arguing that raw capability is necessary but insufficient without governance that scales mechanically. It proposes that correctness depends on decisions and contracts outside the code, that such knowledge decays, and that once agent throughput $r$ outstrips fixed human review capacity $c$, the share of stale-built changes caught by reading falls toward zero. An exploratory snapshot of 452 popular GitHub repositories finds widespread stale agent context files, and the paper introduces OSTEON, a released discipline of governed knowledge repositories, mechanical gates, and decay control. The author commendably marks OSTEON's unmeasured claims as opinion and frames a falsifiable research agenda.

Multi-agent LLM pipelines are widely assumed to outperform single-call setups, yet it is unclear whether gains come from typed inter-agent contracts or merely from decomposition into specialized prompts. This paper conducts an 18-condition controlled experiment on requirements classification and MoSCoW prioritization using three small language models on the PROMISE NFR+ dataset. The core finding is that most accuracy gains are associated with decomposition and task-specific prompts rather than with Pydantic-validated typed states, though the typed contract shows other quality-assurance properties such as earlier fault containment.

Frontend coding agents are usually graded only on whether the requested target gets built, but real engineering demands that the rest of the site keeps working. ProductWebBench evaluates this second obligation—website continuity—through 400 frozen tasks where a simulated user reveals requirements stage by stage and every cleared gate remains armed for the rest of the run. Across 8,737 runs from 13 models, 28.2% of Change attempts that pass every requested-work check still fail a continuity check, showing that scoring a patch against the target alone is insufficient.

When agents share a tool catalog and pool their usage experience, adding that usage to the retrieval index boosts tools with usage but can push unused tools out of the top results. This paper shows that folding usage examples into the index severely lowers held-out Hit@5 on two of three catalogs, and proposes Synapse, which scores documentation and usage as separate z-normalized fields before combining them with a learned weight. The finding matters because multi-agent systems risk silently failing on rarely-used tools unless the combination rule explicitly protects them.

VLTest addresses the problem of testing vision-language models (VLMs) where mutating an image, a text prompt, or both can change the expected answer, making naive output-change counting unreliable. Its core idea is to generate controlled visual variants via VQGAN latent code editing and textual variants via a rewrite–expand–prune pipeline, then confirm failures with an automated oracle and attribute joint failures to the specific image-text combination using matched four-cell evidence. This matters because VLMs are increasingly embedded in software systems, and existing robustness evaluations systematically overcount failures by ignoring whether an input still has the same correct answer.

AI coding agents can now edit native PDF financial documents while preserving issuer layouts, making identity-swapped forgeries harder to spot than image-level fakes. This paper tests whether experienced loan underwriters and processors detect such edits in a first-pass document-only review. The authors gave three hired reviewers 132 agent-edited forged PDFs derived from 47 genuine documents and checked whether their concerns actually matched the identity-altering edits. The core finding that only about one in ten judgements contained an edit-supported concern is a clear warning about relying on visual human review for financial fraud detection.

Architectural drift is expensive to detect when the mapping between source files and architectural modules is missing or outdated. This paper frames code-to-architecture mapping as semi-supervised node classification over a directed multi-relational dependency graph, expanding a small labeled seed via iterative self-training. By holding features, training loops, and classifier heads fixed across GCN, GAT, and a graph-free MLP, the authors isolate whether dependency structure genuinely improves mapping accuracy.

Multimodal code intelligence demands programs that are correct, well-structured, and visually faithful, yet existing training signals are either too coarse or confined to single tasks. This paper introduces PrismaForge, which synthesizes per-instruction evaluation rubrics that separately assess code-level and visual-level criteria, then uses them to filter training data and reward reinforcement learning. The resulting dataset, PrismaCoder-2M, and the trained 9B and 35B models advance the open state of the art and compete with proprietary systems on charts, webpages, and scientific graphics.

Doc2Harness evaluates whether frontier LLMs can engineer a coding-agent harness from scratch, a capability that existing SWE benchmarks ignore by holding the harness fixed. The benchmark asks a build model to generate a complete harness from a development document and an interface-only skeleton, then evaluates it deterministically with a scripted mock model across $328$ black-box test cases. The results reveal a $26$-point performance gap between the best and worst models, with even the strongest scoring only $53.05\%$, suggesting substantial room for improvement in long-horizon infrastructure construction.

Automated vulnerability repair typically validates patches against the same static alerts or proof-of-concept inputs used to generate them, risking overfitting to partial evidence. This paper introduces P2Patch, a counterexample-guided adversarial repair protocol in which an exploitation agent repeatedly tries to bypass the current patch, with each confirmed bypass becoming a permanent repair requirement. The core idea is to treat the patched program itself as a new attack target and to evaluate final patches against held-out security oracles strictly separated from construction-time evidence.

Coding agents consume substantial tokens as they repeatedly decide where to look, what to try, and when to stop during automated repair. This paper asks which of these decisions can be offloaded to a cheap probabilistic judge that answers fixed questions without generating explanations. The authors introduce JevCoder, a workflow built around the proprietary Jev judge to recommend initial code snippets, guide action types during repair, and trigger submission. On 400 SWE-bench tasks across three repair models, the system cuts repair-model token use by $21.6\%$–$50.6\%$ while raising observed repair rates.

CI pipelines for LLM training frameworks are drowning in GPU-heavy end-to-end tests that can take hours per run. FastCI attacks this bottleneck by replacing coarse file-level mappings with a runtime evidence graph built from lightweight stack sampling, then pruning redundant tests via caller/callee context, scheduling risky tests first, and shrinking workloads along dimensions outside each test's validation scope. Deployed at ByteDance, it claims a 77.5% latency reduction and 63.9% GPU savings, though the evaluation rests on an internal codebase with no open-source artifacts.

The paper asks when a nominally weaker LLM can reliably audit patches produced by a stronger coding agent. Its central claim is that groundability—the property that evidence exposes an independently checkable claim about missing behavior—matters more than reviewer scale. On a benchmark of 411 real execution-labeled traces and 101 controlled cases, the authors show that weak reviewers given official execution evidence can achieve near-perfect catch and near-zero over-rejection, while a deployable cascade of static checks and generated tests recovers only part of that upper bound. The finding that parameter count is not a monotonic predictor of review quality reframes the oversight problem from building bigger judges to building better checks.

Choosing an agent system means picking both a model and the execution harness around it, but most evaluations fix one and vary the other. This paper runs a full cross-product of four configurable harnesses (OpenHands, DeepSeek Harness, PI, openJiuwen) and five models across three terminal-use benchmarks (TUA-Bench, ALE-CLI, Terminal-Bench 4), plus two native vendor pairings, for 66 configurations total. The core finding is that model rankings reverse depending on the harness, best harnesses change across tasks for four of five models, and cost does not reliably predict score, suggesting that model-harness fit should be evaluated as a unit.

ActiveSaddler addresses the overlooked problem of which training scenarios to use during automated LLM-agent harness optimization. It treats curriculum selection as a non-stationary bandit whose arms are dynamically instantiated failure patterns, allowing the scenario distribution to co-evolve with the harness rather than being fixed upfront. On GAIA2 and Terminal-Bench 2.0, this yields test Pass@1 gains of 4.4 and 7.5 percentage points over the same optimizer with a fixed scenario order, arguing that adaptive curricula are a critical new dimension for harness optimization.

The paper mines 22,848 closed GitHub issues from 21 open-source LLM-based multi-agent systems, filtering down to 944 MAS-related reports to catalog what goes wrong in practice. Through qualitative coding, the authors map issues, causes, and solutions, finding that orchestration and workflow design dominate practitioner pain points. It matters because it replaces anecdotal MAS debugging lore with an empirical taxonomy grounded in actual maintenance traces.