Your paper timeline
Scroll AI takes the way you would scroll a great paper aggregator: quick signal first, deeper critique when something earns your attention, and challenges when a claim feels off.
698 papers in cs.AI
Trending mixes fresh papers with community signal.
0
cs.AI Huiting Liu, Jinsong Wu · Oct 22, 2026

Autoregressive sequence models can suffer hidden future-token leakage when auxiliary operations—such as pooling, routing, or normalization—broadcast full-sequence summaries outside the causal attention mask. This paper audits these violations through prefix-invariance checks, a LeakScore metric, and layer-wise FirstLeakLayer localization, demonstrating that such leakage can produce false architectural gains: a CDR-AttnRes checkpoint drops teacher-forced validation loss from roughly 3.6 to 1.43 while its free-running generation perplexity explodes from under 20 to about 700. The core lesson is that teacher-forced validation loss must be paired with whole-model causal audits and open-ended generation diagnostics before claiming progress.

Hidden future-token leakage can create false architectural gains: an autoregressive model may appear substantially better under teacher-forced evaluation while its free-running generation deteriorates. We investigate this failure in an adaptive residual-routing decoder whose sequence-level routing summary exposes prefix positions to future tokens despite a correct causal attention mask. An anomalous 24-layer checkpoint reduces validation loss from the baseline's 3.5958 to 1.4259, yet it fails a prefix-invariance audit and has substantially higher reference-model perplexity during generation. Layer-wise diagnostics and implementation inspection trace the violation to a full-sequence summary broadcast through the dynamic routing-query path. Two independently trained 12-layer causal repairs eliminate detectable leakage and return validation loss and reference perplexity to the control scale, removing the large apparent advantage. As diagnostic tools, we use the established prefix-invariance contract, maximum prefix-output discrepancy (LeakScore), complementary behavioral metrics, and layer-wise profiles. Supporting experiments cover six synthetic implementation pairs spanning five module families, clean pretrained controls, controlled fault injection, and cache consistency across numerical precisions. These results illustrate how hidden future-token access can produce misleading claims of architectural progress and why lower teacher-forced loss should be interpreted together with whole-model causal checks and free-running evaluation.
0
cs.LGcs.AI Jo\~ao Kenji Suwa, Jo\~ao Pedro Silveira e Silva, Bruno Iochins Grisci · Oct 22, 2026

This paper adapts the Feature Selection Layer (FSL) into a post-hoc interpretability tool for pre-trained neural networks on tabular data. Instead of training an embedded feature weighting layer jointly with the network, the authors freeze the pre-trained model and insert a lightweight diagonal scaling layer that learns per-feature weights via fine-tuning. The approach aims to let practitioners retrofit already deployed models to obtain feature importance scores without retraining from scratch, bridging embedded feature selection and post-hoc attribution methods.

The interpretability of complex neural networks remains a critical challenge, especially for models already deployed in high-stakes domains. To address this, we adapted the Feature Selection Layer (FSL) to optimize feature weights post-training, a variant we term post-hoc FSL. Our approach reframes the FSL as a lightweight, trainable module that integrates with already frozen pre-trained models on tabular datasets to highlight the features the original model considers most important. This post-hoc FSL learns feature relevance by fine-tuning its weights based on the original model's learned outputs. Crucially, this process is non-invasive, operating without altering the original model's architecture or its learned parameters. We conducted our experiments using both statistical and visual metrics, including accuracy, F1 score, recall, precision, weighted t-SNE and silhouette score, and also analyzed the stability of the post-hoc FSL on high-dimensional synthetic and real-world tabular datasets. We compared the post-hoc FSL feature weighting method using these metrics against the original embedded FSL and other post-hoc interpretability methods, such as Integrated Gradients, Noise Tunnel, DeepLIFT, Gradient SHAP, and Feature Ablation. Experimental results demonstrate that the post-hoc FSL feature weighting method successfully identified relevant features across the different datasets, maintaining the predictive power of the original neural network while enhancing its interpretability.
0
cs.SEcs.AIcs.MA Atharv Chandratre, Jai Datar · Oct 22, 2026

Feature flags become technical debt once fully rolled out, yet manually cleaning them at scale is prohibitively expensive. This paper describes a multi-agent LLM system deployed at DoorDash that automates stale flag removal by querying live rollout data via MCP, coordinating parallel cleanup agents in isolated git worktrees, and producing merge-ready pull requests. Evaluated on 50 stale flags across hundreds of repositories, the system produced usable PRs for 90% of cases at an average cost of $4.79 and 13.8 minutes per cleanup, offering one of the first quantitative industrial benchmarks for LLM-driven feature flag cleanup.

Feature flags are runtime configuration mechanisms used for gradual feature rollouts, experimentation, and kill switches. Once a feature is fully rolled out, the flag and its gated code become technical debt and can introduce operational risk through inadvertently enabling dormant code paths. At DoorDash, over 56,000 feature flags span 623 repositories, with 1,659 currently stale as of May 2026, with internal estimates of one to two hours per manual cleanup. Existing rule-based cleanup tools proved insufficient in DoorDash's codebases. Our contribution is an end-to-end automated LLM-based feature flag cleanup system that integrates querying live feature flag rollout data using MCP, Jira ticket processing, and end-to-end pull request creation. Our system successfully produced engineer-approved flag removal PRs for 45 (90%) of the 50 stale flags it was evaluated on, proving the system's operational viability at scale with each cleanup averaging only $4.79 and 13.8 minutes. We evaluated the system's quality and reliability boundary based on success rates, cost per cleanup and time metrics. Furthermore, we discuss our lessons learned when architecting this system, to provide practical guidance for developing similar multi-agent systems.
0
cs.AI Dezhao Luo, Daniel Beechey, Derek Yuen et al. · Oct 22, 2026

GUI agents struggle in long-horizon mobile tasks because they must repeatedly re-discover intermediate workflows from scratch. This paper proposes a subgoal-aware memory graph that segments past trajectories into contiguous action sequences sharing a single intent, then merges visually similar boundary screenshots into shared graph nodes and attaches annotated subgoal transitions as directed edges. The core claim is that this intermediate granularity is more reusable than either raw actions or full trajectories, which the authors test on a challenging mobile benchmark.

GUI agents aim to automate complex user tasks by interacting with mobile interfaces, but ef- fective decision-making in long-horizon set- tings requires reusing prior experience rather than relying solely on the environment ob- servations. Existing memory-based methods typically store experience either at the single- action level or as entire trajectories. Single- step actions are often too fine-grained to carry meaningful semantics, since the success of an individual action does not necessarily imply progress towards the overall goal. In contrast, whole trajectories are often too coarse-grained and difficult to reuse across different tasks. We propose a subgoal-aware memory graph that distils past trajectories into structured, reusable evidence at the level of subgoals. We define a subgoal as consecutive actions shar- ing a single intent, the granularity at which progress toward the task becomes meaning- ful. To reduce redundancy and support cross- task reuse, we organise subgoal experiences as nodes in a graph, where related memories can be merged, extended, and retrieved based on the environment observation. On SPA-Bench, our method improves success rate by 11.11% over the baseline.
0
cs.LGcs.AI Gabriele Dominici, Teague McMillan, Fazl Barez et al. · Oct 22, 2026

CausalCoT addresses the problem of unfaithful chain-of-thought reasoning in LLMs, where dense attention allows models to bypass their own generated rationales and exploit global context instead. It enforces procedural causal dependence by decomposing queries into a directed acyclic graph of sub-questions and evaluating each node in an isolated context containing only its direct parents. The framework promises more faithful explanations and precise test-time steerability via local interventions.

Chain-of-thought (CoT) prompting elicits step-by-step rationales that appear to explain model predictions, but there is no guarantee these steps causally drive the final answer. In standard autoregressive generation, dense attention allows each reasoning step to attend to the full context, enabling the model to derive answers through direct lookup while generating plausible-sounding but post-hoc justifications. We introduce CausalCoT, an inference framework that enforces causal transparency by decomposing problems into a directed acyclic graph of atomic sub-questions, evaluating each node in an isolated context containing only its direct causal parents. This Markovian structure removes attention pathways to non-parent content, making downstream predictions sensitive to local reasoning rather than global context leakage. Across a diverse suite of reasoning benchmarks, CausalCoT preserves competitive predictive performance while improving explanation faithfulness, increasing causal sensitivity of intermediate reasoning steps by an average of 12 and up to 21 percentage points over the strongest baseline. The structured dependency graph additionally enables test-time interventions in which local corrections propagate predictably through downstream reasoning.
0
cs.AI Liang He, SiYuan Ma, Qishi Zhan et al. · Oct 22, 2026

Tree-based speculative decoding wastes target-model compute when large retrieved candidate trees contain many low-quality branches. ValuSpec addresses this by introducing a plug-and-play verifier-side filter that scores branches with a lightweight model, calibrates a threshold $\tau_{\mathrm{tr}}$ from the target model's greedy trajectories without any parameter training, and prunes low-value candidates before expensive verification. This preserves exact greedy output identity and yields up to $2.38\times$ speedup, showing that candidate supply and verifier-side valuation are complementary optimization stages.

Tree-based speculative decoding accelerates large language model inference by verifying multiple candidate branches in parallel. However, in practical serving scenarios, similar requests repeatedly induce overlapping candidate branches. While retrieval-based methods mitigate this by reusing historical drafts, the resulting candidate trees often contain many low-quality branches as the candidate tree grows, which reduces verification efficiency and wastes target-side computation. To address this, we propose ValuSpec, a plug-and-play candidate valuation mechanism that filters low-quality branches from a supplied candidate tree before expensive target-side verification. ValuSpec introduces transfer-based supervision to align filtering with the target model's behavior. A filtering threshold is estimated from the target model's greedy trajectories on a calibration split, without any parameter training. The target model then verifies the filtered tree under the output-identical greedy rule, so accepted tokens are guaranteed to match standard autoregressive decoding exactly. Experimental results on HumanEval, GSM8K, Dolly, and SpecBench show that, in our main evaluation, ValuSpec achieves up to 2.38x end-to-end speedup in retrieval-based tree settings. Furthermore, integrating ValuSpec into the EAGLE-3 dynamic tree framework yields up to 2.78x end-to-end speedup without modifying its tree construction strategy, demonstrating that ValuSpec serves as a generally applicable and plug-and-play candidate valuation module.
0
cs.DLcs.AI Bastien Le Guellec, Su Hwan Kim, Raphael Bentegeac et al. · Oct 22, 2026

Peer review is buckling under submission volume, and LLM assistance is gaining traction. This paper asks how four current LLMs treat 200 already-published articles from top-tier journals such as Nature, Science, and Cell. The authors find that model recommendations are overwhelmingly harsh and starkly inconsistent: $605$ of $800$ reviews ($75.6\%$) recommended major revision or rejection, yet the rate for any single model ranged from $17.5\%$ to $100\%$ ($p < 0.001$). The takeaway is that LLM editorial signals are too variable to replace human judgment, though convergent criticisms may still be useful.

With the growth of scientific output, the peer review system is approaching a breaking point: reviewers are increasingly hard to secure, and turnaround times lengthen accordingly. LLM-assisted or LLM-generated peer review has been proposed as a remedy, but how such models would treat manuscripts already published in top-tier journals, the closest thing the current system has to a gold standard of scientific validity, is not known. Here we assembled a corpus of 200 articles, each released under a CC BY 4.0 license and published in 2026, from 20 general-science, life-science, and clinical-medicine journals including Nature, Science, and Cell, and asked four current LLMs (two proprietary, two open-weight) to produce independent peer-review reports of each one under a blinded protocol. Their decisions were overwhelmingly harsh and starkly inconsistent with each other: 605 of 800 reviews (75.6%) recommended major revision or rejection, but the rate at which any single model did so ranged from 17.5% to 100% (p < 0.001), disagreement on manuscripts that had, by construction, already survived real editorial review at the journals in question. When the same manuscripts were doctored with one deliberately planted, field-specific methodological or statistical flaw, detection rates varied from 17.0% to 60.5% across models (p < 0.001). Recommending revision for papers that had already passed editorial review (low specificity) while missing flaws deliberately placed in front of them (low sensitivity), these models produce output that should be read for what it is: an uneven, framing- and model-dependent signal.
0
cs.AIcs.LG Thanh Q. Tran, John Russell Himawan, Arun Verma et al. · Oct 22, 2026

DIAL tackles the challenge of fine-grained control over instruction adherence—how strongly an LLM should follow instructions like "Keep it short." The core idea is to distill classifier-free guidance (CFG) into a lightweight, text-conditioned steering adapter that edits hidden states at a single layer. This matters because it promises scalable, efficient, adjustable adherence across diverse instructions without requiring labeled responses or per-instruction fitting.

When a user says &#34;Keep it short&#34;, how short should the LLM's response be? More generally, users and applications require different levels of instruction adherence, but prompting does not reliably provide fine-grained control over them. We therefore study &#34;steerable language models&#34; that provide fine-grained numerical control over the degree to which models follow general instructions specified in text. However, existing approaches do not jointly offer fine-grained control, efficiency, and scalability across diverse instructions. To address this gap, we introduce DIAL (Distilled Instruction-Adherence Levels) to achieve all three properties. For fine-grained control, we perform classifier-free guidance (CFG) with a frozen LLM to provide training signals for on-policy distillation of instruction adherence at different levels. For efficiency, we design a lightweight steering adapter that uses a classifier over the LLM's hidden states to compute a closed-form steering edit. The classifier adaptively sets the steering magnitude at each token based on the target adherence level, while accounting for the geometry of the LLM's hidden states to preserve response coherence. For scalability, the classifier is text-conditioned, allowing a shared adapter to support new instructions without per-instruction fitting, and training requires only instruction and request text. Evaluation across multiple LLM families shows that DIAL achieves a competitive balance between concept expression and response fluency, with concept scores increasing monotonically across adherence levels. We introduce SteerIF, an instruction-only dataset of 98, 625 concept, persona, verifiable, and descriptive instructions. When trained on SteerIF, DIAL provides fine-grained control over personas and, when combined with prompting, improves verifiable instruction following.
0
cs.LGcs.AI Bar{\i}\c{s} Alparslan, Sandy Tanwisuth, Caleb Petterson · Oct 22, 2026

Instrumental convergence predicts that capable agents develop power-seeking subgoals, but it is unclear whether reinforcement learning with verifiable rewards (RLVR) monotonically increases such behavior in language models. This paper evaluates OLMo-3-7B-Think across sixteen post-training checkpoints using InstrumentalEval and finds a nonmonotonic trajectory: instrumental-like outputs rise during early RLVR, peak near step 200, and then fall below the SFT baseline by the final checkpoint. The result challenges simple "more RL equals more convergence" narratives and shows that checkpoint-level monitoring can reveal dynamics hidden by endpoint comparisons.

Instrumental convergence is often invoked as a general reason to expect capable agents to acquire or express power-seeking subgoals: self-preservation, resource acquisition, deception, or resistance to oversight which poses an important threat to the society. For LLMs, reinforcement learning is the first stage of post-training that directly optimizes behavior against an objective; therefore, it is the stage where convergent instrumental behavior, on IC grounds, likely to appear. Does reinforcement learning with verifiable rewards (RLVR) make a model more likely to express instrumental-convergence behavior, or does this depend on the training regime, evaluation frame, and stage of post-training? We study this question as a checkpoint-conditioned behavioral trajectory rather than as a single before/after safety score. Using InstrumentalEval, we evaluate OLMo-3-7B-Think across SFT, DPO, 13 intermediate RLVR checkpoints, and the final RLVR model. The trajectory is nonmonotonic: instrumental rate rises from 25.0% at SFT to 46.1% near RLVR step 200, then falls to 11.8% at the final checkpoint. Paired item-level tests and step-wise regression support the final SFT-to-RLVR reduction. Prompt framing does not explain this pattern: a harmful-prompt indicator combined with other framing effects explains only 15% of within-checkpoint score variance on average, and Iterative Null-space Projection (INLP) removes a detectable framing direction without changing the SFT-to-final-RLVR gap. Together with preliminary 32B endpoint results (35.53% SFT to 22.37% RLVR final), these findings indicate that RLVR does not monotonically increase instrumental behavior in this setting: the rate rises early, then falls below the SFT baseline. The result is specific to this model family and benchmark, and judge agreement is moderate, so absolute rates should be read cautiously.
0
cs.NEcs.AIcs.LG Ze-Yu Peng, Qing-Yu Lan, Zhan-He Wang et al. · Oct 22, 2026

AgenticEvolve replaces the one-shot LLM variation operator in evolutionary program search with a persistent agent that edits code, runs self-designed experiments, and commits candidates for formal evaluation. By letting the agent test hypotheses locally between expensive evaluations, the framework aims to improve sample efficiency and final program quality on tasks ranging from online bin packing to symbolic regression. The authors report gains over a matched No-analysis ablation and over OpenEvolve, while noting that cost trade-offs depend heavily on model pricing.

Evolutionary program search with large language models keeps the structure of an evolutionary algorithm and uses the model as its variation operator. Typically, the model generates each candidate program in a single request from parent programs and their scores. We present AgenticEvolve, an evolutionary program search framework in which each island is run by a persistent large language model agent. Between evaluations, the agent designs and runs experiments on the candidate program. To measure the contribution of experimentation, we compare AgenticEvolve with No-analysis, an otherwise identical configuration with experiments disabled. On two heuristic design tasks with three large language models, AgenticEvolve produces better programs than No-analysis and OpenEvolve on both hidden and unseen instances. It reaches the baselines' final quality with far fewer evaluations and comparable cost. On twelve equation discovery problems, it recovers the ground-truth structure of more equations than No-analysis and existing language-model and genetic programming baselines.
0
cs.CLcs.AI Miguel Mochizuki Silva, Samuel Ismael, Mateus C. Dias et al. · Oct 22, 2026

Retrieval scores for clinical RAG are often computed against a single annotated chunk per query, conflating retriever quality with the completeness of the ground-truth annotations. This paper introduces RAG-MAMA, a breast-lesion question-answering system, and proposes mapping canonical evidence spans to all containing chunks so that overlapping segments with equivalent facts are not scored as competing candidates. The authors report nearly perfect retrieval metrics ($\text{Recall}@3 = 0.94$) on an 18-query development set and cautiously frame the work as a small methodological baseline, though the narrow scale and lack of independent validation severely limit its conclusions.

Purpose: To describe an evidence-aware retrieval evaluation protocol for a clinical retrieval-augmented generation (RAG) system over breast lesion management literature, addressing the dependence of retrieval scores on annotation coverage. Methods: An 18-document corpus and an 18-query development benchmark, with one Yes/No or WH-question per document, were used. Retrieval combined dense embeddings, BM25, and cross-encoder reranking. Chunk sizes of 128 to 1024 tokens, overlaps of 0 to 20%, and four retriever configurations were evaluated. Relevance was defined by evidence spans mapped to all containing chunks, treating overlapping chunks with equivalent evidence as relevant. Retrieval was evaluated using R-Precision, Recall@K, MRR, and nDCG@K. Separately, generation was assessed on a 512-token index at K=1 to 5 using DeepEval claim-level metrics, with deepseek-v4-pro as the LLM judge, and BLEU/ROUGE metrics. Results: The selected hybrid retriever, using 384-token chunks and 20% overlap, retrieved accepted evidence in the top three results for 17 of 18 queries. Recall@3, R-Precision, MRR@3, and nDCG@3 each reached 0.94. Ground-truth curation increased evidence spans from 25 to 37 and accepted chunk targets from 36 to 51 without changing retrieval rankings. Conclusion: Cross-encoder reranking achieved the highest retrieval scores. Expanded relevance annotations increased recognized evidence without changing rankings, highlighting annotation sensitivity. The small development set and retrieval-informed curation require independent validation, including radiologist assessment of evidence sufficiency.
0
cs.AI Gouki Minegishi, Jingyuan Feng, Hiroki Furuta et al. · Oct 22, 2026

Multi-agent LLM systems are becoming common, but it is hard to predict which model steers the group when agents disagree. This paper designs a minimalist naming game with meaningless labels to isolate consensus dynamics from pretrained knowledge or correctness. Across seven model families, the authors find that larger, more capable models dominate the final convention even when no answer is right, and they explain this with a simple energy landscape in which bigger models carry larger assertion weights.

Multi-agent LLM systems are increasingly used in practice, but their behavior is complex and hard to understand. We design a simple task to ask which model has the most influence on the final consensus when several models are mixed in one system. Experimentally, across seven model families, we find that the opinions of the other models are absorbed into the opinion of the stronger model, the one with more parameters. This happens even though the task requires no pretraining knowledge and has no correct answer. To understand this phenomenon, we interpret the agents as weighted particles moving on a simplex. Larger models carry a larger weight and therefore occupy a deeper well in the energy landscape. A large weight means strong in-context learning: a model with a large weight amplifies the majority of its context. Larger models are generally better at in-context learning, and this is what makes the larger model the influencer. We hope that this work helps to interpret multi-agent systems, whose behavior is becoming too complex for humans to understand.
0
cs.SEcs.AI Mohammad Mehdi Gholinejad · Oct 22, 2026

The paper defines superintelligent software engineering as the discipline of building software with agents that exceed human reviewers, arguing that raw capability is necessary but insufficient without governance that scales mechanically. It proposes that correctness depends on decisions and contracts outside the code, that such knowledge decays, and that once agent throughput $r$ outstrips fixed human review capacity $c$, the share of stale-built changes caught by reading falls toward zero. An exploratory snapshot of 452 popular GitHub repositories finds widespread stale agent context files, and the paper introduces OSTEON, a released discipline of governed knowledge repositories, mechanical gates, and decay control. The author commendably marks OSTEON's unmeasured claims as opinion and frames a falsifiable research agenda.

Software agents are being trained to exceed human engineers; we argue that capability alone does not produce better software engineering. We define superintelligent software engineering as the discipline of building and evolving software with agents whose capability exceeds that of human engineers, in which governance (gates, contracts, decisions and decay control) scales with capability. The argument rests on three claims. First, correctness in a project rests on decisions and contracts that the code does not carry. Second, knowledge decays. Third, once an agent outpaces its reviewers, reading catches a shrinking share of the changes built on stale knowledge. A simple model shows that, past a fixed review capacity, escaping changes grow linearly with throughput while the share that reading catches falls toward zero; governance can lower the slope, capability alone cannot. An exploratory snapshot of 452 of the most-starred public GitHub repositories finds a root-level or .github agent context file in 48.67% (95% confidence interval 44.1 to 53.3); in the median file that names repository paths, 9.5% to 22% of those paths do not resolve, depending on how a reference is read. We then present OSTEON, a discipline for the regime, released with a DOI (10.5281/zenodo.23436020): each project gets a governed knowledge repository of decisions and contracts, every change is checked mechanically against it, and the knowledge itself is checked for decay at every session start; planning is gated, and sessions record what they learn without writing rules. We report no measured effect of OSTEON, mark claims about its mechanisms as opinion and the author's practice heuristics as design rationale, and state the questions that would falsify the thesis.
0
cs.AIcs.CL Zehao Lin, Erenkai Zhu, Helan Hu et al. · Oct 22, 2026

This paper develops a decision-theoretic framework for valuing memory in language agents, treating memory not as raw information but as a resource whose worth depends on future decisions, access costs, and revision capabilities. The core idea is to track value from original history through retained state to realized output, using avoided task loss as a common unit and introducing risk-cost frontiers for recovery and revision under uncertain demand. This matters because current memory architectures optimize proxy metrics like relevance or compression ratio without guaranteeing that retained distinctions actually improve downstream decisions.

Memory enables language agents to carry useful distinctions from past experience into future decisions. Its value depends on which distinctions survive, whether a bounded agent can use them, and what can be recovered or revised as demands evolve. We develop a decision-theoretic framework that follows this value from raw history through persistent state, operational use, and costly recovery and revision. Original and retained value are expressed in units of avoided task loss; risk-cost frontiers describe attainable recovery and preservation across uses. Unknown demand yields value profiles, while interactive feedback changes the distribution of future work. A cost-aware simulation condition and five minimal counterexamples expose the roles of information timing, retrieval access, component interaction, feedback and demand uncertainty. In particular, a pointer with zero mutual information about the answer can still reduce the cost of exact recovery. Diagnostics on LoCoMo, a two-project update task and longitudinal cases exhibit failures of preservation, use and measurement. At fixed retrieved evidence, a saved ledger changes subsequent answers; at fixed agent outputs, an evaluator revision changes scores. The framework connects these distinctions to operational risk bounds and testable principles for allocating memory, preserving recovery options and revising evidence dependencies.
0
cs.AI Jeffrey Olmo, David Zhang, Tyler Tracy · Oct 22, 2026

Control evaluations test whether safeguards can stop a deliberately harmful AI agent, but they typically run in separate settings that may not resemble real deployment. This paper introduces simulated control evaluations (SCEs), which use a language model to generate tool responses in place of live execution, allowing evaluators to test attacks on a deployment's own tasks and codebases without causing harm. The central finding on LinuxArena is that SCEs predict safety metrics far more accurately than executing attacks in the mismatched BashArena control setting, suggesting that deployment-specific task coverage can outweigh the loss of execution fidelity.

Control evaluations measure whether safeguards can prevent an AI model from causing undesired outcomes even when it is intentionally trying to do so. These evaluations typically use purpose-built control settings and extrapolate their results to a model's deployment. However, the effectiveness of safeguards in deployment can depend on features of environments and tasks that control settings do not represent. To broaden where control evaluations can be run, we introduce simulated control evaluations (SCEs), which use a language model to simulate tool responses in place of an environment. We evaluate SCEs on LinuxArena, where execution provides ground-truth outcomes. We compare SCE prediction errors with those from a separate control setting, BashArena, and with variation in control metrics within LinuxArena. For independently sampled evaluation and deployment task sets, SCEs predict safety with a root mean squared error of 3.6 percentage points across five monitors at a 5% audit budget, compared with 10.1 for BashArena. SCEs closely reproduce distributions of attack strategies, attacker reasoning, and monitor reasoning, with over tenfold lower KL divergence from execution than BashArena. SCEs explain 78-85% of the between-environment variance in attack success, monitor scores, and safety. This accuracy is sample efficient: SCEs predict safety with less error than BashArena using fewer than 5% as many task-pair draws as the deployment task set. Strategies discovered through simulation also improve monitor performance on held-out LinuxArena attacks, outperforming BashArena and approximately matching improvement from executed strategies. These results suggest that deployment-specific SCEs can be more predictive than control evaluations executed in separate control settings.
0
cs.AIcs.LGcs.MA Ali Atiah Alzahrani · Oct 22, 2026

Most automated agent-design research evaluates candidates on static, immediate-reward benchmarks, ignoring the non-stationarity, delayed feedback, and switching costs of real deployment. Cmas formalizes meta-agent architecture search as sequential incumbent replacement with an explicit false-promotion target, using paired shadow replay, interventional credit, and a conservative gate followed by simulated canary and rollback. The paper matters because it treats the architecture itself as a time-varying decision variable and asks whether we can safely replace a running compound agentic system without backtested-return overfitting. The answer offered is guarded: standard statistical ingredients are integrated sensibly, but the calibration is empirical and the guarantees are deliberately scoped.

Automated agentic-system design is typically evaluated on static task suites with immediate scalar rewards. Deployed systems face persistent shift, delayed dependent feedback, correlated component failures, and costly architecture changes. We formalize meta-agent architecture search as sequential incumbent replacement with switching costs and hard risk constraints and introduce Conservative Meta-Agent Search (CMAS). CMAS retrieves archived architectures and proposes typed graph edits, then scores each against the incumbent by paired shadow replay under common random numbers, with interventional component credit. Promotion requires an empirically calibrated, variance-penalized simultaneous score above a pre-specified margin, and is followed by a simulated canary stage and pre-specified rollback. Since backtests cannot certify meta-level skill, we introduce MetaInvest-Bench, a regime-switching simulator whose specified worker competence makes a surrogate-optimal architecture computable, with leakage-controlled replay, adversarial governance tracks, and five negative controls. Financial streams serve as a risk-sensitive testbed for meta-level generalization; we claim no alpha. Across 96 streams, CMAS cuts normalized mean surrogate dynamic-oracle regret from 0.116 to 0.104 and raises held-out utility from 0.44 to 0.48 over the strongest adaptive baseline; when extra workers add no value, it contracts to near-single-agent teams. Its deployment inference cost is ~6% lower, though its search-inclusive token cost is the highest of the methods compared. The observed per-update false-promotion rate is 3.4% against a nominal 5% target, and 4.8% in a sealed rerun. The calibration is empirical: it concerns an update-level replay statistic on an author-designed benchmark, and guarantees neither full-utility improvement, nor absence of simulated canary-stage harm, nor sequence-level error control.
0
cs.AI Shuang Song Zhongqiao Yuan · Oct 22, 2026

This paper studies fine-grained intent classification on BANKING77 and CLINC150 by adapting a flexible decision-model interface (Laya) into fixed-label classification heads. For BANKING77 it applies sharpness-aware minimization (SAM) to a single encoder, while for CLINC150 it fuses probabilities from one Laya specialist and three DeBERTa-v3-large models. Rather than claiming a universal advance, the authors foreground statistical uncertainty, selection effects, and execution-path drift, making the work a transparent---if narrow---benchmark audit.

We present DeepOpen, a decision-Oriented intent-classification framework with dataset- specific systems for BANKING77 and CLINC150. On BANKING77, a single English classi- fier continued with sharpness-aware minimization (SAM) achieves 94.1234% test accuracy (2,899/3,080). This is the best observed result among three fixed SAM seeds, selected for display after test evaluation; their mean is 94.0368%. On the 150 in-scope CLINC150 classes, probability fusion of one Laya specialist and three DeBERTa-v3-large classifiers achieves 98.0222% (4,411/4,500). We describe the two configurations, reproduce the reference com- parisons displayed in the accompanying evaluation explorer, and separate headline scores from evidence about improvement. Under a uniform BANKING77 evaluation path, the SAM mean exceeds the historical checkpoint by 0.0433 percentage points, with a shared-row 95% interval of [-0.2273, +0.3247]. Thus, the continuation does not establish a reliable accuracy gain. The report documents strong observed task-specific performance, selection effects, and unresolved historical execution-path differences, rather than a new backbone or a unified algorithmic advantage.
0
cs.AI Sun Ze, Zhou Liguo, Xu Yuqing et al. · Oct 22, 2026

This paper addresses multi-horizon photovoltaic power forecasting from 15 to 240 minutes by asking whether residual error from a strong gradient-boosted tree can be reduced through selective, MAE-aligned combination with neural temporal predictors. The core idea is a constrained simplex mixture where the tree anchors the ensemble and weaker candidates can be assigned zero weight, with shared coefficients keeping the combiner parsimonious. The contribution matters because it isolates genuine aggregation value against a competitive baseline rather than inflating model count or comparing across mismatched tasks.

A strong direct tree model may remain the best standalone photovoltaic forecaster, yet a weighted combination can reduce its residual errors. We examine whether complementary temporal predictors can lower the tree's mean absolute error (MAE) over 15--240 minutes without weaker ensemble members degrading performance. The candidate bank contains LightGBM and seven neural implementations with different temporal representations. A combiner fitted to MAE has nonnegative weights that sum to one. Unhelpful candidates can receive zero weight, while the tree anchors the combination and provides a development-stage fallback. The main design shares weights across horizons; horizon-specific weights and alternative losses test whether further complexity is needed. On 56,078 windows from two co-located public PVDAQ systems, the historical shared-MAE policy reduces scaled MAE by 3.21\% relative to the tree. The latest bounded neural correction does not reduce MAE further than this static combination. The results support selective combination of complementary predictions in this bank rather than equal averaging or indiscriminate addition of modules. Both analyses reuse a previously inspected evaluation period. They do not establish independent replication, weather-specific causation or universal gains from eight experts.
0
cs.CLcs.AIcs.LG Mateusz Modrzejewski, Tomasz Radzikowski · Oct 22, 2026

Language model rankings on question-answering benchmarks are known to shift with prompt wording, raising doubts about single-prompt evaluation. This paper investigates whether augmenting models with deterministic tools stabilizes pairwise comparisons, proposing a matched protocol that isolates answer readout, reasoning, tool invocation, and report access to attribute changes in wording sensitivity. Applying the protocol to six open models across five domains and five prompt wordings, the authors find that tools improve accuracy where a program directly answers the question yet frequently widen ranking instability because call frequency itself becomes wording-dependent. The work matters because it shows that deterministic tool output is not enough to guarantee stable evaluation, and it recommends reporting call frequencies and pairwise stability alongside headline accuracy.

Language models are compared on question-answering benchmarks that pose each question in one wording, and a reworded question can change which of two models scores higher. We ask whether giving the models tools makes such comparisons more stable. An accuracy gain cannot answer this, because tools change several steps between the question and the scored answer at once. We propose a matched protocol that separates those steps: the answer readout, reasoning, the decision to call, the tool's answer, and the report supplied without a call. We apply it to six open models in five domains under five wordings. We find that tools raise accuracy where a program answers the question and widen the wording sensitivity of many pairwise comparisons. The widening follows how often a model calls, and call frequency follows the wording for some models and not others. It also depends on the interface and the instructions: a native tool-calling interface and a second tool description leave no detected widening in molecules, and sensitivity persists where tools return intermediate results. These findings show that deterministic reports do not guarantee stable comparisons. We recommend reporting call frequency and comparison stability under several wordings beside accuracy.
0
cs.AIcs.RO Taekoan Yoo, Wonkyung Jung, Kyunghun Kim et al. · Oct 22, 2026

PhysTrace evaluates physical consistency in videos by temporally aligning MLLM-generated, history-conditioned expectations with actual observations in a shared embedding space. Unlike prior work that probes internal representations or assigns video-level scores, it pinpoints when and why physical violations occur. The authors also introduce IntPhys2-Timing, adding temporal annotations to benchmark violation localization. Applied to diffusion and flow-based video models, PhysTrace reveals how physical inconsistencies emerge across generative timesteps—a crucial diagnostic as pretrained video backbones become policy priors for robot control.

World action models increasingly rely on pretrained video models assumed to capture physical dynamics for prediction, planning, and action generation. Existing evaluations assess plausibility, probe representations, or measure predictive surprise, but rarely identify which history-conditioned physical expectations conflict with observations. We introduce PhysTrace, a training-free framework aligning MLLM-generated, history-conditioned expectations with observed video windows to temporally localize physical violations and provide interpretable descriptions without task-specific probes or classifiers. We also introduce IntPhys2-Timing, adding violation intervals to IntPhys2. PhysTrace achieves a Hit Rate of up to 70.7% across physical principles on IntPhys2-Timing and outperforms the compared methods in overall possible-impossible discrimination accuracy on IntPhys2. Applied to reconstructed videos along the trajectories of pretrained diffusion/flow-based video models, PhysTrace reveals attribute- and noise-dependent patterns in physical violation localization, enabling interpretable physical consistency analysis across timesteps.