Your paper timeline
Scroll AI takes the way you would scroll a great paper aggregator: quick signal first, deeper critique when something earns your attention, and challenges when a claim feels off.
1579 papers
Trending mixes fresh papers with community signal.
0
cs.LG Th\'eau d'Audiffret, Mariia Vladimirova, Jean-Yves Franceschi · Oct 1, 2026

Diffusion models amplify demographic biases present in training data, and current debiasing methods usually require costly attribute annotations, model retraining, or sacrifice output diversity. This paper proposes Debias Anything, a training-free guidance method that steers both unconditional and text-conditional diffusion models toward arbitrary target proportions for any sensitive attribute described in free text. By linking the frozen denoiser to a vision-language embedding space via a lightweight adapter, it jointly enforces batch-level fairness through optimal assignment and restores diversity through a semantic disagreement score, offering broad applicability without labeled data.

Although diffusion models produce high-quality images, they also reproduce and amplify demographic imbalances in their training data. Debiasing their generation process post-training w.r.t. some sensitive attribute usually relies on classifier guidance or explicit text extra-conditioning, but this reduces methods' applicability and output diversity. Conversely, methods promoting diversity alone do not ensure fair attribute representation. In this paper, we propose a method tackling fairness and diversity jointly that is generally applicable to any diffusion model and any sensitive attribute. To this end, an adapter connects the frozen diffusion model to a pretrained vision-language embedding space, enabling fairness and diversity guidance without sensitive-attribute annotations. For fairness, pairs of text prompts define attribute directions which guide batch composition towards specific proportions. For diversity, we introduce a score measuring disagreement between the semantic estimates derived from this representation. The formulation supports unconditional and text-conditional diffusion models, while requiring no prior knowledge or data of sensitive attribute. Experiments confirm that our method improves quality and diversity scores at comparable fairness levels.
0
cs.AIcs.LG Jude Waide, Robert Lieck · Oct 1, 2026

Long-term dependencies pose a fundamental challenge for sequential decision making because Transformers suffer quadratic attention scaling and RNNs lack expressive hidden states. This paper proposes Decision Titan (DTi), which grafts Titans-style Test-Time Training layers onto Decision Transformer to store episodic memories in fast weights updated by gradient descent at both train and test time. The authors introduce X-Maze, a controlled extension of T-Maze, and show that DTi can learn dependencies 20× longer than its context window and extrapolate to unseen episode lengths, while visualizing gate dynamics to explain how momentum propagates long-range information. Although the experiments are limited to synthetic memory tasks, the mechanistic ablations and honest characterization of catastrophic forgetting provide a valuable foundation for future research.

Long-term dependencies remain a major challenge for sequential decision-making in the field of AI: RNNs suffer from vanishing gradients and the limited expressivity of vector-based hidden states, whilst Transformer-based models are limited by the quadratic scaling of attention. Recent work has proposed tackling this problem with the Test-Time Training (TTT) framework, which stores episodic memories in the parameters of a neural network through gradient descent at both train and test-time. This approach has seen success in the domain of Natural Language Processing, however, to the best of our knowledge it has not yet been applied to the domain of Reinforcement Learning (RL), nor has there been a study analysing how this memory practically functions. In this paper, we study the potential of the TTT framework for offline RL by augmenting a Decision Transformer with TTT layers, dubbed the Decision Titan. We analyse performance and properties of the model in the X-Maze environment, an extension of T-Maze designed to test sequential memory, and investigate how the memory mechanism learns by visualising gate values over time. Our key findings are that Decision Titan can learn long-term dependencies with ranges 20x longer than the context window, generalises to lengths 1.7x the training data, but crucially temporal generalisation depends on the time embeddings used, and the ability to learn long-term dependencies depends on how the relevant information is encoded.
0
cs.LGmath.OCstat.ML Konstantinos Ziliaskopoulos, Alexander Vinel, Alice E. Smith · Oct 1, 2026

Contextual linear optimization faces a discontinuity: small prediction errors can flip the LP solution to a different vertex, making regret non-differentiable pointwise. This paper proves that, after averaging over the data distribution, the population regret is locally quadratic with curvature concentrated on the walls of the normal fan. The authors derive the exact curvature measure $K_0$, provide a tractable single-projection approximation $\mathsf{K}_{\lambda,s}$, and apply it to scenario generation, showing a $30.8\%$ regret reduction on a battery arbitrage task.

Decision-focused learning for linear optimization is complicated by the discontinuity of the optimizer, where small cost errors may leave the decision unchanged or move it to a different vertex. We show that this non-smooth pointwise behavior becomes locally quadratic after averaging over the data distribution, and we derive the curvature in closed form, specifically, a matrix-valued measure supported on the walls of the normal fan. This measure depends only on the feasible set, with the data distribution entering only as a weight. We then offer a tractable approximation for this curvature, computable with just one projection to the feasible set. We prove that the approximation weakly converges to the true population curvature. We offer one application of our findings, a decision-aware scenario generation method for expected-cost linear optimization. Our experiments test the quadratic and weak convergence laws and show a 30.8% regret improvement over uniform allocation on battery arbitrage.
0
cs.CVcs.LG Ahmed Sharshar, Asif Hanif, Naveen Kumar Kummari et al. · Oct 1, 2026

Medical imaging models often fail when deployed across scanners or hospitals because they overfit to domain-specific textures and appearance statistics. PhaseAT addresses this by adversarially perturbing the Fourier phase of the luminance channel while keeping the amplitude spectrum fixed, generating hard training views that stress spatial organization under matched style. By encouraging the model to ignore brittle amplitude-driven correlations and focus on stable geometric structure, PhaseAT achieves large gains on both single-source and multi-source domain generalization benchmarks.

Reliable clinical deployment of deep medical image models is hindered by distribution shifts across scanners, sites, and acquisition protocols. Existing domain generalization (DG) methods often focus on style or intensity diversification, but they can still leave networks dependent on domain-specific texture correlations. Inspired by evidence that Fourier phase encodes semantic structure, we introduce PhaseAT, a phase-aware adversarial training framework for medical DG. PhaseAT forms phase-perturbed training views in the Fourier domain by iteratively updating a bounded phase perturbation while keeping the amplitude spectrum unchanged, thereby stressing spatial organization under matched appearance statistics. Perturbations are applied only to the luminance channel in YCbCr color space to avoid chromatic artifacts. Additionally, a simple phase-saliency mask concentrates updates on the most influential frequencies. The model is trained with a weighted combination of losses on clean and phase-perturbed samples, supporting both single-source and multi-source DG. We validate our method on two challenging medical datasets and demonstrate that PhaseAT achieves over 20% improvement in single-source domain generalization, outperforming several state-of-the-art DG methods. The code implementation is available at: https://github.com/ahmed-sharshar/PhaseAT.
0
cs.LGcs.CLcs.CR Sahil Kadadekar · Oct 1, 2026

Can response safety be scored by cosine similarity to a safe-response embedding centroid? The paper shows that such a raw positive prototype identifies a location, not a separating direction, and formalizes why positive-only observations cannot determine the sign of the safe-unsafe gap. On three prompt-controlled safety corpora and four frozen encoders, the unreferenced prototype is weak or inverted while an explicit safe-minus-unsafe reference recovers useful ranking; the result is carefully scoped to the centroid rule, not all one-class methods.

Can response safety be scored by cosine similarity to the mean embedding of known-safe responses? A recent sleeper-agent detector proposes exactly this score, yet the raw positive-centroid rule is not identified: positive observations locate the safe class relative to an encoder origin, but do not determine which direction separates safe from unsafe responses. We audit the rule on two prompt-controlled, human-labeled corpora and one auxiliary jury-labeled source control, using four frozen encoders and prompt-grouped splits. On the human-labeled corpora the safe prototype reaches ROC-AUC 0.457-0.545, with two cells significantly below chance and one above, while an explicit safe-minus-unsafe reference reaches 0.588-0.738 on the same embeddings; on the jury control the prototype is inverted (0.358-0.405) and the reference reaches 0.754-0.793. At validation-calibrated 5% false-safe thresholds, the reference accepts more safe responses on PKU-SafeRLHF (0.153-0.263 versus 0.039-0.061 across encoders) and Aegis (0.189-0.291 versus 0.004-0.045), but not reliably on BeaverTails. A fully unlabeled held-out reference recovers part to most of the referenced ranking, much less when only 5% of the pool is unsafe, whereas 80-634 labeled unsafe responses recover most of it. Prompt-only ablations show that prompt-label composition can inflate uncontrolled evaluations. This is a bounded result about a raw positive centroid, not all one-class methods or safety-specialized guards. A class mean is a location, not necessarily a safety direction; a declared reference with enough unsafe mass identifies orientation.
0
cs.LGcs.SE Tianshuo Qiao, Naiqian Zheng, Xiaopeng Liu et al. · Oct 1, 2026

CI pipelines for LLM training frameworks are drowning in GPU-heavy end-to-end tests that can take hours per run. FastCI attacks this bottleneck by replacing coarse file-level mappings with a runtime evidence graph built from lightweight stack sampling, then pruning redundant tests via caller/callee context, scheduling risky tests first, and shrinking workloads along dimensions outside each test's validation scope. Deployed at ByteDance, it claims a 77.5% latency reduction and 63.9% GPU savings, though the evaluation rests on an internal codebase with no open-source artifacts.

As large language models (LLMs) keep growing in size and complexity, their training frameworks evolve at a rapid pace as well. Therefore, continuous integration (CI) is critical for maintaining the quality and stability of these frameworks. However, unlike traditional software, CI for LLM training frameworks relies on GPU-intensive tests, which usually involve complete model training or evaluation. This leads CI itself to become a new bottleneck for fast-paced development. In this paper, we introduce FastCI, a framework that improves the efficiency of CI for LLM training frameworks. FastCI leverages runtime evidence to select affected tests and prune tests that execute changed code in equivalent contexts. Then FastCI prioritizes high-risk tests to expose potential failures earlier, and optimizes test workloads along dimensions outside the intended validation scope of each test. Evaluated on the CI workload of our LLM training framework, FastCI reduces the CI latency by 77.5% and the GPU resource usage by 63.9%, while improving the modified code coverage retention by 3.2%, compared with the currently deployed CI pipelines. FastCI has now been integrated into the CI pipelines of our LLM training framework at ByteDance.
0
cs.LG Kuo Zhou, ZiXion Yang, Lu Zhang · Oct 1, 2026

SkillEvoLean tackles the challenge of improving LLM agents for Lean theorem proving without updating model parameters. The core idea is to evolve natural-language procedural skills through progressive refinement from execution trajectories and, crucially, a mutation operator that perturbs skills in a semantic concept space to escape local optima when all proof attempts fail. This matters because existing skill-evolution methods stagnate on hard theorems where no sampled trajectory succeeds, leaving the solver with no signal for improvement.

Skill evolution offers a promising way to improve large language model agents without updating their parameters, but its use in formal theorem proving remains underexplored. Existing methods mainly target natural-language reasoning, improving skills by analyzing successful and failed trajectories and incrementally revising solving strategies. Although the Lean verifier provides reliable execution feedback, when all sampled trajectories fail, existing skill evolution methods lack successful trajectories from which to infer effective update directions. Furthermore, these methods also focus mainly on the root instruction file, thus underexploring the evolution of reference knowledge including mathematical concepts and proving techniques. To address these limitations, we propose a mutation-enhanced skill self-evolution framework for building skill-augmented Lean provers. The framework jointly evolves a high-level solving policy and its reference knowledge through progressive and mutation-based updates. Progressive evolution derives local improvements from successful and failed trajectories, while mutation is triggered when no complete proof can be generated, sampling mathematical concepts to produce and select new skill candidates under verifier feedback. We evaluate our method on MiniF2F, PutnamBench, the 2025 International Mathematical Olympiad (IMO 2025), and the 2026 USA Mathematical Olympiad (USAMO 2026). Under the same backbone model, trajectorysampling budget, and test-time compute, our method achieves proof success rates of 100.0%, 90.6%, 4/6, and 4/6, respectively, with GPT-5.5, outperforming the baseline methods. Further analysis shows that concept-guided mutation outperforms random-text-guided mutation by 6.9 and 8.2 percentage points on MiniF2F and PutnamBench, respectively, while solving one additional problem on both IMO 2025 and USAMO 2026.
0
cs.AIcs.LG Changdae Oh, Qi Zeng, Qi Qi et al. · Oct 1, 2026

Does RL post-training create new agentic capabilities or just redistribute probability mass toward high-reward behaviors that a base model already possesses? This paper shows that, across 14 base/post-trained checkpoint pairs and three process-centric agentic benchmarks, pre-trained base models equipped with a lightweight harness often surpass their post-trained counterparts in solution coverage (pass@$K$) given enough test-time compute. To diagnose this coverage-accuracy trade-off, the authors propose Sharpening Tax, a single-number metric for the loss in test-time scalability after post-training, and present Posterior-Tempered Group Sampling (PTGS), a Bayesian sampler that mitigates the tax during RL training.

An emerging hypothesis about reinforcement learning (RL) post-training of large language models (LLMs) is that it merely sharpens existing behaviors of a base model, improving single-shot accuracy at the cost of solution coverage. Although this trade-off has been observed in math and coding tasks, it need not extend to agentic tasks, where multi-turn tool use and interaction may require capabilities newly acquired during post-training. Our surprising finding is that pre-trained LLMs, equipped with a light inference harness, can serve as capable agents. Despite far lower accuracy (pass@1), they often surpass their post-trained counterparts in solution coverage (pass@K) given a sufficient test-time budget. We further analyze the underlying mechanism and show that post-training pushes tasks toward two extremes, always solved or never solved, and thereby improves sampling efficiency and consistency at the cost of solution coverage. To measure this cost, we propose Sharpening Tax, a diagnostic metric that quantifies the loss in test-time scalability after post-training. Across 14 base/post-trained model pairs from four families and three agentic benchmarks (42 cases in total), the tax is prevalent in most settings, can be estimated from a few rollouts, and correlates well with other metrics. Finally, we present posterior-tempered group sampling (PTGS), a simple plug-and-play Bayesian sampler that adapts the sampling temperature per prompt to its estimated difficulty. Applied during RL training in two agentic environments, PTGS pays a smaller tax than the fixed-temperature baseline, solving more tasks under repeated sampling while also improving single-shot accuracy.
0
cs.AIcs.CLcs.LG Manar Aljohani, Brandon Ho, Kenneth McKinley et al. · Oct 1, 2026

Emergency Department triage assigns acuity scores that can be skewed by non-clinical patient descriptors. This paper asks whether open-source LLMs change their pediatric Emergency Severity Index predictions when only demographic, socioeconomic, or contextual variables are injected into otherwise identical clinical vignettes. The authors audit ten models and find that a small fine-tuned Qwen2.5-7B is the most counterfactually stable, challenging the assumption that larger or medically pretrained models are necessarily fairer. The study matters because locally deployable open-source LLMs are increasingly considered for privacy-preserving clinical decision support, yet their fairness profiles across model families were poorly understood.

Emergency department (ED) triage is a high-stakes prioritization task in which demographic, socioeconomic, and system-context information may improperly influence acuity assignment. Although open-source large language models (LLMs) are increasingly considered for local and privacy-preserving clinical decision support, it remains unclear how counterfactual bias varies across model families, sizes, medical-domain models, and domain-adapted models. We present a comparative counterfactual audit of ten open-source LLMs for pediatric Emergency Severity Index (ESI) prediction. Starting from real and handbook-style clinical vignettes, we construct paired counterfactual variants that change only one injected demographic, socioeconomic, healthcare-access, behavioral, social, or system-context variable while holding the clinical presentation fixed. Models include Qwen2.5-7B, Qwen2.5-14B-Instruct, a QLoRA fine-tuned Qwen2.5-7B, MedGemma variants, MedLLaMA2-7B, GPT-OSS-20B, and GPT-OSS-120B. We measure any counterfactual shift, undertriage, overtriage, shifts greater than one ESI level, mean shift, and mean absolute shift. Counterfactual sensitivity varied substantially and did not consistently decrease with larger model size or medical-domain pretraining. The fine-tuned Qwen2.5-7B showed the lowest overall sensitivity, with a 5.27% any-shift rate and mean absolute shift of 0.0534, versus 16.02% and 0.1706 for the base model. Several larger or medical-domain models showed more significant shifts. Stratified and correlation analyses further revealed clinically important directionality and shared failure patterns hidden by aggregate rates. These findings support counterfactual auditing as a lightweight, clinically interpretable framework for comparing fairness risks in open-source LLMs before clinical deployment.
0
cs.LGcs.CV Si Qi Goh, Cap Dang Xuan Kiet, Tat-Jen Cham et al. · Oct 1, 2026

SIEVE tackles selective unlearning in vision-language models (VLMs), where a model must forget specific sensitive image-question-answer associations about an individual while retaining permitted facts about the same person. The core idea is to suppress attention-value representations for forget examples toward zero and anchor retain-example values to a frozen reference model, coupled with sequence-level negative cross-entropy. This matters because VLMs can recover biographical information from faces, creating privacy risks, and prior methods struggle to disentangle closely related forgotten and retained knowledge without degrading utility.

The ability of vision-language models (VLMs) to associate visual identities with biographical information creates a need for selective unlearning of personally identifiable information (PII) while preserving permitted knowledge about the same individual. This setting is challenging because both sensitive and retained information can share the same visual inputs and intermediate representations. We introduce SIEVE, a simple and effective framework for selective VLM unlearning. SIEVE directly regularizes attention-value representations while also controlling model outputs. SIEVE suppresses attention values for forget examples toward a constant zero, while preserving retain-example representations by matching them to a frozen reference model. These objectives are combined with sequence-level forget and retain supervision, enabling targeted forgetting without largely affecting retained knowledge. Extensive experiments show that SIEVE achieves state-of-the-art performance on unlearning with multiple model-modality settings, while maintaining competitive retained utility. Ablation studies further show that value suppression and negative cross-entropy contribute complementary forgetting signals, while reference-based value matching substantially reduces utility degradation. These results demonstrate that attention values provide an effective intervention point for selective multimodal unlearning when sensitive and retained knowledge are closely related.
0
cs.LG Aneesh Barthakur, Mathias Niepert, Luiz F.O. Chamon · Oct 1, 2026

Guided diffusion forces a costly trade-off between enforcing requirements and preserving sample quality, usually resolved by small guidance scales and long sampling trajectories. This paper introduces LEEGS, which learns a time-dependent guidance schedule end-to-end by backpropagating the guidance loss through the sampling trajectory. A cheap gradient approximation—treating the guidance gradient as a stop-gradient—cuts training time by roughly 4$\times$, letting the method train on only 30 examples and match or beat strong baselines across image inverse problems, face-ID guidance, and PDEs with as few as 50–100 NFEs.

Diffusion models are a powerful generative paradigm used across multimedia and scientific applications. Guided diffusion methods impose requirements on the generation by adding the gradient of a differentiable loss (the guidance function) as a drift term during inference. The weight of this drift (the guidance scale) is critical for the trade-off between data quality and requirement satisfaction. To achieve both of these goals, guided diffusion must resort to small guidance scales and lengthy sampling, incurring high computational costs. This work proposes learned end-to-end guidance schedules (LEEGS) to achieve these objectives with fewer sampling steps. LEEGS trains a time-dependent schedule by minimizing the guidance function over a small set of examples using stochastic gradient descent. Backpropagating through guided sampling is computationally expensive, so LEEGS uses an approximation of the gradient that cuts training time by a factor of 4. We evaluate LEEGS on diverse guidance tasks, including (a) image inpainting, (b) noisy image inverse problems, (c) face-ID-guided generation, and (d) forward and inverse PDE problems, outperforming baselines at equal budget (50 or 100 NFEs), or matching constant guidance with only 10% of the steps.
0
quant-phcs.CRcs.IT Marcel Mordarski, Benjamin Gras, Abdelrahman Shehata et al. · Oct 1, 2026

Quantum key distribution deployments provision security margins assuming stationary channels, yet physical devices drift between calibrations. This paper frames adaptive eavesdropping under trusted channel noise and device drift as a constrained Markov decision process, where an attacker selects compact learnt quantum circuits each round while the noise follows an Ornstein–Uhlenbeck process. By bounding the adaptive value between the best fixed circuit and a dynamic-programming upper bound, the authors let operators measure how conservative their margins are and tune recalibration intervals, addressing a key economic barrier to QKD adoption.

Quantum key distribution (QKD) links are provisioned from security analyses of stationary channels, whereas the devices that determine the channel drift between recalibrations. Whether an eavesdropper who cannot alter the channel's own noise gains by following that drift has not been quantified. Adaptive eavesdropping is posed here as a constrained Markov decision process in which the attacker selects one circuit per round while the noise level follows an Ornstein--Uhlenbeck process and the abort condition is a budget over each block of rounds. The value of adaptation is bounded by the best fixed circuit and a dynamic-programming upper bound. The actions are learnt attacks. Whereas Decker et al. trained a parametrised circuit on a fixed gate template against a fixed channel, here the gate structure and rotation angles are searched jointly. This yields circuits compact enough to form a discrete action set, extending the construction to noise models lacking a known template, including the amplitude damping channel. On device-independent E91 under bilateral depolarising noise, a reinforcement-learning attacker raises her Holevo information from $0.135$ for the best fixed circuit to $0.348$ at zero detection, $98\%$ of the upper bound. On BB84 under a drifting bit-flip channel, she exceeds a conservative noise-indexed rule by $0.024$ in fidelity, reaching $99\%$ of the upper bound. Under stationary noise, the attacker's gain from basis asymmetry changes sign between an averaged and a per-basis error-rate constraint. The search, started from random gate sequences, recovers the analytical cloners and the collective-attack key rate, and meets the lower bound of the Winick--L\"utkenhaus--Coles objective from above.
0
cs.AIcs.LG Negin Ashrafi, Jia Luo, Stacey M. Frumm et al. · Oct 1, 2026

Molecular tumor boards must distinguish truly unsupported therapies from evidence-supported recommendations that still require oncologist review because of incomplete clinical information or poor performance status. This paper introduces OpenMTB-Audit, a 500-case synthetic NSCLC benchmark with four safety labels, and exposes pervasive over-refusal: every tested LLM configuration collapses the Partially Supported label, misclassifying valid-but-caveated therapies as either Unsupported or Supported. To address this, the authors propose MTB-AuditAgent, a deterministic seven-module pipeline that reduces over-refusal to 6.7% and achieves 91.2% accuracy. The work matters because label collapse in high-stakes oncology auditing can conceal unsafe endorsements or inappropriately block appropriate treatments.

Molecular tumor boards integrate genomic findings, clinical context, and therapeutic evidence to support precision oncology. As AI enters this workflow, a key safety challenge is distinguishing truly unsupported recommendations from evidence-supported options that still require oncologist review because of incomplete information, poor ECOG performance status, or other clinical caveats. We introduce OpenMTB-Audit, an open-source benchmark of 500 synthetic non-small cell lung cancer cases spanning five adversarial error categories and four safety labels: Supported, Partially Supported, Unsupported, and Insufficient Information. Across eight large language model configurations, we identify pervasive over-refusal: all LLM configurations failed to retain the Partially Supported label in 83.3-100% of true Partially Supported cases, achieving high aggregate safety scores through label collapse rather than clinically calibrated reasoning. To address this limitation, we developed MTB-AuditAgent, a deterministic seven-module framework separating evidence verification, missing-information detection, safety classification, and abstention. It reduces over-refusal to 6.7% and achieves 91.2% accuracy (95% CI: 88.6-93.6%). A two-oncologist annotation study found disagreement concentrated at the boundary between information sufficiency and treatment optimization, underscoring the need to preserve clinically meaningful distinctions.
0
cs.LGcs.AI Ashok Prasad Neupane, Saugat Adhikari, Pramish Paudel et al. · Oct 1, 2026

iADD tackles the alignment-diversity trade-off in reinforcement-learning finetuning of diffusion models such as Stable Diffusion. Its core idea is to update early denoising timesteps rather than late ones, combined with a sparse incremental curriculum and Feynman–Kac particle branching during training. Better alignment without mode collapse matters because existing reward-optimization methods frequently degrade visual quality and diversity.

Reinforcement learning based post training of diffusion models, such as Denoising Diffusion Policy Optimization (DDPO), optimizes a reverse diffusion process under a reward function. However, current approaches to reward optimizations do so at the cost of diversity and quality. In this paper, we provide better tradeoffs through careful theoretical considerations and method design. We analyze the theoretical framework and mathematically demonstrate that \emph{only-latter timestep} updates of diffusion model may be harmful for diversity contrary to the conclusions presented in a previous work. Additionally, we propose an incremental Feynman-Kac training based on strong theoretical foundations in order to achieve the best-yet alignment-diversity tradeoffs. We perform extensive experiments and compare our method against related diffusion policy optimization approaches in three different tasks and also provide strong ablations for each component, thus validating strong performance gains in both alignment and diversity.
0
cs.LG Riccardo Ali, Alessio Borgi, Mario Severino et al. · Oct 1, 2026

The paper isolates matrix-valued transport from the broader sheaf-diffusion framework by casting it through quiver representations. Its core insight is that standard multi-head attention implements diagonal transport over the head space, and that allowing edge-dependent off-diagonal routing before aggregation yields a strictly richer message-passing primitive. The authors introduce Topological Attention (Top-A), which adds cross-head communication while exactly recovering vanilla attention when the learned routing vanishes. Benchmarks on relational, heterogeneous, and algorithmic tasks show that the gains are task-dependent: largest when individual interactions require distinct transformations, but not systematically present under heterophily alone.

Sheaf Neural Networks generalize scalar-weighted message passing by replacing scalar edge weights with linear transport maps between local feature spaces. Yet the role of this matrix-valued transport is entangled with the broader sheaf-diffusion construction. We isolate the transport primitive through quiver representations and establish a direct connection with multi-head attention. Treating attention heads as coordinates of a local transport space reveals that standard multi-head attention implements diagonal edge maps: along each directed interaction, a source head can contribute only to the corresponding receiver head. Allowing off-diagonal entries instead enables edge-conditioned communication across heads before neighborhood aggregation. We show that this operation cannot, in general, be absorbed into a single shared linear map applied after aggregation. Building on this characterization, we introduce Topological Attention (Top-A), a multi-head attention that learns edge-dependent off-diagonal routes while preserving the original same-head paths and exactly recovering vanilla attention when the additional routing vanishes. We evaluate Top-A on relational reasoning, heterogeneous graph learning, and algorithmic reasoning, including out-of-distribution generalization, with heterophilic node classification as a contrast setting. The results show that cross-head transport is most useful when the task benefits from interaction-dependent transformations, while heterophily alone provides no systematic advantage. These findings identify edge-conditioned cross-head communication as a distinct computational primitive of matrix-valued transport.
0
cs.ROcs.LG Michael Y. Fatemi, Jinhao Liang, Ferdinando Fioretto · Oct 1, 2026

Multi-agent motion planning with diffusion models currently demands costly training on large feasible-trajectory datasets, making learned planners map-specific and data-hungry. This paper introduces TFDP, a training-free diffusion-style planner that replaces global neural score networks with analytical local scores derived from smoothness, obstacle-avoidance, inter-agent separation, and velocity-limit terms. By exploiting the spatial and temporal locality of trajectory refinement, the method scales to hundreds of agents and obstacles in seconds without any demonstrations or neural network training.

Path finding and multi-robot motion planning require trajectories that are smooth, goal-directed, and collision-free in environments with complex geometric constraints. Recent diffusion-based planners have shown that trajectory generation can be cast as iterative denoising which has opened the doors to learning-based approaches that can handle multi-modal trajectory distributions and refine entire trajectories. However, a key limitation is that diffusion planners require training on large collections of feasible trajectories, rendering them map-specific, and difficult to deploy when high-quality demonstrations are unavailable. This paper introduces a training-free diffusion-based motion planner that replaces learned global trajectory scores with analytical local scores derived from obstacle, smoothness, velocity, and inter-agent feasibility terms. The proposed idea relies on a key observation: the score of a trajectory can be reconstructed by considering only local interactions between neighboring waypoints and nearby constraints. This structure exploitation yields a decomposed denoising procedure that retains the optimization structure of classical trajectory methods while inheriting the iterative refinement behavior of diffusion models. Experiments on a large collection of complex environments and large multi-agent planning tasks show that the proposed analytical score produces smooth and feasible trajectories within limited computational costs, for example in generating feasible paths for 300+ agents in environments containing 100+ obstacles in under 6 seconds on a GPU, outperforming strong learning-based and optimization baselines, while avoiding the data requirements of learned diffusion planners.
0
stat.MLcs.LG Leyang Wang, Yakun Wang, Song Liu et al. · Oct 1, 2026

Comparing high-dimensional discrete distributions is notoriously difficult because the state space grows exponentially and differences can hide in complex joint dependencies. This paper introduces Zero Flux, a flow-based discrepancy that extends the continuous Zero Flow criterion to discrete state spaces. The core idea is to construct a discrete bridge between two distributions via coordinate-wise independent coupling, measure local probability fluxes at the midpoint, and show that all fluxes vanish if and only if the distributions are identical. This yields a decomposable, sample-based discrepancy with finite-sample guarantees, offering both a global comparison score and interpretable coordinate-wise contributions.

Comparing two high-dimensional discrete distributions has always been a challenging task due to the exponentially growing state space and complex changes in interactions. A recent work suggests comparing distributions through a vector field trained using flow matching between two continuous distributions. The resulting vector field at mid-point vanishes if and only if two distributions identical. However, such a flow-based criterion does not naturally apply to discrete distributions. We extend this principle to the discrete domain and introduce the \emph{Zero Flux} criterion, a discrepancy based on local probability fluxes. Under independent coupling, we show that all local probability fluxes vanish at the midpoint if and only if two distributions are the same. This discrepancy decomposes the joint distributional difference into smaller, local contributions and can be efficiently estimated from samples. We establish finite sample error bounds for our estimator. Experiments on synthetic and real categorical data demonstrate reliable recovery of sparse dependence signals and stable tracking of distribution shifts in high dimensions.
0
cs.LGcs.IR Parthiv Chatterjee, Dhiraj Golhar, Ummesalma Diwan et al. · Oct 1, 2026

Personalization encoders compress lengthy user interaction histories into a fixed preference state for downstream ranking or generation, but this compression can suppress evidence that remains available in cached per-timestep representations. This paper proposes REPAIR, a lightweight module that identifies and aggregates corrective signals from a frozen encoder's own cache across long, short, and episodic temporal views, then adds the correction back to the compressed state before the task head. By keeping the host encoder and task head frozen, REPAIR offers a practical way to improve deployed personalization systems without retraining them.

Personalization encoders compress evolving interaction histories into preference states used to rank items or condition text generation. A task head operating only on this state can miss useful evidence that remains in the frozen encoder's cached representations for individual timesteps. We study this recoverability gap and propose REPAIR, which compares cached representations with the current preference state in a compact learned coordinate space. It resolves corrective evidence over extended history, recent interactions, and localized bursts. It then selects which patterns at which timesteps contribute and adds their aggregate correction to the state before the task head. Encoder-host repair reuses representations from the existing forward computation without re-encoding the history. Across MovieLens, PENS, MIND, and Amazon Reviews 2023, training only REPAIR improves MRR and nDCG@10 for all twelve representative recommendation hosts while both encoder and task head remain frozen. Head-only finetuning of the same hosts yields smaller gains. For example, Mamba4Rec on MovieLens gains 3.96 MRR points, compared with 0.19 from head-only finetuning. Rank and temporal diagnostics support a compact, host-dependent corrective structure. In personalized generation, IMPerSumm improves the two reported weighted PerSEval variants, which assess responsiveness to user preference, by up to 25.23%. These results support post-compression state correction and distinguish the availability of preference evidence from its downstream use.
0
cs.LGmath.OC Jichao Jiang (1), Cristian McGee (1), El Houcine Bergou (2) et al. · Oct 1, 2026

TACO attacks the memory bottleneck of full-parameter LLM fine-tuning by replacing dense AdamW states with a geometry-induced sparse update. The core idea is to use a dimension-normalized $\ell_1 \to \ell_1$ operator norm, whose exact steepest-descent direction is a ternary, column-wise one-sparse matrix. By retaining only a tiny FP8 heavy-hitter history per column, TACO slashes persistent optimizer state by over $170\times$ and peak memory by nearly $3\times$, enabling 30–32B parameter models to fine-tune on a single 80 GB GPU.

Full-parameter fine-tuning of large language models (LLMs) incurs substantial optimizer state memory overhead, limiting the model sizes that fit on modern GPUs. Existing approaches either compress optimizer state, abandon first-order gradients, or change the update geometry while retaining dense state. The recently introduced Muon optimizer reduces optimizer memory through matrix-valued updates. Still, its geometry differs from AdamW and can lead to performance degradation when fine-tuning AdamW-pretrained models. To reduce optimizer memory without sacrificing accuracy or computational efficiency in LLM fine-tuning, we propose Ternary Absolute-max Column-wise One-sparse optimizer, or TACO, which follows Muon's operator-norm steepest-descent view but takes the geometric route further. TACO computes the exact steepest-descent direction under a dimension-normalized $1\to1$ operator norm by selecting the sign of the largest magnitude entry in each column of two-dimensional weight matrices. This retains first-order gradients while making optimizer state memory nearly negligible. Our practical TACO optimizer maintains only a small set of low precision gradient components per column, reducing persistent optimizer state by $174\times$ relative to AdamW8bit (from 27.7 GB to 0.16 GB) and peak training memory by $2.9\times$ (from 80.6 GB to 27.5 GB) on OPT-13B, while achieving comparable accuracy and runtime. TACO further enables full-parameter fine-tuning of 30-32B-parameter models on a single 80 GB H100 GPU across multiple model families and tasks.
0
cs.LGstat.ML Luca Geminiani, Nadja Klein · Oct 1, 2026

IQS-BO tackles the per-iteration cost of Bayesian optimisation by replacing surrogate fitting and acquisition-function maximisation with a single transformer forward pass. The core idea is to cast query selection as supervised classification: given a finite candidate set, the model predicts which candidate maximises the objective, trained with cross-entropy on synthetic priors where the true best candidate is known for free. The method can run fully in-context or on top of a fixed surrogate, and it introduces a mixture prior of GP and non-stationary functions to improve optimisation performance.

Bayesian Optimisation (BO) is a powerful framework for the optimisation of expensive black-box functions, but typically requires refitting a surrogate and maximising an acquisition function at every evaluation step. In-context approaches based on Prior-data Fitted Networks (PFNs) amortise part of this cost by pre-training transformers on functions drawn from synthetic priors. PFNs4BO amortises the surrogate but still relies on a numerically maximised acquisition function, while FIBO performs BO fully in-context by sampling optimiser locations from a learned density, which fixes the decision rule and admits no surrogate. Learned acquisition functions score a finite candidate set with a trained network, but, lacking a label for the query, learn the score by reinforcement learning on previously solved tasks. We propose IQS-BO, a PFN that learns the query decision by supervised learning on synthetic priors. In a single forward pass, IQS-BO predicts the probability that each candidate maximises the objective over the set, and we show that the minimiser of its objective is the posterior probability of this event. The model can be pre-trained without a surrogate for fully in-context BO, or take the predictions of a fixed probabilistic surrogate as additional input, amortising only the decision step. Our method proposes queries at a fraction of the cost of acquisition-based methods, while either matching or outperforming standard BO with Gaussian processes (GPs) and available in-context methods on synthetic and real-world benchmarks. Finally, we propose a mixture prior for pre-training PFNs which combines samples from GPs with functions exhibiting warped inputs, isolated narrow optima, or plateaus that are poorly modeled by stationary kernels common in GP surrogates. We show that pre-training on this prior can lead to improved optimisation performance.