Your paper timeline
Scroll AI takes the way you would scroll a great paper aggregator: quick signal first, deeper critique when something earns your attention, and challenges when a claim feels off.
797 papers in cs.LG
Trending mixes fresh papers with community signal.
0
cs.CVcs.LG Faias Satter, Sk. Md. Masudul Ahsan · Oct 1, 2026

This paper tackles handwritten Bangla word recognition by casting it as character-level object detection rather than whole-word classification. The authors ensemble SSD-MobileNetV2 and Faster R-CNN-InceptionResNetV2 to spot individual characters and diacritics in unsegmented word images, then rearrange the predictions into text. They introduce a modified Non-Maximum Suppression and a heuristic called Preserving Necessary Overlap (PNO) to cope with the cursive matra line and overlapping diacritics that make Bangla OCR notoriously difficult.

An optical character recognition (OCR) can scan a paper and extract text using technology, making people's jobs easier. While various OCR systems are available in the software industry, finding a reliable equivalent solution for Bangla takes much work. When it comes to handwritten texts, the situation is much more unusual. Recognizing words from word images is the most critical stage in any OCR process. It is the second stage after segmenting words from text pictures. If this stage fails, the overall performance of the OCR will be poor, regardless of how well the other phases perform. This study aims to recognize words using deep learning in a handwritten Bangla word image. Three object detection models, SSD with MobileNetV2, Faster R-CNN with InceptionResNetV2, and an ensemble model of these two, have been used to train and test handwritten word images. A modified Non-Maximum Suppression has been introduced to enhance the effectiveness of the models' results. A customized dataset of 9841 handwritten Bangla word images has been compiled, featuring diverse handwriting styles from various individuals. All three models' performances have been checked against the test dataset, and the ensemble model has been the most impressive, with an F1-score of 92.61%. Also, at the word level, the ensemble model correctly recognizes 96.12% of the words to some extent. The system can be further improved by introducing a post-processing phase to correct errors generated by the system.
0
cs.LG Shuo Xing, Zilin Dai, Chengyuan Qian et al. · Oct 1, 2026

LLMs can hit high accuracy on hard math problems while lacking genuine structural insight. This paper proposes the Mathematical Primitive---a concise, non-procedural statement of the key idea that makes a solution possible---and introduces Prim, a benchmark that disentangles structural understanding from procedural execution across Discovery, Generation, Digestion, and Execution. The authors show that independent Discovery is the dominant bottleneck and introduce Absorb, a primitive-privileged self-distillation framework that selectively transfers structural guidance into the student model without requiring primitives at inference time.

While Large Language Models (LLMs) have demonstrated striking capabilities on frontier mathematical problems, it remains unclear whether they possess the structural mathematical understanding underlying their solutions. In this paper, we take a first step toward systematically studying mathematical understanding in LLMs, from diagnosing its distinct capabilities to leveraging these findings to improve post-training. First, we introduce the notion of Mathematical Primitive to probe structural mathematical understanding and propose \hlei{}, a novel benchmark that evaluates mathematical reasoning along four distinct dimensions: Discovery, Generation, Digestion, and Execution. Second, our systematic diagnosis shows that solution accuracy masks distinct capability profiles, primitives unlock substantial latent execution capacity, and Discovery is the dominant bottleneck in mathematical reasoning. Our post-training analysis further shows that discovery-limited failures are particularly amenable to repair. Finally, building on these findings, we introduce \abs{}, a primitive-privileged self-distillation framework that selectively transfers primitive-guided reasoning into the student model. Extensive experiments demonstrate that \abs{} consistently improves mathematical reasoning over baselines across model scales and challenging benchmarks.
0
cs.LGcs.AIcs.CL Yafei Zhang, Songshuo Lu, Sicong Liao et al. · Oct 1, 2026

CARM targets off-policy drift in LLM reinforcement learning by preventing signed token log-ratios from canceling in sequence-level response masks. The core idea is to average absolute log-ratios rather than signed ones, which stops bidirectional policy drift from being hidden by the geometric mean. It also proves a joint bound on the fraction and severity of out-of-band token ratios for any accepted response. The method is evaluated on Qwen3.5 models and shows consistent gains over geometric-mean and token-level baselines.

Recent years have witnessed the rapid adoption of reinforcement learning (RL) in large language model (LLM) post-training, with substantial gains in mathematical reasoning and code generation. In practical systems, however, policy updates and differences between rollout and training engines can make sampled responses off-policy. Sequence-level masking addresses this mismatch by deciding whether an entire response should contribute to optimization. A common masking rule uses the length-normalized geometric mean of sampled token probability ratios. Its signed log-ratios can cancel across positions, concealing substantial bidirectional policy drift. We propose \emph{Cancellation-Aware Response Masking} (CARM), a sequence-level mask that takes the absolute value of each token log-ratio before averaging, preventing opposing probability changes from canceling. We prove that accepted responses satisfy a joint bound on the fraction of sampled-token ratios outside a prescribed band and their mean log-distance beyond its boundaries. Experiments on mathematical reasoning and code generation show that CARM improves mean@16 averaged over AIME 2024/2025/2026 and BeyondAIME by up to $3.13$ percentage points over geometric-mean masking, and increases average pass@1 across four code benchmarks by $2.88$ points over the strongest evaluated baseline. These findings support CARM as a theoretically grounded and effective method for response-level off-policy control in LLM reinforcement learning.
0
cs.LG Chuqin Geng, Li Zhang, Haolin Ye et al. · Oct 1, 2026

Mechanistic interpretability tries to identify sparse circuit subgraphs that explain model behavior, typically evaluating candidates via intervention-defined faithfulness. This paper demonstrates that faithfulness objectives can prefer an equally sized circuit that reproduces intact-model behavior worse than an alternative, because ablating excluded components distorts the computational context of retained ones. The authors argue this creates an objective-level recovery gap: better discovery algorithms cannot recover true mechanisms if the evaluation metric itself systematically rewards the wrong candidate.

Mechanistic interpretability aims to recover the internal computations responsible for model behavior. Progress in automated circuit discovery is often framed as a search problem: better attribution or optimization should identify better mechanisms. This assumes that the evaluation objective can recognize a better circuit once it is found. We show that intervention-defined faithfulness can instead prefer an equally sized circuit that reproduces the model's behavior less well, creating an objective-level recovery gap. Across four human-reference tasks and InterpBench, we compare validation faithfulness with behavior on held-out prompts under fixed ordinary resampling. The behavioral criterion is agreement with the intact model, including its mistakes, except on Greater-Than, where we use semantic accuracy. Controlled reference edits reveal misranking without any discovery algorithm, and outputs of EAP, EAP-IG, ACDC, and Edge-SP exhibit the same failure. Under resampling, KL misranks 9.4%-41.2% of candidate pairs across these methods on the human-reference tasks. We investigate context distortion as an explanation: replacing excluded signals changes the inputs on which retained components operate. Restoring selected signals from the recipient's intact-model execution repairs 96 of 100 persistent KL misrankings from the discovery pool on both validation and held-out prompts. The circuits and their original behavioral scores remain unchanged. These findings show why better discovery alone is insufficient when its objective rewards the wrong candidate.
0
cs.CVcs.AIcs.HC Ramazan Fazylov, Stamatis Lefkimmiatis, Ivan Laptev · Oct 1, 2026

While 3D Gaussian avatars render photorealistic humans efficiently, their animation pipelines are often throttled by costly per-frame neural inference. This paper introduces GALA, which shows that the animation outputs of diverse pretrained avatar networks can be closely approximated by an identity-shared linear blendshape basis extracted via distillation, coupled with a shallow coefficient predictor. By constructing the basis with block-local PCA under a rendering-aware metric and a strict memory budget, the method reduces CPU animation cost by up to three orders of magnitude and enables 60 fps on mobile devices with only modest quality loss.

3D Gaussian avatars support fast rendering, however, their real-time animation is often challenged by the costly neural inference. We address this bottleneck and show that the animation of pretrained avatar models can be closely approximated by a linear combination of identity-independent blendshapes. Building on this finding, we introduce GALA (Gaussian Animation via Linear Approximation), a distillation method that replaces per-frame heavy neural decoding with a shallow coefficient predictor and a linear blend. To improve fidelity and reduce memory requirements, we propose to construct the basis using block-local PCA under a rendering-aware metric and a memory budget. Our method learns a shallow MLP network to predict blendshape coefficients and applies to various animation architectures without retraining original models. We validate GALA by accelerating the inference of three distinct avatar models for 3D animation of facial expressions and full-bodies with clothing dynamics. Across these models, our distillation generalizes to held-out identities and reduces CPU animation cost by up to three orders of magnitude while preserving most of the rendering quality. Excellent results of our method confirm the shared linear structure of learned avatar representations and enable highly efficient and accurate animation at frame rates reaching up to 60fps on mobile devices. Project page: https://ramazan793.github.io/gala/
0
cs.LG Yidi Wang, Yunhe Zhang, Bangchao Deng et al. · Oct 1, 2026

Most trajectory generators require check-ins from the deployment city, which is unrealistic when only a public POI map is available. This work studies target-trajectory-free generation: synthesizing human mobility in a new city using only source trajectories and the target city's POI coordinates and categories. Its key insight is to replace absolute destination prediction with transferable relative transitions—semantic and geographic displacements between place contexts—and ground them onto the target POI graph via a behavior graph and an exploration–return walk. This factorization yields a theoretically principled and empirically evaluated method for zero-shot mobility synthesis.

Individual mobility trajectories support urban analysis and location-based services, yet most trajectory generators require observations from their deployment city. This assumption excludes precisely the cities where trajectories are unavailable even though points of interest (POIs) and their attributes can be obtained from public maps. We study target-trajectory-free generation: learning from POIs and trajectories in source cities while utilizing only POI coordinates and categories in a target city, with no target trajectory or trajectory-derived statistic available for training, model selection, or generation. Existing trajectory generators typically predict absolute destinations, entangling reusable movement behavior with city-specific POI identities and spatial layouts. Our core insight is to replace this city-bound output with context-conditioned relative transitions. We propose Nomad, a transfer-and-ground framework that separates learning how people move from determining where those movements are realized. Specifically, a history-conditioned flow-matching model learns from source trajectories a transition prior over semantic displacement between POI contexts, geographic displacement, and elapsed time; at inference, a behavior graph and an exploration--return walk ground sampled transitions onto the target POI map. This factorization enables a direct test of representation level transferability without assuming invariance of the full mobility distribution. Extensive experiments across ten cities and 14 transfers show that Nomad outperforms adaptation baselines in trajectory fidelity and downstream utility, lowering the average error over the best baseline of each metric by about 15% in distributional fidelity and about 3% in downstream utility.
0
cs.LGcs.CL Yohan Chatelain (1), Pablo de Oliveira Castro (2) ((1) Krembil Centre for Neuroinformatics, CAMH et al. · Oct 1, 2026

The paper asks whether stochastic rounding (SR) can improve low-precision transformer inference over round-to-nearest (RN), and shows that the benefit is site-dependent: SR outperforms RN in long MLP reductions because its $O(\sqrt{n}u)$ probabilistic error envelope beats the $O(nu)$ deterministic drift of RN, while RN is superior at the language-model head because SR's non-uniform variance is heavily penalized by the Fisher-weighted cross-entropy. The authors support this with a fast variable-precision emulation tool (PRISM), a vector Bienaymé–Chebyshev bound, a second-order loss decomposition, and experiments on DistilGPT-2 that motivate mixed-precision rounding recipes.

Should low-precision transformer inference use stochastic rounding (SR) or round-to-nearest (RN)? The answer depends on where in the network you look. We isolate this effect by holding the numerical format fixed and varying only the rounding rule at individual operation sites. To enable experiments at freely chosen precisions, we extend the PRISM vectorized rounding library to arbitrary virtual precision via a variable-precision stochastic rounding (VPSR) algorithm, proving that the rounding decision is evaluated exactly in hardware floating point. We develop two analyses providing complementary insight into this site-level trade-off. First, a probabilistic forward-error bound for linear projections shows that SR's error envelope grows as $O(\sqrt{n} u)$ in reduction length $n$, versus $O(n u)$ for RN, a gap that widens rapidly at low precision and is most pronounced in the long multilayer perceptron (MLP) down-projection. Second, a second-order decomposition of expected cross-entropy loss change at the output softmax into signed drift, drift curvature, and a Fisher-weighted variance penalty reveals why the two sites behave oppositely: MLP noise is predominantly a uniform logit shift to which softmax is invariant, so SR's variance is largely discounted; head noise is non-uniform across the vocabulary and is not. On DistilGPT-2 at $t=6$ significand bits, observations match theory: SR in the MLP raises perplexity to 1.15x the full-precision reference, versus 2.21x for RN. At the language-model head, the ordering reverses because SR introduces non-uniform variance, whereas deterministic RN carries none. In a mixed-precision configuration (MLP output at $t=6$), assigning SR to the MLP and RN to the head brings perplexity within 1.10x of the full-precision reference, a 28% reduction over matched-bit RN.
0
cs.CLcs.AIcs.LG Zilin Du, Bowen Yang, Boyang Albert Li · Oct 1, 2026

Training-data selection for LLMs is usually heuristic or sample-specific, which prevents generalization to unseen data. This paper tackles the harder problem of learning a shared meta-network that scores any example, and shows that plugging such a network into the standard bilevel objective (ScaleBiO) causes unstable optimization and poor transfer. The authors trace the failure to two properties of the ScaleBiO loss—competitive weight suppression and easy-feature dominance—and propose TESS, a method that replaces the bilevel objective with a simple pointwise MSE regression on pseudo-labels derived from loss gaps. TESS achieves strong transfer across datasets, corpus scales, and model sizes while cutting peak memory by over 20%.

Data selection is critical for training large language models on massive and heterogeneous corpora. Meta-learning for Training-data Selection offers a principled alternative to heuristic scoring by learning data weights from a target validation objective, but existing methods face a trade-off between fine-grained valuation and transferability to unseen data. A natural solution is to replace per-sample weights with a selection network. However, we find that directly incorporating such a network into existing MTS objectives leads to unstable optimization and poor generalization, caused by weight suppression and persistent reliance on easy-to-learn features. To address these issues, we propose Transferable Example Scoring and Selection (TESS), a scalable data-selection framework built on a Pointwise Value Matching objective (PVM). Experiments on LLM safety and targeted instruction tuning demonstrate strong transfer across datasets, from subsets to full corpora, and from smaller to larger models.
0
cs.LGcs.AI Michael Sullivan, Alexander Koller · Oct 1, 2026

Recent reasoning models show impressive capabilities thanks to RLVR post-training, but their chain-of-thought traces increasingly drift into unusual, non-standard language. This paper formally proves that RLVR optimization allows unbounded language drift while supervised fine-tuning imposes a hard finite bound, and shows empirically that such drift arises specifically when RLVR must discover novel behaviors on tasks beyond the base model's existing capabilities. The results suggest a fundamental tension: improving chain-of-thought monitorability by constraining drift necessarily limits the expected reward a model can achieve.

Recent advances in LLM reasoning models---driven primarily by the paradigm of post-training via reinforcement learning with verifiable reward (RLVR)---have enabled them to accomplish impressively complex tasks. However, in parallel with their rising capabilities, LLMs have increasingly displayed signs of language drift in their chains of thought (CoTs): unusual, non-standard, and seemingly nonsensical language use. Although it is well-documented---and can potentially impair CoT monitorability---the causes of language drift are thus far poorly understood. In this paper, we identify the conditions under which language drift occurs: we prove theoretically that RLVR optimization pressure permits unbounded language drift, while supervised fine-tuning does not. We then show empirically that language drift specifically arises during RLVR on novel reasoning tasks---i.e. when the target behavior cannot be drawn out of the base model. Finally, we prove that it is not possible to constrain language drift without constraining expected reward, suggesting that CoT monitorability cannot be improved without harming performance during RLVR post-training at the frontier.
0
cs.LGstat.ML Haodong Wen, Kaiyue Wen, Jiaye Teng · Oct 1, 2026

This paper studies gradient descent with a large constant stepsize on linearly separable logistic regression, a canonical model for the Edge of Stability phenomenon. It proves that the worst-case exact transition time from oscillatory loss to monotone decrease scales as $\Theta((\log \eta)^{\min\{n-2,d-2\}})$ uniformly over dimension $d \ge 2$, disproving a conjecture that stepsize independence holds beyond $d=2$. The result translates into substantially stronger finite-horizon optimization guarantees by permitting exponentially larger constant stepsizes than prior $O(\eta)$ transient bounds.

We study logistic regression on linearly separable data under gradient descent with a large constant stepsize $\eta$. Such dynamics may exhibit a characteristic Edge of Stability phenomenon, in which the loss initially oscillates before transitioning to a stable phase of monotone decrease. Existing work provides a tight $\Theta(1)$ bound in dimension $d=2$ as $\eta \to \infty$ and conjectures a bound independent of $\eta$ in arbitrary dimensions $d\geq 2$. In this paper, we disprove this conjecture by showing that, for every fixed sample size $n\geq 2$ and sufficiently small margin $\gamma$, the worst-case transition time is $$\Theta\!\left((\log\eta)^{\min\{n-2,d-2\}}\right)$$ uniformly over $d\geq2$. The key challenge in establishing a tight bound is that the sample contributing most strongly to the gradient can change repeatedly across iterations. To address this issue, we control such changes by induction on dimension and sample size, and construct matching hard instances.
0
cs.LGmath.GN Jennifer Rozenblit, Chenguang Yang, Yuxin Liu et al. · Oct 1, 2026

This paper tackles the tendency of graph diffusion models to lose global topology—such as cycles and higher-order motifs—during local denoising. It introduces MG-Diff, a framework that extracts a discrete Morse skeleton from a graph and conditions generation on this sparse topological backbone. By framing diffusion as traversing a discrete cobordism, the authors aim to anchor the generative trajectory to structurally critical events rather than treating topology as an afterthought. The promise is better preservation of connectivity and improved uncertainty quantification in spatio-temporal forecasting.

Topology is, by its nature and design, suited to structure that is nonlinear, multiscale, and nonstationary - however, within machine learning, its use remains largely confined to topological data analysis. We advocate that tools from low-dimensional topology which have remained almost exclusively contained within the domain of pure mathematics (such as Morse theory) offer a strong, complementary, and yet virtually unexplored perspective on the hidden structure of data-generating processes and learning tasks built upon them. Here we introduce concepts from cobordism theory and harness tools from discrete Morse theory to improve the performance of graph diffusion models through our pipeline MG-Diff. Further, we derive theoretical guarantees and sufficient conditions so that under a positive decision-gap, the Morse-theoretic tools and their application for induced diffusion guidance are stable under small perturbations. Finally, we illustrate the utility of discrete Morse theory in application to graph diffusion models for spatio-temporal graph forecasting and graph regeneration, and argue that these applications are only a small window into the part of what low-dimensional topology can offer to the field of machine learning.
0
cs.LG Seungjun Lee, Ensieh Khazaei, Dimitrios Hatzinakos et al. · Oct 1, 2026

Federated learning suffers from client drift when local data is heterogeneous, slowing convergence and degrading the global model. This paper proposes Activation Sharing Federated Learning (ASFL), a method in which clients exchange hidden-layer activations and labels with a randomly paired peer each round, then perform a partial tail round updating only the deeper layers using the pooled activations before standard local training. The premise is that aligning latent representations directly across clients mitigates heterogeneity more effectively than parameter-space regularization, while partial backpropagation cuts per-round computation.

Federated learning (FL) is a communication-efficient distributed learning paradigm. However, client drift remains one of the most critical challenges, hindering the efficient training of a global model. In this study, we propose a novel latent information sharing scheme that directly mitigates data heterogeneity across clients. Our theoretical and empirical results show that sharing a small amount of hidden-layer activations significantly improves training efficiency while preserving convergence guarantees and data privacy. Furthermore, we compare our method with existing FL approaches designed to address client drift, including FedProx, SCAFFOLD, FedPVR, FedProto, and SplitFed, and demonstrate superior model accuracy under a fixed round budget without incurring excessive communication overhead. Overall, this work presents a promising new knowledge aggregation scheme and provides a comprehensive analysis of the impact of activation sharing on federated optimization.
0
math.NAcs.LGcs.NA Tan Phuong Dong Le · Oct 1, 2026

This paper develops a physics-informed Kolmogorov-Arnold network (KAN) solver for free-boundary PDEs, including elliptic obstacle problems and the time-dependent Stefan problem. The core idea is to replace fixed MLP node activations with learnable univariate B-splines on network edges, aiming for localized resolution of contact sets and moving interfaces. Accurate, parameter-efficient approximation of these coupled unknowns matters because classical mesh-based methods require explicit interface tracking or costly refinement, while standard PINNs may blur sharp free boundaries.

We study free-boundary problems within a physics-informed framework using Kolmogorov-Arnold network (KAN) approximations. The proposed approach incorporates obstacle constraints, partial differential equation (PDE) inequalities, complementarity conditions, and boundary conditions through residual-based loss functions. We consider a linear elliptic obstacle problem, a nonlinear $p$-Laplacian obstacle problem, and a time-dependent one-phase Stefan problem. The proposed KAN solver is compared with physics-informed neural network (PINN) and residual-network baselines. Numerical experiments show that KANs achieve low relative $L^2$ and $L^\infty$ errors while accurately resolving contact regions and moving interfaces. The results indicate that KAN representations provide an effective alternative for solving free-boundary PDEs.
0
cs.LG Bangji Yang, Jingyuan Li, Jiajun Fan et al. · Oct 1, 2026

Test-time scaling is often proposed as a cheaper substitute for larger models, but its practical returns remain murky because prior evaluations usually focus on one domain and rarely charge selection costs to the budget. This paper introduces SELF-POT, a cross-domain benchmark and evaluation framework that charges every model call—including selection, critique, and voting—in actual dollars under fixed output-token caps. Across 350 sealed tasks in mathematics, programming, and agentic workflows, the authors show that extra inference yields wildly different returns depending on the domain and selection rule, and that the real bottleneck is often converting generated coverage into a valid submission rather than generating a correct candidate in the first place. By reframing test-time potential as a measured conversion profile tied to price and failure handling, the work connects empirical scaling behavior to concrete deployment trade-offs.

How much can test-time computation improve a language model, and at what cost? Test-time scaling is widely proposed as a substitute for larger models, but existing comparisons mostly evaluate one domain at a time and rarely charge selection to the budget. We introduce SELF-POT, a benchmark and evaluation framework that measures the test-time potential of a model across competition mathematics, competitive programming, and agentic workflows. SELF-POT separates candidate coverage from final accuracy on static tasks, tracks correctness transitions under revision, and measures protocol completion alongside task success in agentic environments. Under a unified budget rule, it compares Direct inference with parallel sampling and self-revision under fixed multiples of the Direct budget, and charges every model call, including selection and critique, in dollars. This design supports two kinds of comparison: the gain a model obtains from additional inference, and a lower-cost model with additional inference against a stronger model. Across five low-cost reasoning models on 350 sealed tasks, with Claude Opus 5.5 Direct as the reference, the returns depend on the domain, the selection rule, and failure handling. When we replay the retained programming candidate pools, public-example selection raises correct submissions from 376 to 453 of 500 scheduled cells while saving 12-49% of logical API cost across models, and simply retaining an available candidate when judging fails recovers 61 submissions at unchanged cost. On identical mathematics pools, judging with fallback yields 186 correct submissions versus 182 for voting, while voting saves 12-21% of logical API cost. These controlled replays show how selection and failure handling change the gains realized from the same generated candidates, and they quantify the marginal value of a model judge.
0
cs.LG Azal Ahmad Khan, Keshav Ramji, Tahira Naseem et al. · Oct 1, 2026

Adapting LLMs at test time usually requires gradients, task labels, or auxiliary demonstrations. This paper studies whether the input query alone contains enough signal to predict a useful parameter update. The authors propose a query-conditioned distributional hypernetwork that learns to output a distribution over LoRA adapters, trained with a differentiable Monte Carlo approximation of the marginal likelihood. They show that the mean update already outperforms deterministic hypernetworks, and that sampling diverse adapted models at test time provides a novel compute-scaling axis competitive with standard token sampling. Results are demonstrated on models from 1B to 4B parameters.

Hypernetworks have recently shown success in dynamically adapting the parameters of Large Language Models (LLMs) at runtime based on signals such as task descriptions or additional demostrations. Here we ask: how much adaptation signal can be obtained using only the input query to an LLM?. To answer this, we study query-conditioned Hypernetworks for LoRA estimation. Further, we introduce distributional Hypernetworks, able to produce not only point estimates of parameter adaptors, but also a distribution over possible LoRAs. For this we propose a simple end-to-end loss using a differentiable Monte Carlo approximation and explore multiple distribution parametrizations including regression and convex combination variants. Results show that even using the mean of the learned distribution can outperform deterministic hypernetworks. Crucially, the learned distribution enables a different form of test-time scaling: instead of spending additional compute only by sampling more token sequences from a fixed model, we sample weight updates, yielding multiple adapted models for the same query. Performance improves as more weight samples are considered and remains stronger than corresponding token-sampling adaptation baselines. Finally, we find that generated updates can transfer across queries, suggesting that the hypernetwork learns reusable structure in how the model should adapt. Together, these results show that query-conditioned distributions over weight updates can support both adaptation and test-time scaling.
0
stat.MLcs.LGmath-ph Daniel McBride, Pratik Khandagale, Cristina Garcia-Cardona et al. · Oct 1, 2026

Many recent sampling algorithms built on Wasserstein gradient flows (WGF) and forward-only diffusion processes (FODP) come with exponential-convergence guarantees that are often read as proof of fast practical mixing. This paper argues that guarantee is misleading: the canonical KL-induced WGF and overdamped forward diffusion produce the same marginal Fokker--Planck evolution, so they inherit the same metastability and exponentially slow inter-mode transport known in statistical physics. Using spectral-gap and mean-first-passage-time analysis on multimodal targets, the authors show that well-separated modes induce mixing times that scale exponentially with barrier height, and that standard log-linear annealing does not remove this scaling. The conclusion is that purely local, gradient-driven sampling is structurally limited for multimodal distributions, motivating nonlocal transport mechanisms.

There has been a proliferation of sampling algorithms based on Wasserstein gradient flows (WGF) and forward-only diffusion processes (FODP), often accompanied by theoretical guarantees of exponentially fast convergence to the target distribution. These guarantees are frequently interpreted as evidence that such methods can efficiently sample complex multimodal distributions, often supported by empirical results. In this work, we argue that this interpretation is fundamentally misleading. By invoking the Jordan-Kinderlehrer-Otto (JKO) scheme and Otto calculus, we establish that the canonical WGF sampling dynamics and overdamped forward diffusion share the same density evolution and therefore inherit the same metastability and slow-mixing phenomena long understood in nonequilibrium statistical physics. We analyze this family of samplers using two complementary tools -- spectral analysis and mean first-passage time (MFPT) analysis -- and show that well-separated multimodality can induce exponentially long mixing times associated with small spectral gaps and rare inter-mode transitions. For the commonly adopted log-linear annealing schedule studied here, we find that introducing intermediate distributions does not remove the exponential scaling of the total transport time. The limitation is structural rather than implementation-specific: purely local, gradient-driven transport mechanisms can require exponentially long times to transport probability mass across well-separated modes. We argue that this represents a fundamental limitation of WGF- and FODP-based sampling in their standard forms, and motivates future development of fundamentally nonlocal mechanisms for efficient multimodal sampling.
0
cs.LG Jason X. Liu, Sebastian Ibarraran, Frank Hu et al. · Oct 1, 2026

Intrinsically disordered protein regions (IDRs) are essential to cellular function but cannot be designed with structure-based methods because they lack stable folds. This work introduces IDiom, a protein language model pretrained on 54 million predicted IDRs, and RL-SAE, a reinforcement learning method that rewards the activation of interpretable sparse-autoencoder features to control sequence patterns. Together they enable composable, function-targeted generation of disordered sequences, though the results remain purely computational.

Intrinsically disordered protein regions (IDRs) play central roles in cellular processes such as transcriptional regulation, signal transduction, and subcellular localization, yet their functional design remains challenging. Structure-based design methods do not readily apply to IDRs, and existing protein language models are trained on full-length protein sequences, thus learning a prior that is biased towards folded domains. Here, we present IDiom, an autoregressive protein language model trained on IDiom-DB, a dataset of 54 million predicted IDRs curated from the AlphaFold Database. IDiom generates diverse sequences that recapitulate the composition, patterning, motifs, and predicted disorder of natural IDRs. To control function-associated sequence patterns, we also introduce reinforcement learning with sparse autoencoder features (RL-SAE), a post-training method that rewards the generation of sequences that activate specified feature sets. Across eight IDR design tasks, RL-SAE sequences activate, on average, 90% of 30 targeted features, compared to 24% for activation steering. We demonstrate that RL-SAE improves the predicted subcellular localization and transcriptional activity of generated IDRs compared to steering and supervised fine-tuning, and enables features associated with distinct biological functions to be combined within individual sequences. Thus, IDiom and RL-SAE enable interpretable and composable IDR design through explicit control of function-associated sequence features. More broadly, RL-SAE could extend to other protein design settings where interpretable features provide useful design targets. Code is available at https://github.com/rotskoff-group/idiom.
0
cs.LGphysics.app-phphysics.chem-ph Laura Zichi, Gil Harari, Chuin Wei Tan et al. · Oct 1, 2026

Equivariant machine learning interatomic potentials are held back by costly tensor product operations that dominate inference at scale. BranchIP addresses this by introducing lightweight, learned routers inside a single model that adaptively terminate tensor product refinement for individual features, trained with a novel distillation loss. On two realistic reactive systems—CO/Pt catalysis and CsH2PO4 proton conductors—it delivers up to 2.4× inference speedup and 2.6× memory reduction while preserving exact energy conservation and offering chemically interpretable routing patterns.

Equivariant machine learning interatomic potentials (MLIPs) have revolutionized atomistic modeling, but accurate treatment of complex materials and molecular systems demands expensive models. This limits simulation length- and time-scales, with tensor products a key computational bottleneck. The recent emergence of foundation-scale MLIPs further exacerbates this challenge. We present Branch Interatomic Potential (BranchIP), a single-model framework for learned adaptive tensor product computation, trained with a novel distillation loss. In our experiments on two systems of physical interest, a heterogeneous catalysis system and a proton-conducting solid acid electrolyte, BranchIP accelerates MLIPs across model sizes by up to $2.4\times$ while reducing memory usage by up to $2.6\times$. This is achieved while maintaining physical fidelity. Furthermore, the learned adaptive computation provides model interpretability by revealing which interactions demand deeper computation and showing how computational depth relates to chemical complexity and dynamics.
0
physics.ao-phcs.LG C. Daniel Boscu, Daniel Hernandez, Fabio Alvarez Ventura et al. · Oct 1, 2026

Rare weather regime transitions such as sudden stratospheric warmings are challenging to emulate because training data is overwhelmingly dominated by stable regimes. This paper develops a probabilistic ResNet-inspired Conditional Variational Autoencoder (CVAE) for the stochastic Holton-Mass model, a 75-dimensional idealized system with two metastable vortex states and rare noise-driven transitions. By training only on one-day-ahead predictions, the authors show that the emulator reproduces both short-term dynamics and demanding long-term statistics, and that unsupervised clustering of its 32-dimensional latent space yields four physically interpretable regime clusters. The results suggest a proof-of-concept pathway toward interpretable early warning systems for extreme events, although the study remains confined to a simplified testbed.

Rare weather regime transitions pose a challenge for data-driven modeling due to class imbalance. In this study, we develop a probabilistic deep learning emulator for a prototypical system with regime transitions, the stochastic Holton--Mass model of stratospheric variability, and analyze the structure of its learned latent space. The Holton--Mass model exhibits two metastable regimes, a strong and a weak polar vortex, maintained by nonlinear wave--mean flow interactions, with weak stochastic forcing intermittently triggering rare transitions between these regimes that qualitatively represent SSW events. We employ a ResNet-inspired Conditional Variational Autoencoder with six-layer encoder and decoder layers and explicit current-state conditioning to model the distribution of the system's state at the next time step (one day). The emulator accurately reproduces short-term dynamics, steady-state probability distributions, regime persistence statistics, rare transition rates, the transition committor function, and the transition expected lead time of the physical model. Beyond emulation fidelity, we interrogate the learned latent representation to understand how the model internalizes the underlying metastable structure of the dynamics. Principal Component Analysis of the 32-dimensional latent space reveals a clear and unsupervised separation into four physically interpretable clusters corresponding to strong versus weak vortex regimes and stable versus transition-prone configurations. Such emergent regime separation in latent space is hard to identify for deep generative models applied to high-dimensional stochastic systems. Our results show that carefully designed probabilistic emulators can uncover physically meaningful manifolds governing extreme-event dynamics, potentially aiding the development of improved operational advanced warning systems.
0
cs.NEcs.LG Jose A. Ayala-Romero, Andres Garcia-Saavedra, Xavier Costa-Perez · Oct 1, 2026

TRACE targets dynamic resource assignment in industrial systems where processing times, energy profiles, and queue capacities are unknown a priori. While existing LLM-based Automatic Heuristic Design learns only from scalar fitness scores, TRACE couples an evolutionary loop with an agentic workflow that distills noisy system logs into insights and executable surrogate models. This feedback turns blind trial-and-error into data-driven design, yielding heuristics that cut cost by over an order of magnitude relative to prior AHD methods and reduce energy by 65% against industry standards without sacrificing reliability.

Dynamic resource assignment, the real-time allocation of task streams to heterogeneous processing nodes, is the backbone of modern computing infrastructure. While learning-based schedulers excel in research, industrial deployments still rely on hand-written rules that operators can read, audit, and execute within tight latency budgets. LLM-based Automatic Heuristic Design (AHD) promises to automate writing such rules. However, existing AHD frameworks were developed for combinatorial problems fully specified to the LLM, and they learn only from a scalar fitness score. In real systems, the behaviour that determines a good heuristic, such as processor speeds or power consumption, is unknown a priori: the score reveals which heuristic performs better, but not why. This missing information is recorded in the system logs that every evaluation produces. Exploiting it is non-trivial: logs are massive and noisy, the relevant signals depend on the objective, and their content and format vary across hardware and software stacks, so they can neither be fed to an LLM as is nor processed by a fixed parser. We propose TRACE, which couples an evolutionary AHD loop with an agentic knowledge-extraction workflow. A Reasoner agent analyzes the log schema in light of the objective and formulates hypotheses about the system dynamics; a Coder agent writes and executes schema-specific code to test them, producing insights or executable tools for the evolved heuristics. We evaluate TRACE on a synthetic cloud benchmark and a 5G vRAN scenario built from industrial testbed measurements and operational traffic traces. TRACE consistently outperforms state-of-the-art AHD methods in resource assignment problems and yields more auditable heuristics at under 2% overhead.