When Do Biological Reasoning Models Use Their Biological Inputs?

The experimental design is a clear highlight. The authors do not simply remove an input; they pair perturbation tests with evidence conflicts, linear probes, and trace audits to disentangle three distinct explanations for low sensitivity: uninformative inputs, overlapping inputs, and actual underutilization. The linear-probe results are decisive: on genome-dependent queries, probes on Evo2 output reach 0.736 accuracy, and ESM3 probes reach mean per-family AUROCs of 0.766–0.921, proving the representations are predictive. By showing that BioReason and BioReason-Pro ignore these informative embeddings while ChatNT, Prot2Text-V2, and CellWhisperer do not, the authors avoid sweeping generalizations and instead pinpoint specific failure modes. The large-scale checkpoint sweep across 42 BioReason variants and the scale comparison for Cell2Sentence-Scale (2B vs. 27B) add robustness.

Several limitations temper the conclusions. First, the analysis is confined to six recently released models, so it is unclear whether the observed text bias generalizes to other fusion architectures or pre-training schemes. Second, while the authors acknowledge that shuffling DNA or protein sequences can produce out-of-distribution inputs for the foundation encoders, the perturbation-based results for BioReason and BioReason-Pro could in principle reflect encoder brittleness rather than reasoning-model neglect; however, the evidence-conflict experiments—built from unmodified inputs—largely allay this worry. Third, the reasoning-trace analysis, though illuminating, is selective rather than systematic: it highlights dramatic failures (e.g., BioReason misstates variant bases on held-out queries) but does not quantify trace correctness across all six models with a uniform metric. Finally, the auxiliary-supervision experiment in Section 4.4 yields only narrow gains that fail to generalize: edited-base prediction accuracy drops from 0.671 to 0.148 on unseen genomes, and disease prediction remains almost flat, leaving open whether any practical training fix exists.

The evidence convincingly supports the central claim that benchmark accuracy and biological-input use can be dissociated. The comparison between SFT and RL checkpoints is particularly striking: BioReason-Pro’s RL training boosts $F_{\max}$ from 0.829 to 0.855, yet replacing $\mathbf{Z}_{\mathrm{ESM3}}$ with a shuffled-sequence representation shifts $F_{\max}$ by only +0.004 for SFT and +0.001 for RL. Likewise, across 42 BioReason checkpoints, accuracy varies widely without a corresponding increase in Evo2 contribution. The authors fairly contrast these failures against models that do use their inputs—ChatNT splice-donor accuracy collapses from 0.988 to 0.502 upon shuffling, and CellWhisperer’s perplexity quantile drops sharply when counts are permuted—demonstrating that the tests are sensitive enough to detect usage when it exists. A gap remains in explaining why some architectures become text-biased while others do not; the paper hints at prompt structure and chain-of-thought templates as culprits but does not systematically ablate them.

Reproducibility is exceptionally thorough. The authors provide exact checkpoint identifiers, prompt templates, scoring code, and dataset preprocessing steps in extensive appendices, and they report hardware (NVIDIA H100 GPUs) and random seeds where relevant. The GitHub repository and evaluation data are public. Even so, full reproduction depends on accessing several non-trivial upstream artifacts—notably the Evo2, ESM3, and GO-GPT foundation-model weights and the 42 BioReason checkpoints sourced from Fesser et al. (2026)—which could pose a barrier for researchers without the same compute or licensing access. The appendices also disclose key hyperparameters (e.g., auxiliary-supervision training on 257-bp and 2,048-bp windows), though some analyses rest on small held-out sets (145 queries) that limit statistical precision.