Same molecule, Different Knowledge: Cross-Identifier Binding Failures in LLMs For chemistry
This paper shows that large language models fail to treat co-referring molecular identifiers—such as trivial names, IUPAC names, and SMILES strings—as interchangeable routes to the same knowledge. The authors introduce ChemIDBench, a controlled benchmark of 896 approved drugs, and demonstrate a striking double dissociation: lexical names unlock entity-associated facts like therapeutic indications, while structural identifiers unlock structure-computable properties like benzene-ring presence. Even when a model recognizes a molecule from its SMILES, the corresponding factual knowledge often remains inaccessible, a phenomenon the authors term cross-identifier binding failure. They trace this to a missing access relation rather than missing knowledge, and propose Binding Self-Distillation to transfer lexical access directly to SMILES without retraining from scratch.
The paper makes a methodologically rigorous and empirically compelling case that molecular identifier choice is not merely a superficial formatting issue but a determinant of which knowledge an LLM can surface. By holding the molecule and property fixed while varying only the identifier, the authors establish a causal-like behavioral isolation that is rare in LLM chemistry benchmarking. The depth of analysis—spanning recognition controls, conflict experiments, activation patching, representation geometry, knowledge editing, and a lightweight training mitigation—gives the claim strong internal validity. The proposed Binding Self-Distillation is practical and effective, though its per-entity formulation currently limits scalability.
The core dissociation is robust and well quantified. Across seven models from 7B to 72B parameters, therapeutic-indication accuracy under SMILES collapses to near zero—for example, ChemDFM-v2.0-14B falls from $38.8\%$ under the trivial name to $2.1\%$ under SMILES—while structure-computable tasks like benzene detection show the reverse pattern. The recognition controls are particularly convincing: under 2AFC, ChemDFM-v2.0-14B recognizes 576 molecules yet achieves only $4.0\%$ SMILES indication accuracy versus $42.7\%$ via the trivial name, directly demonstrating that recognition does not imply retrieval. Activation patching provides causal evidence, with peak recovery of the name–SMILES log-probability gap reaching $76.4\%$ to $138.4\%$ across models.
The most significant limitation is scalability: Binding Self-Distillation is trained per entity, and the authors acknowledge that generalization to unseen molecules remains future work. Additionally, the clean double dissociation does not hold uniformly across all benchmark tasks; structure-computable tasks such as molecular-formula prediction and ring count show mixed identifier preferences rather than the stable reversal seen for benzene and nitrogen counting. The benchmark is also restricted to approved drugs in ChEMBL, leaving open whether the same binding failures occur for less common or more complex chemical entities. Finally, while the knowledge-editing diagnostic is valuable, its noisy transfer results suggest that cross-identifier propagation is fragile, yet the paper does not fully explain why Llama-3.1-8B-Instruct responds so differently from chemistry-specialized models.
The evidence overwhelmingly supports the central claim. Behavioral asymmetries are validated by paired McNemar tests showing both directions of the dissociation are significant across all seven models ($p<10^{-7}$), and the conflicting-identifier experiment neatly shows that lexical and structural routes can drive distinct outputs within the same prompt. The comparison to related work is careful: the authors explicitly distinguish their work from representation-preference studies like Baker et al. and consistency analyses like Yan et al. by focusing on access to knowledge already present in the model. One minor gap is that the paper does not benchmark against retrieval-augmented or tool-augmented baselines, which are common in chemical AI pipelines and might mitigate the observed failures without parameter updates.
Experimental detail is excellent—prompts, decoding settings, scoring procedures, activation-patching protocols, LoRA hyperparameters, and filtering pipelines are all documented in the appendices. However, the manuscript contains no explicit statement about code or data release, which is a notable omission for a paper that introduces a new benchmark. Without released artifacts, independent reproduction of ChemIDBench construction, the per-entity BSD adapters, and the causal patching experiments would require substantial reimplementation effort, even though the textual descriptions are precise.
Chemical AI workflows routinely represent the same molecule through multiple co-referring identifiers, including trivial names, IUPAC names, and SMILES, implicitly assuming that these representations provide interchangeable access to an LLM's knowledge. To test this assumption, we first introduce a controlled benchmark and show that it does not hold. Across foundation models spanning 7B to 72B parameters, even chemistry-specialized models, co-referring molecular identifiers consistently expose different subsets of the model's knowledge. We then conduct a series of controlled behavioral, representational, and causal analyses to characterize this failure. We show that even when a model can identify a molecule from its SMILES and retrieve a fact through the corresponding name, the same fact can remain inaccessible directly from SMILES. We term this phenomenon \emph{cross-identifier binding failure}. Further analysis of internal representations, conflicting identifiers, and parameter editing shows that neither improved chemical representation competence nor modification of individual facts is sufficient to establish reliable access across identifiers. Together, these results indicate that the core failure lies in the binding between an identifier and knowledge already present in the model. Motivated by this diagnosis, we introduce Binding Self-Distillation, a simple mitigation that uses lexical identity during training to transfer the model's existing knowledge access to the structural route. This enables factual retrieval directly from SMILES at inference time, without requiring a name lookup or training the underlying model from scratch.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.