Interpretability Doesn’t End at the Model

Interpretability has become a familiar ambition in biomolecular AI. Yet a model’s ability to highlight a gene, a pathway, or an amino acid does not establish whether the highlight is reproducible or biologically meaningful. The difficulty is not simply that explanations can be hard to read. It is that the evidence supporting them is often more limited than the evidence used to evaluate predictions.

A 2024 review of interpretable deep learning in functional genomics examined 123 research articles. Among the 121 studies that could be categorized by their choice of interpretation strategy, 115 used only one strategy; just six combined multiple approaches.[1] This does not mean the interpretations were incorrect, but it shows how seldom they were approached through complementary methods. Different techniques can answer different questions, and agreement between them is not something we can take for granted.

The difficulty becomes measurable when an explanation is tested against a known signal. In a 2025 study in PLOS Computational Biology, Reynolds and Pan assessed several popular feature-attribution techniques using genetic data from approximately 300,000 UK Biobank participants. By adding synthetic variants with known effects, the researchers could test whether the explanations recovered them. Among the top 1% of ranked variants, Gradient SHAP, DeepLIFT, and Integrated Gradients each recovered only 1% of the inserted recessive-effect variants. A smoothing procedure increased that figure to 24%. Recovery of variants with interaction effects was approximately 20–21% at the same threshold, regardless of smoothing.[2]

The same benchmark found excellent recovery for simpler additive effects. Its lesson is therefore not that feature attribution is useless, nor that every missed signal is a failure of the explanation alone: the model may also have failed to learn part of the relevant relationship. Rather, explanation quality depends on the type of biological effect, the underlying model, the attribution method, and how success is measured. A plausible-looking output is not a substitute for checking these dependencies.

Consider a researcher investigating a protein interaction. A model predicts that a substitution might improve binding affinity, and an attribution map highlights the residues most influential for that prediction. The researcher must still establish whether those positions relate to a direct interface contact, an effect on protein stability, or some other feature learned from the training data. These alternatives suggest different experiments, and the attribution map cannot distinguish among them on its own.

In a recent Molecular Cell Perspective, Mukund Varma and Trey Ideker argue that interpretability should be treated as a central design objective for biological foundation models, rather than an exercise performed only after training.[3] Their argument raises a broader question for biomolecular research: how can explanations be designed, tested, and preserved so that they help researchers move from a computational prediction to a biological hypothesis worth investigating?

The biology we already know

One reason this is a particularly interesting question in biology is that models are rarely asked to discover a world about which nothing is known. Researchers have spent decades characterizing pathways, molecular interactions, protein complexes, regulatory relationships, and the conditions under which those relationships change. At the same time, emerging multimodal foundation models aim to learn from a far broader range of biological measurements, bringing together information that has often been studied separately.[4]

Varma and Ideker make the case for using this accumulated knowledge as more than background material for interpreting a prediction after the fact. Biological relationships can inform the structure of a model, constraining how information travels through it or defining intermediate representations with recognizable biological meaning. This is the intuition behind approaches such as visible neural networks, in which parts of the architecture correspond to known cellular organization.[3]

The idea has a concrete history. In DCell, Ma and colleagues organized a neural network around a hierarchy of cellular subsystems and used it to connect genetic variation with growth phenotypes.[5] Later, P-NET incorporated curated pathways into a model of prostate cancer progression. Its interpretable structure helped identify candidates associated with advanced disease, including MDM4 and FGFR1, which the researchers went on to investigate experimentally.[6] These are not examples of the modern large-scale foundation models envisioned in the new Perspective, but they show why encoding biological organization into a model can make its outputs more scientifically investigable.

There is an important qualification. Biological priors are not equivalent to biological truth. Pathway resources are incomplete, annotations can be context dependent, and experimentally established relationships may behave differently across cell types or conditions. A model can be easier to inspect because its architecture uses familiar labels, while still relying on associations that deserve scrutiny. Making the architecture more legible creates an opportunity for interpretation; it does not complete that interpretation.

When an explanation looks convincing

A different family of approaches starts with a trained model and asks which parts of its input were most influential for a particular output. Feature attribution, sensitivity analyses, and computational perturbations can be valuable here. Applied to our hypothetical protein interaction, an attribution method might highlight residues whose presence strongly affects the predicted binding score.

What that highlight means depends on the method, the model, and the question being asked. It might represent local sensitivity to changes in the input, a contribution under a chosen baseline, or a property of an internal representation. None of these descriptions is automatically synonymous with a physical binding mechanism. In particular, a residue can matter to a sequence-based prediction without being in direct contact with the partner protein.

The challenge is not merely philosophical. Adebayo and colleagues showed that some saliency methods can produce visually plausible maps even when essential aspects of a model or its training signal are altered.[7] In genomics, Majdandzic and colleagues found that networks with strong predictive performance can nevertheless differ substantially in the consistency of their attribution-based interpretations.[8] Together, these studies illustrate why explanation quality cannot be inferred from prediction accuracy or visual appeal alone.

A useful explanation should therefore be evaluated for fidelity to the model: does it capture the dependence that actually drives the prediction? Repeating an explanation across seeds, comparing methods, checking its sensitivity to appropriate perturbations, or examining what happens when the model is retrained can help expose weaknesses. Each test addresses a different failure mode. Rather than asking whether a figure makes biological sense at first glance, we should ask what, precisely, the figure has demonstrated.

From model behavior to a testable hypothesis

Even a faithful explanation of model behavior does not settle the biological question. Suppose computationally replacing the highlighted residue makes the predicted affinity drop. We have learned that the model depends on that position under the chosen perturbation. We have not yet shown that the residue controls affinity in a purified binding assay, nor whether any measured effect would arise through a direct interface contact, altered folding, or a change in protein abundance.

This distinction matters because model-level evidence and experimental evidence answer different questions. Computational analyses help establish what information the model uses, how stable that dependence is, and whether a prediction is consistent with existing knowledge. Experiments test what happens in the biological system, and additional controls may be required to identify a mechanism. Neither perspective is interchangeable with the other.

There are productive ways to connect them. P-NET offers one example of a biologically informed model whose output was used to prioritize hypotheses for laboratory follow-up.[6] CellBox, which combines models of cell dynamics with machine learning, illustrates another route in which perturbation data support predictive modeling and the investigation of inferred interactions.[9] The designs differ, but both suggest that interpretation becomes more useful when it participates in a cycle of prediction, challenge, measurement, and revision.

In practice, this also requires attention to details that may appear secondary in a model paper: the biological system in which the prediction is valid, the experimental readout used for evaluation, the uncertainty attached to the prediction, and alternative explanations that remain compatible with the data. These details determine whether an interpretation can guide an experiment or merely accompany a result.

Interpretability does not end at the model

The argument for models that are interpretable by construction is compelling, but the story does not end when the model leaves the team that developed it. A computational result may be reused by people who did not choose the training data, implement the architecture, or evaluate the explanation method. They may see a predicted effect and a visual attribution without knowing which assumptions, caveats, or validation experiments made either result meaningful.

For scientific use, continuity of evidence matters. A result is easier to challenge and build upon when researchers can trace it back to the model version, the data and biological context, the interpretation method, the relevant prior knowledge, and the experiments that have or have not supported the claim. This is not simply better documentation. It is part of keeping the scientific reasoning inspectable as a computational result moves between disciplines and research stages.

This is one reason we are interested in the possibility of an interpretable layer around biomolecular AI workflows. Such a layer would not make an opaque model transparent after the fact, and it could not transform an unvalidated importance score into a biological mechanism. Its more realistic role would be to preserve connections between predictions, model-specific explanations, supporting evidence, and the experimental decisions they are meant to inform. That would allow scientists to see where a claim is strong, where it is provisional, and what would be needed to test it further.

Varma and Ideker's Perspective[3] points toward a future in which biological knowledge and interpretability are considered during model development rather than appended at the end. Following that principle into everyday scientific practice means treating explanations as claims that must remain open to inspection and revision. As biomolecular AI becomes more capable, the most interesting question may be not how many predictions a model can explain, but what kind of scientific understanding those explanations can genuinely support.

References

1. van Hilten, A., Katz, S., Saccenti, E., Niessen, W. J., and Roshchupkin, G. V. (2024). Designing interpretable deep learning applications for functional genomics: a quantitative analysis. Briefings in Bioinformatics, 25(5), bbae449. Link
2. Reynolds, J., and Pan, C. (2025). Benchmarking interpretability of deep learning for predictive genomics: Recall, precision, and variability of feature attribution. PLOS Computational Biology, 21(12), e1013784. Link
3. Varma, M., and Ideker, T. (2026). Toward interpretable foundation models for molecular biology. Molecular Cell. Published online October 7, 2026. Link
4. Cui, H., Tejada-Lapuerta, A., Brbić, M., et al. (2025). Towards multimodal foundation models in molecular cell biology. Nature, 640, 623–633. Link
5. Ma, J., Yu, M. K., Fong, S., et al. (2018). Using deep learning to model the hierarchical structure and function of a cell. Nature Methods, 15, 290–298. Link
6. Elmarakeby, H. A., Hwang, J., Arafeh, R., et al. (2021). Biologically informed deep neural network for prostate cancer discovery. Nature, 598, 348–352. Link
7. Adebayo, J., Gilmer, J., Muelly, M., et al. (2018). Sanity Checks for Saliency Maps. Advances in Neural Information Processing Systems, 31. Link
8. Majdandzic, A., Rajesh, C., Tang, Z., et al. (2022). Selecting deep neural networks that yield consistent attribution-based interpretations for genomics. Proceedings of Machine Learning Research, 200, 131–149. Link
9. Yuan, B., Shen, C., Luna, A., et al. (2021). CellBox: Interpretable Machine Learning for Perturbation Biology with Application to the Design of Cancer Combination Therapy. Cell Systems, 12, 128–140.e4. Link

Editorial note: The protein-binding scenario is hypothetical. The benchmark numbers describe specific experimental designs and should not be generalized to all biomolecular AI systems. This article is an interpretive essay based on the cited literature, not a report of new TenesimBio experiments.