Why genome-to-trait models fail across datasets

Can a bacterial genome predict whether its organism grows on a particular carbon source? Within a single dataset it can, at a balanced accuracy of 0.86. Across datasets that falls to 0.62, and the usual explanations of too few genomes and too much taxonomic overlap do not account for the gap. The gap tracks whether the experimental label agrees with the annotated mechanism, and training only on the samples where the two agree changes what the models learn.

Two standard explanations, both testable

Microbial carbon utilisation is a foundational ecological phenotype that remains difficult to predict from genomes, even where the pathways are well characterised. Machine-learning models score well on their own data and generalise poorly to anyone else’s, a failure usually attributed to training-set size and taxonomic bias. Both explanations are testable. Four binary growth datasets were integrated, covering 819 strains across 240 carbon sources, and evaluated under split regimes that hold out whole clades and whole datasets rather than random rows.

Accuracy drops from 0.86 to 0.62 when a whole dataset is held out

Predictor and evaluation	Balanced accuracy
CatBoost, repeated random holdout	0.75–0.90 (median 0.86)
CatBoost, leave-one-dataset-out	0.52–0.84 (median 0.62)
CatBoost, out-of-clade	0.53–0.86
GapMind, strict threshold	0.66 ± 0.04
GapMind, permissive threshold	0.70 ± 0.08
Phylogenetic k-NN (k=3), random holdout	0.72–0.87
Phylogenetic k-NN (k=3), out-of-clade	0.46–0.63

The models beat GapMind, the mechanistic pathway-completeness predictor, under random holdout (paired Wilcoxon across the 15 phenotypes, p < 0.001). They gave most of that margin back as soon as a whole dataset was held out.

Neither taxonomy nor the choice of algorithm explains the drop

Phylogeny is not the explanation. Restricting the cross-dataset test set to genomes that do have relatives in training, the in-clade split, recovered only about 0.03 of the drop, and the phylogenetic nearest-neighbour baseline itself fell to near null-model levels out of clade. The algorithm is not the explanation either. LightGBM, elastic net and random forest sat within a few points of CatBoost, and 88% of phenotype-by-regime cells tied within 0.05. That leaves the composition of the training set, which pools mechanistically inconsistent genotype–phenotype relationships and pushes models onto dataset-correlated shortcuts.

Training only on label-mechanism agreement changes what models learn

GapMind and the experimental label agree on 71.9% of labelled samples. Restricting training to those concordant samples, with feature space, architecture and evaluation unchanged, raised mean cross-dataset balanced accuracy from 0.646 to 0.68. The more useful effect was on what the models learned. Stable SHAP feature clusters shared between a model trained on three datasets and one trained on the held-out dataset alone rose from 6 of 15 phenotypes to 12, and the mean shared-cluster count per comparison from 0.5 to 1.3. A size- and class-matched random subset reproduced only the unfiltered level (0.38 ± 0.03), so the gain comes from concordance rather than from subsetting. For histidine the shared count went from 4 to 8 and the held-out-only clusters from 6 to 3, with the share of clusters sitting on the KEGG histidine map rising from 17% to 33%, trimming dataset-idiosyncratic features and keeping pathway features.

Adding 8000 more genomes did not do better than filtering

Adding over 8000 literature-curated BacDive genomes reached 0.73 against concordance filtering’s 0.71 on a far smaller set (paired Wilcoxon p = 0.20), and made feature selection markedly less reproducible.

A mechanism-free filter recovers the accuracy but not the recall

Concordance filtering needs a mechanistic predictor, which exists only for well-characterised pathways, so a mechanism-free alternative was tested. It uses a soft-label confidence filter over phylogenetic k-NN agreement, GapMind pathway completeness and the experimental label, sweeping the GapMind weight from 0.5 down to 0. It recovered the balanced accuracy (0.647 against concordance training’s 0.68, p = 0.30) but not the recall. Concordance-trained models beat GapMind’s true-positive recovery on 9 of 15 phenotypes and lost on 4, while the mechanism-free filter split 5 and 6. Neither passed GapMind’s own rule-based baseline of 0.69 on that test set.

A thousand pathway features transfer better than seventeen thousand

The focal phenotype’s GapMind columns alone matched the combined GapMind + KOFAM

RAST matrix, which holds around 17,000 features and is mostly RAST subsystem terms, under random holdout. Across datasets the smaller set improved on it (p < 0.001, q = 0.003).

The model recovers growth that the annotated pathway said was impossible

On the concordant portion of the cross-dataset test set the model reached a median balanced accuracy of 0.82, below GapMind, which is correct there by construction: general KOFAM annotations resolve organism-specific enzymes and transporters less precisely than curated pathway rules. Retraining the same models on GapMind step features lifts the median to 0.90 (p = 0.01), and the gap is worst for glucose at 0.51 against 0.77.

On the discordant portion the model still reached 0.34. It recovered growth that the annotated pathway said was impossible about twice as often as it corrected a present-but-silent pathway (44% against 19%, p = 0.02, higher in 12 of 15 phenotypes). Across the unstratified cross-dataset test the two effects cancel, and the mean shift against GapMind was undetectable (p = 0.80), because sensitivity gains came with lower specificity for 8 of 15 phenotypes. The rescued false negatives were mostly metabolic generalists, distinguished by broad-specificity transporters and sugar-catabolism regulators rather than by substrate-specific enzymes.

Confidence supports abstention but is not calibrated

Prediction confidence, max(p, 1−p), is the per-genome trust signal available at inference time. It ranks correct predictions above incorrect ones only modestly (ROC AUC 0.66 concordance-trained, 0.63 full-data), and both models are overconfident under dataset shift, so it supports ranking and abstention rather than calibrated probability. Abstaining on the least confident predictions lifted sucrose from 0.75 accuracy at full coverage to 0.88 at half coverage, against a majority-class rate of 0.61.

It does not work everywhere. m-Inositol was already at 0.82 and gained little, reaching 0.85 at half coverage. For glycerol the concordance-trained model never rose meaningfully above its majority-class rate at any coverage, while the full-data model stayed below it throughout and fell to 0.16 over its most confident tenth. Abstention cannot rescue a model that never learned the phenotype, and no label-free summary tested, including mean confidence and the fraction of high-confidence predictions, anticipated which phenotypes those would be.

Measuring the least confident genomes next beats measuring the most novel

In a retrospective simulation adding 25 held-out genome labels to the concordant training set, choosing the least confident genomes improved cross-dataset transfer more than choosing at random (mean Δ balanced accuracy 0.09 against 0.07, p = 0.002). Choosing the most novel or most diverse genomes did not beat random. The advantage tracked how badly the model already transferred: glucose 0.24 against 0.13, mannose 0.13 against 0.10, histidine 0.02 against 0.03.

What the model is and is not usable for

A genome-to-trait model of this kind is usable for prioritising candidates inside a declared applicability domain. It can flag likely growers for follow-up where recall matters more than precision, and rank which genomes are worth measuring next. It is not a general substitute for a mechanistic pathway call. Outside the concordant domain accuracy is low, the probabilities are not calibrated, and which phenotypes a model will handle reliably cannot be certified in advance. Generalisation here depends on label–mechanism agreement in the training data and on knowing where the domain ends, not only on sample count and algorithm choice.

Four datasets, three annotation tools, four split regimes

Four binary growth datasets were harmonised, covering 819 input strain records across 240 carbon sources, 795 of which survived genome quality control and 780 of which matched processed genomic features. The analysis was restricted to the 15 carbon sources measured in all four.

Dataset	Strains	Substrates	Feature-linked	Growth assay
ATLeaf	224	45	206	agar plate
Biolog	362	64	362	Biolog PM, colourimetric
Marine	175	100	157	liquid culture OD
Populus	58	190	55	liquid culture OD

Genomes were annotated with KofamScan, RAST and GapMind. KOFAM contributed around 5500 features after variance and correlation filtering, and GapMind around 1000. Both filters are label-free, so they ran once on the full feature matrix rather than per split, keeping the feature space identical across the regimes being compared. Classifiers were still fitted on training samples only.

Gradient-boosted trees (CatBoost) on the KOFAM matrix are the headline model, with LightGBM, elastic net and random forest as algorithm controls, and two baselines: GapMind and a phylogenetic k-nearest-neighbour classifier. Each was evaluated under four split regimes.

Regime	Test set
Random holdout	a random fraction of the strains
In-clade	a share of every phylogenetic cluster, so every test genome has relatives in training
Out-of-clade	whole clades lifted out of the pruned phylogeny
Leave-one-dataset-out	one of the four datasets entire

The splitters, classifiers and evaluation machinery are packaged as trait-prediction, a configuration-driven Python library in its own repository. The growth datasets and genome assemblies came from collaborators across ORNL, LBNL and ANL, with the processed data deposited alongside the analysis.