G2PAI: a genotype-to-phenotype data lakehouse
- genomes
- traits
- measurements
one store
G2PAI is a genotype-to-phenotype AI effort at Oak Ridge National Laboratory, building models that learn the link between what an organism carries in its genome and what it does. The data those models read sits in a shared lakehouse, fed by GPTgp, by PMI and by other projects. This note describes the data model and the pipelines that fill it.
The sources do not join without a shared model
Storage is not the difficult part. The difficulty is that every source arrives in its own shape, and the shapes disagree. Plant imaging, genomics, eQTL studies and photosynthesis experiments each use different identifiers and measurement conventions. A survey of 431 genome folders found roughly a third of them using non-standard annotation columns.
The microbial side of the programme has the same problem in a different vocabulary, with half a dozen annotation tools producing half a dozen output formats. That side is normalised upstream by TraitWeaver, whose Parquet tables are one of the sources feeding the lakehouse.
The platform is shared across several programmes
The lakehouse is built at Oak Ridge and shared across GPTgp, PMI on microbial communities, and WINTR on winter biology, with links out to KBase and BRIDGE. It is a separate store from KBase’s BERDL, which the Microbial Discovery Forge note describes. It has two parts.
| Component | What it covers |
|---|---|
| Lakehouse infrastructure | Storage, catalog and ingestion. Iceberg tables plus scientific data files (FASTQ, BAM, VCF, images) in object storage, project-scoped and multi-tenant. |
| Data model and harnesses | Schemas (LinkML), validation (Pydantic), lineage, the file registry, and the client library every governed write goes through. |
Data lands in three zones: Structured (LinkML-validated Parquet, internal to the
pipeline), Published (governed Iceberg tables visible to users) and Personal
(per-user space for experimentation). Tables carry a flat two-level name such as
leafweb.gas_exchange_canonical, and users read through Trino or work in
notebooks. Notebooks and Airflow operators are the canonical writers, and there
is no privileged path that bypasses the harness.
One LinkML schema generates validation, documentation and diagrams
The schema is written in LinkML rather than as a collection of hand-written classes. A single model generates the validation layer, the field documentation and the ER diagram across six domains.
| Domain | What it connects |
|---|---|
| Genomics | Genomes, genes, gene calls, variants and genome metadata |
| Phenotype | Species, genotypes, experiments, sites, samples and measurements |
| Photosynthesis | Gas-exchange curves and their individual measurements |
| Annotation | Functional annotations, GO terms and cross-references |
| Ontology | The controlled vocabularies behind those annotations |
| Images | Species images and their provenance |
The shape is a star schema. Dimensions carry the identity of a genome, gene,
species, genotype, experiment, site or leaf sample; facts carry the measurements
that hang off them; and fact_gene_go_term bridges genes to the GO ontology.
These six domains cover the plant genomics and phenotyping side of the
programme. The microbial trait tables are dimensions and facts as well, but they
carry their own keys and their own schema, in TraitWeaver.
Measured fields carry units and ontology mappings in the schema itself. The ETL then turns source files into validated Parquet tables, preserving where each measurement came from and giving every entity a stable identity.
Generated schemas and computed identifiers keep the tables joinable
Because everything is generated from one schema, the documentation and the validation code cannot drift away from the data. There is a single description, and the rest is derived from it.
Identifiers are computed from the source record rather than assigned at load time, so ingesting the same file twice produces the same keys and the tables still join. Every field has a declared type, which prevents a missing value from silently turning a numeric column into text partway through a load.
There is also a small application that draws the schema as a graph of tables and the keys between them. Six domains is more than anyone holds in their head, and being able to point at the diagram in a meeting has been worth the afternoon it took to build.