The Microbial Discovery Forge: an AI co-scientist over a data lakehouse

The Microbial Discovery Forge is an AI co-scientist and research observatory for microbial genomics, developed under BERIL, the BER Intelligent Layer. It runs as a set of skills inside a coding assistant, connects to the KBase BER Data Lakehouse (BERDL) for data, and keeps a shared knowledge base of what earlier projects learned. I built it and am its main contributor. Paramvir Dehal is also a major contributor.

Knowing the schema is not enough to query BERDL well

BERDL is large. The gene table has a billion rows, genome_ani has 420 million, and the AlphaEarth environmental embeddings cover 83K of 293K genomes. Getting a useful answer out of it depends on knowing which identifiers line up, which collections can fairly be compared, and which queries will finish at all. None of that is recorded in the schema, and so far every project has worked it out again from scratch.

An agent that can query a billion rows will always find something

The second problem is the agent itself. When a system can query data at this scale, the risk is not that it finds nothing. The risk is that it finds something in every direction it is pointed, and after a hundred such attempts there is no way to distinguish a result from an artefact of having looked repeatedly.

Skills, a registered plan, and an independent reviewer

The Forge reads from BERDL, which holds pangenomics, mutant fitness, biochemistry, multi-omics and structural biology data as Iceberg tables over Ceph object storage, queried with Spark SQL. Data products the observatory generates are written back into a microbialdiscoveryforge namespace.

Abilities are written down as skills rather than typed as prompts

The agent’s abilities are recorded as skills instead of being retyped as prompts each time. They cover the steps a project actually goes through: identifying which tables hold the relevant data, writing and running the queries, searching the literature, moving a dataset in or results out, and recording what was learned. Because they are written down once, a fix to any of them carries over to every later project rather than staying in one person’s chat history.

The plan is registered before any query runs

A project begins with a research plan that states the question, the competing hypotheses, which tables will be used and how, what each hypothesis predicts, and the confounders already known. The plan is recorded before any query runs. If it changes later, the change is dated rather than overwritten. This ordering is what distinguishes a result from one of the many directions that happened to be tried.

A second agent reviews the work

The review is carried out by a separate agent that has not seen the first one’s working. It reads the finished project together with the schema documentation and the accumulated notes, and assesses the method, the reproducibility, and whether each claim is supported by the analysis behind it.

Notebooks and lessons about the data are kept alongside the answer

Two things are retained beyond the answer itself. Every notebook is saved with its output, so the path from raw table to figure can be re-read rather than taken on trust. Anything learned about the data is written back with the project that found it, including which identifiers align, which comparisons are safe, and which queries are too expensive to run. The next study starts from that record instead of rediscovering it.