Back to notes

ReLearn: bioreactor control learned rather than derived

notes · microbial ecology · reinforcement learning
Schematic. The agent learns the policy from what the bioreactor returns.

A chemostat holding a microbial community offers one usable control, the composition of the feed, and no practical way to write down what changing it will do. The species interact through their metabolisms, nonlinearly and depending on the environment they sit in. ReLearn learns the control law instead of deriving it. It pairs a community simulated with dynamic flux balance analysis with a reinforcement learning agent that sets the inflowing concentrations. On the two-species test system the agent held the community near where it already was and did not reach new targets. The work stopped at training, not at the formulation.

The control law cannot be written down by hand

Two species sit in a chemostat. Each is unable to make a metabolite the other secretes, and both draw on one shared carbon source. The target is a particular ratio of the two, and the only control is the composition of the feed. Deriving the control law would mean tracing how a change in the feed propagates through both metabolisms and back out through cross-feeding, which is where the equations stop being tractable by hand.

The motivation is biomanufacturing. A single engineered species carries the whole metabolic load and may not be able to catalyse every reaction a pathway requires. Synthetic consortia split the pathway across organisms instead, but they come with no design procedure, because the organisms and their environment depend on each other.

ReLearn takes the other route. Rather than deriving the policy, it simulates the community, lets an agent try feed adjustments, and keeps what moves the system towards the target.

This was the last chapter of a PhD at Boston University, advised by Daniel Segrè, with Pankaj Mehta advising this project.

The agent held the community in place but did not reach new targets

The first case study used two cross-feeding auxotrophs in a chemostat. It was chosen because it is small enough that the correct answer is known in advance, which makes it a check on the method rather than a result in itself. The agent could hold the community near the composition it started from, but it did not reliably move it to a new target. One single-worker A2C run looked promising and then failed to reproduce, and the vectorised version did worse.

A second case study was planned around a lignocellulose-degrading community, aiming at a compound rather than a composition. It was scoped but never run.

The obstacle was training, not the formulation

The loop itself works. A community is simulated by flux balance analysis, an agent acts on the feed, and a reward is computed from the resulting composition. What the agent did not do was learn a policy good enough to steer with. The obstacles were training stability, how the reward was shaped, and how long a horizon the agent was given, rather than anything about how the problem was set up. That is where the work stopped.

How the environment and agents were set up

The environment combines species abundances, nutrient concentrations and a shared carbon source. Its action is the composition of the inflowing feed, and its reward measures distance from the target community composition. One version uses Monod growth equations, and the other replaces them with a flux balance solve for each species at every step.

DQN, A2C and PPO agents from stable-baselines3 were evaluated over the same control loop. The results came from the actor-critic models, with the reward normalised per species so that the same absolute error did not weigh equally on small and large populations.