Does mini-AGI forget? A small-scale replication

The short version

The README's headline replicates: reading one subject with a slowly-updated shared trunk leaves the other subjects intact. What it demonstrates is a known principle, clearly: forgetting tracks how far the shared weights move. Here that looks like roughly the square of the displacement, as in Biderman et al.'s "learns less, forgets less". It also means the protection holds for a bounded amount of reading, not indefinitely.

Forgetting = mean change in held-out loss on the 7 subjects not being read, nats/char. Pairs are seed 0 / seed 1 throughout.

• The trunk's learning rate is the lever. A 1× trunk forgets ~30× more than a 0.1× trunk; the experts' rate barely matters.
• The architecture's distinctive parts don't do the protecting. Running every rate at 0.1× forgets as little as the trunk/expert split, and freezing the expert working set changes nothing.
• Forgetting grows roughly with the square of the trunk's displacement (learning rate × optimizer steps). That's why a 4× difference in step density produced a ~25× difference in forgetting.
• The floor is horizon-bound. Forgetting stays at the control level for about 0.5M characters, then climbs steadily, while chess itself improves by only ~0.04 nats.
• Recovery replicates: about 75% of heavy damage returns within 131k characters of mixed reading, then plateaus at 80–86%.

Claim by claim

How we tested it

1 · The headline probe replicates

The 0.1× trunk stays within our noise floor for the whole probe; the control's own spread (−0.05 to +0.02) is the measure of that floor. The contrast with a 1× trunk is ~30×. Removing the trunk/expert split, by slowing the experts too, changes nothing.

2 · Step density, and why it matters

Our first version ran every probe at 4× upstream's optimizer steps per character: we shrank the chunk from 2,048 to 512 characters to fit the scaled model, without matching the step count. Holding everything else fixed and varying only the steps:

• Steps and learning rate are one lever. Adam normalizes each update to roughly the learning rate in size, so how far the trunk moves per character is about learning rate × steps per character. Varying either lands on the same curve, within seed noise.
• The curve is close to quadratic. At small displacement, the slope on a log-log plot is 2.1 on both seeds, flattening toward 1–1.5 as the damage saturates. That's why a 4× difference in steps produced a ~25× difference in forgetting.
• More displacement bought no learning. Chess gains ~0.02 at the smallest displacement and ~0.01 at the largest. The extra movement is pure cost.

The practical consequence: tokens per optimizer step is a hidden hyperparameter in any streaming-learning claim. A forgetting number without the learning rate, the tokens per step and the horizon isn't comparable across setups. It nearly misled this replication.

3 · Read for longer

Forgetting holds at the control level through about 0.5M characters, then climbs steadily to +1.49 / +1.09 by 4M: 65% / 75% of the gain over chance retained. Chess improves by just +0.045 / +0.038 over the same stretch. The README's "never leaves the floor" is accurate for the window it measured; the square law explains why the window is finite.

4 · Recovery

Reading all subjects recovers 79% / 77% within 65k characters. By then only 2 of the 8 subjects have been read, so recovery is general rather than per-subject relearning, which fits the README's "displacement, not destruction". Recovery then plateaus at 80–86%, with +0.64 / +0.69 nats still missing after 1M characters. (Subjects rotate every 32,768 characters, so the README's 131k window covers 4 of the 8.)

What this demonstrates

The result is a clean, cheap demonstration of a principle established in prior work, reproduced in a byte-level, sparse-expert model that learns from a stream.

• Learns less, forgets less. Biderman et al. (2024) showed that constraining how much a model's weights can change during fine-tuning (there, with low-rank LoRA updates) preserves capabilities outside the target domain, at the cost of learning less of it. mini-AGI constrains the same thing a different way: a slow learning rate on the shared trunk. Our results show the "forgets less" half directly. Forgetting falls with the trunk's displacement, whether that displacement is reduced by the learning rate or by the number of steps. The "learns less" half is invisible here, because the probe subject was already learned (see below).
• Why it's roughly quadratic. A base trained to a minimum on its subjects has near-zero gradient on them, so a small step costs nothing to first order, and the loss rises only through curvature: ΔL ≈ ½·δᵀHδ. Mirzadeh et al. (2020) analyse forgetting exactly this way, as a function of displacement and curvature, and show training-regime choices such as learning rate and batch size move it. Our slope of ~2 at small displacement, and the flat-then-rising curve over 4M characters, are what that view predicts.
• Slowing the shared layers is established practice. Layer-wise learning rates, lower in the shared body than the head, are a standard fine-tuning tool (Howard & Ruder, 2018). The trunk multiplier is that idea applied to continual reading.
• The curvature view also points past uniform slowing. EWC (Kirkpatrick et al., 2017) penalizes movement weighted by each parameter's importance to earlier tasks: the same quadratic, made selective.

What's specific to mini-AGI is where the effect lives. The expert pool confines each update to a minority of experts, yet that doesn't protect anything; the dense shared trunk carries the interference. That's a useful data point, given how often sparse or modular architectures are proposed as a continual-learning mechanism.

Paths that would qualify the claims further

These are the open questions this replication couldn't settle, offered as directions rather than gaps in the original work:

1. A subject the model hasn't learned. The chess probe improves chess by only ~0.02–0.04 nats, so it measures stability, not plasticity. Probing with a subject held out of base training would show the "learns less" side of the trade-off. The quadratic view predicts that learning grows linearly with displacement while forgetting grows quadratically, so slower reading should improve the gain-to-forgetting ratio, at the cost of time.
2. Step count vs learning rate at matched displacement. On one seed, accumulating gradients forgot about 3× less than lowering the learning rate to the same displacement; on the other it didn't. If averaging out batch-1 gradient noise is a real bonus, accumulation is the better knob for streaming learners. More seeds would settle it.
3. Selective vs uniform slowing. An EWC-style or L2-to-anchor penalty on the trunk, compared with the 0.1× trunk at equal learning, would test whether protection can be targeted rather than bought with a uniformly slow trunk.
4. Base maturity. The quadratic picture assumes the base sits at a minimum. On a less-trained base the first-order term returns; a short pilot on a 12-minute base showed large forgetting even at 0.1×.
5. Scale. Our trunk is a quarter of upstream's. A larger trunk may have a wider, flatter basin and hold the floor longer.

Caveats

• Scale. 2.1M trunk parameters against 8.3M; the bases start the probe at a comparable loss on the unread subjects (1.24–1.28 vs 1.12).
• Reconstructed probe. Rebuilt from the README's description; where it is silent we took the most literal reading.
• Mixed densities in supporting results. The headline, trunk-1×, no-split, control and 4M results are at matched density. The frozen-working-set comparison and the recovery curve were measured at 4× density; their conclusions (no effect; ~75% then a plateau) don't depend on the absolute level.
• Two seeds. They differ in magnitude by up to 2–3× but agree on every verdict.

Notes for the author

1. Tokens per optimizer step is worth reporting with any forgetting number. Forgetting here scales roughly with the square of learning rate × steps per character, so the same 524k-character probe gives +0.03 or +0.73 depending only on the chunk size. We learned this by getting it wrong first.
2. The corpus builder drops the chat_hermes held-out set. Hermes writes its held-out files to data/val/chat, then the synthetic-chat expand step wipes that folder, so a fresh python -m corpora all can't produce the README's eight held-out subjects. Noted on PR #11, which fixes the wipe as a side effect.
3. GradSNR.observe crashes when a parameter's grad is None on some steps: a depth-1 step leaves the halting head gradless. Reported in #19; fix proposed in #20.
4. The chess probe barely teaches the model anything (≤0.04 nats even over 4M characters). A probe on an unlearned subject would test the full stability-plasticity trade-off.
5. The probe tooling is referenced but not published (tools/, runs/cl/, runs/results/). Publishing it would make the headline directly checkable.

Reproduce

python -m corpora all        # upstream corpus, then restore chat_hermes (see code/README.md)
python harness/run_upstream.py configs/rep_s.yaml read upstream/mini-AGI/data/train --save \
  --weights-dir runs/rep_s_seed1/weights --held-out upstream/mini-AGI/data/val \
  --sample-every 0.67 --minutes 240 --seed 1 --shuffle-seed 1
harness/run_accum4.sh        # headline arms at upstream's step density (--accum 4)
harness/run_stepdensity.sh   # the same probe at 1, 2 and 4 chunks per step
harness/run_matrix.sh runs/bases/s_seed1 runs/probes/s_seed1 0    # first-version arms (4x density)
python harness/summarize.py runs/probes/*/*/probe.json && python harness/figures.py v2

Code, configs, every probe's raw results (probe.json) and the full deviation and run log (NOTES.md) are published at huggingface.co/spaces/dreddnafious/mini-agi-replication. The two summary commands above regenerate every table and figure from them without a GPU.