The Constraint Nobody Can Check
Continual learning is written as a promise about every prior capability and measured as an average over a handful.
Continual learning is written as a promise about every prior capability and measured as an average over a handful. Until that gap closes, “no catastrophic forgetting” is a statement about the benchmark, not about the model.
Thesis. The stability–plasticity problem is stated as a constrained optimisation: minimise the loss on what is new, subject to not degrading any of the old capabilities by more than a tolerance. The quantifier in that constraint ranges over every capability the system already has — a number in the millions for a frontier model. Every instrument the field uses reports an average over a few dozen benchmark tasks. An average moves in proportion to the fraction affected, so a small number of complete losses is arithmetically indistinguishable from noise; and the standard forgetting measure, which takes a maximum over noisy historical estimates, is biased upward even when nothing has been forgotten at all. The result is a metric that overstates diffuse drift and conceals catastrophic loss, which are precisely the wrong two errors. Until the measurement problem is addressed, replay, regularisation and parameter isolation are optimisation against an objective nobody can evaluate.
Abstract. I take the ten open components of continual learning in turn and sort them by what kind of problem each actually is. The stability–plasticity trade is real and structural, and the three method families surveyed in the current literature — architecture-based, regularisation-based, replay-based — do not remove it; each relocates its cost into parameter growth, constrained adaptation, or stored data plus privacy exposure. That much is well established. What is not established, and what this essay is mainly about, is whether the constraint can be checked. I show that the standard evaluation metrics are means, that means are blind to tails, and that the maximum operator inside the forgetting measure introduces a positive bias that grows with the length of the training history. I note that the field does not currently agree on what the phenomenon is — the same survey contains claims that degradation is misalignment rather than forgetting, and that it is activation bias rather than parameter drift — which means published mitigation results are claims about benchmark scores rather than about a characterised mechanism. Four of the ten items are not learning problems at all and belong elsewhere in the architecture; one, the conflict between deletion obligations and retention, has no answer I can offer. I close with what a defensible claim would look like: not lifelong learning, but a budget. Two figures, both schematic.
I. The constraint, read carefully
The stability–plasticity problem is usually written down and then moved past. It is worth stopping on.
Minimise the loss on the new task, subject to: for every prior capability i, the loss on that capability must not increase by more than ε.
Three features of that statement deserve attention before any method is proposed.
It is a constraint, not a penalty. A penalty trades off — some forgetting is acceptable if the new learning is valuable enough. A constraint does not trade off; it forbids. The distinction matters, because almost every deployed method is in fact optimising a penalised objective with a Lagrange multiplier that nobody reports, while the language used to describe it is the language of the constraint.
ε is doing enormous work and is never specified. How much degradation on a prior capability is acceptable? A tenth of a per cent? Two per cent? The answer differs by capability — degrading a model’s ability to write limericks is not degrading its ability to refuse a dangerous request — and yet ε appears as a single scalar, when it should be a vector indexed by consequence. A constraint whose tolerance is unstated is not a constraint. It is a mood.
The quantifier ranges over i. Over every prior capability. Not over the benchmark. Over what the model can do.
That third point is the whole essay, and I will come back to it. First, the part of the problem that is genuinely settled.
II. The trade is structural
The trade-off is not an engineering failure to be designed away. It follows from what makes these systems work.
A distributed representation is one in which a given parameter participates in many capabilities and a given capability is carried by many parameters. That property is the source of generalisation: a model can handle an input it has never seen because the features that compose it were learned across many other inputs. It is also, immediately and unavoidably, the source of interference. If parameters were partitioned by capability, updating one could not damage another — and the model would not generalise, because nothing would be shared. Sharing is the mechanism of both.
So the frontier is real. You can move along it and you cannot step off it.
Figure 1. The trade every continual-learning method is making. New competence on the horizontal axis, retained prior competence on the vertical. Frozen weights sit at the top left: perfect retention, no adaptation. Naive fine-tuning sits at the bottom right and, being inefficient, below the frontier rather than on it. Each method family buys retention with a different currency. Note where the ε-band intersects the frontier: if the constraint is taken seriously, it caps how much new competence may be acquired at all. Schematic — the positions illustrate the trade, and the frontier’s shape is invented.
The last observation in that caption is the one people skip. If the constraint is binding — if you really may not degrade any prior capability by more than ε — then you have capped the rate at which the system can learn anything new. That is not a flaw in the analysis. It is the honest reading of the constraint, and it explains why every practical system quietly relaxes it.
III. Three families, three currencies
The current survey literature sorts the methods into three paradigms, each motivated by an analogy to biological memory: modular organisation, synaptic consolidation, and hippocampal replay. I will take the methods seriously and the analogies lightly, for a reason worth stating: a biological analogy is a source of hypotheses, not evidence for them. That the brain appears to do something is a reason to try the corresponding mechanism. It is not a reason to believe the mechanism works, and the survey’s own results are the test of that, not the motivation.
Architecture-based methods isolate new knowledge in new parameters — expanded backbones, low-rank adapters, task-specific prompts — while freezing what came before. Interference is largely eliminated by construction, which is the point. Two costs follow. The first is growth: parameters accumulate with tasks, and “without indefinite computational growth” is on the list of unresolved requirements precisely because this family does not satisfy it. The second is subtler and, in the survey’s own words, a key challenge for the class: at inference, the system must select the appropriate sub-module for the input. You have not removed the problem of integrating knowledge; you have moved it from training time to inference time and renamed it routing. If the router is wrong, the knowledge is present and unreachable, which from the user’s side is indistinguishable from forgetting.
Regularisation-based methods keep one set of parameters and add a term penalising movement away from the previous solution — either in parameter space, constraining how far the weights may travel, or in feature space, constraining how far the representations may travel. The cost is stated plainly by the central challenge of the family: design a penalty that retains prior knowledge with minimal impact on acquiring new knowledge. That is the trade-off restated as an engineering task. There is no penalty that binds only on the parameters you did not need to move, because knowing which those are is equivalent to having solved the problem.
Replay-based methods keep a buffer of prior data — real or synthetically regenerated — and train jointly on it. This works well, and it works for an unmysterious reason: it partially reconstructs the joint training distribution that would have avoided the problem in the first place. The costs are the two the survey names for rejecting full joint retraining: storage, and data privacy. Some approaches store intermediate representations rather than raw samples to reduce both. That helps with volume. Whether it helps with privacy is an empirical question about what can be reconstructed from stored representations, and it should not be assumed.
None of the three removes the trade. Each converts it into a different currency: parameters, constrained adaptation, or retained data and the obligations that come with it.
IV. The quantifier
Now the part that is not settled.
The constraint quantifies over every prior capability. What does the evaluation quantify over?
The standard benchmark suites in this literature are, in ascending order of size: five text-classification datasets; that set extended with a handful of GLUE and SuperGLUE tasks and IMDB; eight datasets spanning domain knowledge, multilinguality, code and mathematics, with three delta metrics for general ability, instruction following and safety; and, at the top end, a large instruction collection of 1,616 tasks across 76 task types and 55 languages, from which continual-learning studies typically draw a subset.
Set that against the object being measured. A frontier model’s competences are not enumerable. They are compositional: knowledge of a domain crossed with a language crossed with a register crossed with a format crossed with a reasoning depth. Whatever the true number is, it is not five, and it is not 1,616.
So the honest description of current practice is this: the constraint is asserted over an unenumerated space, and evaluated on a sample from that space which was not drawn to represent it, whose coverage of it is unknown and unestimated, and which is fixed rather than resampled. Nothing in that procedure supports a claim about the capabilities not in the sample. It supports a claim about the sample.
That is not a rhetorical complaint. It is the ordinary standard applied to any inference from a sample to a population, and it is not met. A survey researcher who sampled 1,616 people from a population of unknown size by unknown mechanism, and then reported a population parameter without an interval, would not be published.
And the sample is not merely unrepresentative; there is reason to think it is biased in a specific direction. Benchmarks are built from capabilities that can be measured cheaply and scored automatically, which means capabilities with abundant labelled data, clear correctness criteria and heavy representation in the training corpus. Those are, on any plausible account of how distributed representations degrade, among the most robust things the model knows — carried redundantly by many parameters, reinforced from many directions, and therefore the hardest to dislodge.
The capabilities most at risk are the opposite in every respect: thin, idiosyncratic, supported by little data, awkward to score, and consequently absent from the suite. A minority language. A rare document format. A specific refusal behaviour that fires on a narrow class of prompt. A domain convention that appears in a few thousand documents worldwide.
So the selection mechanism that produces the benchmark is correlated with the property being measured, and correlated in the direction that flatters the result. That is not sampling noise; it is a bias with a known sign. Any estimate of retention drawn from such a sample is an upper bound on true retention, and it should be reported as one.
V. A mean cannot see a tail
It gets worse when you look at what the metrics actually compute.
The forgetting measure is defined per task as the difference between the historically highest accuracy on that task and its current accuracy, and then reported as the mean of that quantity across all previously seen tasks. Backward transfer is likewise a mean, over tasks, of the difference between current performance and performance immediately after that task was learned.
A mean is a statistic about the centre. It moves in proportion to the fraction of the population affected. Which means it is, by construction, unable to distinguish between two situations that could not be more different:-
every capability degrades slightly and uniformly;
-
almost everything is untouched, and a small number of capabilities are destroyed outright.
Figure 2. A mean cannot see a tail. Left: the distribution of per-capability change when twenty of a thousand capabilities are destroyed and the rest are untouched. Right: how many must be destroyed before the average moves by more than a plausible detection threshold. Stipulated arithmetic, not measurement — but the relationship is arithmetic, not empirical, and holds for any threshold you choose.
Work the numbers. Suppose a thousand capabilities, one of which is destroyed — a drop of 0.78 in whatever units you like — and the other 999 unchanged. The mean shifts by 0.00078. Set the detection threshold anywhere reasonable given seed-to-seed and sampling variation, and you can lose dozens of capabilities completely before the reported metric moves at all.
Now recall which of the two situations we actually care about. Uniform mild degradation is a nuisance. The destruction of a specific capability — the ability to refuse a particular class of request, the ability to handle a minority language, the ability to do a specific safety-relevant check — is the failure that matters, and it is the one the metric is structurally incapable of seeing.
The remedy is not exotic. Report the distribution, not its centre. Report the worst case. Report the count of capabilities whose degradation exceeded a stated threshold. All three are trivially computable from data that is already collected, and none of them appears in the standard metric set.
VI. The other error, in the opposite direction
Having established that the metric is blind to catastrophe, note that it is simultaneously biased toward reporting forgetting that did not occur.
Look again at the definition. The per-task forgetting is the maximum over all previous evaluation points of that task’s accuracy, minus its current accuracy.
Suppose a capability is entirely unaffected — its true accuracy is constant throughout training. Each evaluation of it is nevertheless a noisy estimate: finite test sets, sampling temperature, prompt sensitivity, seed. So the recorded history is a constant plus noise. The maximum of a set of noisy estimates of a constant is larger than that constant, in expectation, and the gap grows with the number of draws — roughly with the square root of the logarithm of the history length.
Subtract the current (unbiased) estimate and you have a positive number. Nothing was forgotten. The metric reports forgetting anyway, and reports more of it the longer the model has been trained.
Two consequences follow, and they point in opposite directions, which is what makes this bad rather than merely imprecise:-
Comparisons across methods with different numbers of evaluation points are contaminated. More checkpoints, more apparent forgetting, no difference in the model.
-
A method that reduces reported forgetting may have reduced evaluation noise rather than forgetting. Fewer probes, smoother estimates, more stable prompts — all lower the maximum, and all lower the metric, and none of them retain a single capability.
Correcting this is not difficult. The bias can be estimated by simulation under a null of no forgetting and subtracted; or the maximum can be replaced by a smoothed or pre-registered reference point. But it has to be done, and the results reported to date have not done it, so the direction of the error in any given published comparison is unknown to me and, I think, to the authors.
I should say where the general point comes from. In work I have in progress on the measurement of long-horizon model capability, the minimum detectable change per capability, under realistic replication counts, sits well above the tolerances these constraints are written to; and the variance introduced by defensible analyst choices exceeds the variance from task sampling. That work is unpublished and I am not going to lean on its numbers here. The structural argument above does not need them: the bias of a maximum over noisy draws is a fact about order statistics, and the blindness of a mean to a tail is a fact about arithmetic.
VII. The field does not agree what the phenomenon is
Here is the finding that should trouble anyone reading mitigation results.
Within the same current survey, two different lines of work locate the cause of degradation in two different places, and neither is the folk account. One holds that observed degradation in continual instruction tuning often stems from misalignment rather than forgetting, and shows that freezing lower layers to stabilise low-level representations improves performance across benchmarks. Another identifies biased function activations rather than parameter drift as the main cause, and mitigates it by maintaining activation consistency across tasks.
Take those seriously and the standard picture — new gradients overwrite old weights, knowledge is lost — is at least incomplete and possibly wrong for some of the cases it is invoked to explain. If the knowledge is still in the parameters but is not being reached, that is not forgetting. It is a routing or alignment failure, and it has different remedies, different failure modes, and different implications for whether the capability can be recovered.
Now apply the ordinary standard. A field that has not converged on what the phenomenon is cannot have established that a given intervention mitigates it. What has been established is that certain interventions raise certain benchmark scores. That is a real finding and a much smaller one, and the gap between the two is currently being crossed by the word “forgetting” doing work it has not earned.
This is the same structure as the argument I made about “hallucination”. A word that names a mechanism you have not demonstrated will organise your research programme around a mechanism that may not exist.
VIII. Four items that are not learning problems
Of the ten open components, four are misfiled. They are not stability–plasticity problems and they will not be solved by better replay.
Updating a fact without creating contradictions elsewhere. This is a belief-revision problem, and it has a formal treatment. If claims are represented as propositions with justification links, then updating one traverses the dependency graph, re-evaluates what depended on it, and surfaces contradictions where the update conflicts with something else held. If claims are represented as weight patterns, the update has no locus, its consequences have no addresses, and the contradiction it creates is undetectable in principle. The problem is not that gradient descent is bad at fact editing. It is that a system with no propositional representation has nowhere to put the dependency.
Separating local user knowledge from generally valid knowledge. This is a scoping problem, and the answer is to attach scope to the claim at representation time — a contextual signature recording the domain, the source, and the conditions under which the claim holds — so that “my invoice terms are net 30” and “invoice terms are net 30” are different propositions that cannot be confused, because they differ in a field rather than in a nuance of phrasing. Retrofitting scope onto a system that stored both as text is not possible, because the distinction was destroyed at ingestion.
Deciding which experiences deserve permanent storage. This is a triage problem with a decision-theoretic answer: keep what has the highest expected effect on future correctness, discount what is redundant with what is already held, and let unreinforced material decay. All three are specifiable. None requires solving continual learning, and all three are easier if the store is a structured claim base rather than a buffer of raw examples.
Detecting poisoned or deceptive learning experiences. This is a security problem, and treating it as a learning problem is a category error with consequences. The defences are source admissibility, provenance verification, credibility weighting of the source, and quarantine of material that contradicts a well-supported base — all applied before the material reaches the learner. A system that ingests first and hopes the optimiser is robust has already lost, because a sufficiently well-crafted poisoned example is, by construction, one the optimiser finds convincing.
The pattern across all four is the same. Each becomes tractable when the unit of storage is a claim with provenance, scope and dependencies, and each is intractable when the unit is a weight. That is an argument for a hybrid architecture, and it is the same argument I have made about epistemic status: some things cannot be repaired inside the generator because the generator has no representation for them.
IX. Two that really are learning problems
Two items on the list are not misfiled, and I want to be as clear about those as about the others, because a critic who reclassifies everything has usually stopped thinking.
Learning a new skill from a handful of demonstrations. This is a genuine learning problem, and it is in direct tension with stability for a reason that follows from the same structure as everything else in this essay. If three examples are to change the model’s behaviour, each must carry a large share of the update. Large updates are precisely the condition under which shared parameters move far enough to disturb what they were also carrying. Sample efficiency and stability pull against each other, and they pull for the same reason plasticity and stability do: the update is not localised, because nothing in the representation is localised.
There is a second problem underneath, and it is epistemic rather than optimisational. From three demonstrations you cannot determine whether you have learned the skill or the surface features of those three demonstrations. Neither can the system, and — this is the part that connects to everything I have written about calibration — it has no channel to report the difference. A generalisation supported by three points and a generalisation supported by three million are held in the same way, expressed in the same register, and applied with the same confidence. The right output after three demonstrations is a provisional skill, marked provisional, applied with a stated tolerance and revised on the next contradicting case. That is a representational capability the system does not have, and adding a claim base does not supply it, because the object in question is a procedure rather than a proposition.
Consolidating episodic experience into general knowledge. Also genuine, also open, and with a trap in it that is worth naming because the obvious solution walks straight into it.
The obvious consolidation signal is repetition: what recurs across contexts is general, what occurs once is incidental. In a corpus, repetition is confounded with popularity, and popularity is not validity. A system that consolidates on frequency will consolidate the most-repeated falsehood in its input, promote it from episode to background knowledge, and thereafter treat it as the settled ground against which new claims are judged. This is the fallacy of mere probability applied to memory rather than to assertion, and it is worse there, because a consolidated error is no longer visible as a claim at all.
The criterion has to be evidential rather than frequency-based: consolidate what has survived attempted disconfirmation, what has been independently corroborated, what carries provenance from sources that have been right before. Some of that machinery I have specified elsewhere. But specifying the criterion is not the same as knowing how to apply it inside a gradient-based learner, where the object being consolidated is a distributed pattern and not a claim you can attach a replication score to. I do not have that mechanism, and I have not seen one.
X. The one I cannot answer
Remembering useful information while satisfying deletion and privacy requirements is on the list, and I do not have an architecture for it.
State the conflict exactly. A deletion right requires that, on request, the effect of a datum be removed. An audit requirement — the kind I have argued for elsewhere, where every belief traces to an immutable, tamper-evident origin — requires that the record of what was believed and why cannot be altered after the fact. These are not in tension at the margin. They are opposed at the core.
Partial answers exist and should be named honestly for what they cover. Committing hashes rather than content to the permanent record lets you delete the content while retaining proof of what was there and when: this satisfies the audit requirement and does not satisfy the deletion requirement, because a hash of a small-domain datum is not meaningfully private. Replay buffers can store representations instead of raw data, which reduces exposure by an amount nobody has measured.
And beneath both sits the harder fact: a model that has trained on a datum has been changed by it, and deleting the datum does not reverse the change. There is a research literature on removing a training example’s influence from a trained model. I have not read it, and I am not going to characterise its state from the outside. What I can say is that the requirement as stated — remember what is useful, forget what you are told to forget, prove you did both — is not satisfied by anything in the three method families above, and that the conflict is legal and architectural before it is technical.
XI. What a defensible claim looks like
I do not think anyone should promise lifelong learning. I think they should promise a budget.
The shift is the same one I have argued for in capability evaluation generally: stop asserting a property and start accounting for the uncertainty in the assertion. Applied here, that means five things, none of which requires a research breakthrough.
Pre-register the capability set. Name, before training, the capabilities the constraint is being asserted over, and the probes that measure each. A capability that is not on the list is not covered by the claim, and the claim should say so rather than implying universality by silence.
Report the distribution, not the mean. Per-capability deltas, with intervals. The worst case. The count exceeding a stated tolerance. The mean may be reported alongside; it must not be reported alone.
State ε per capability, by consequence. The tolerance for a safety-relevant refusal behaviour is not the tolerance for stylistic range, and pretending otherwise is how the important losses get averaged into the unimportant ones.
Correct the estimator. Debias the maximum, or replace it. Report the noise floor — the change that could not have been detected — so that a null result can be read as “no degradation above X” rather than as “no degradation”.
Hold out probes that are never trained against. The moment a capability set is used to select methods, it stops measuring retention and starts measuring fit to the capability set. This is the standard problem with any fixed benchmark, and continual learning is more exposed to it than most, because the whole point is repeated adaptation.
Concretely, the header of such a report would read something like this. Capability set: 412 pre-registered probes across nine domains, of which 340 are public and 72 are held out and never used for method selection. Tolerance: 0.5 percentage points for safety-relevant behaviours, 2 points for domain competence, 5 points for stylistic range. Replications: twelve seeds per probe. Noise floor: changes below 1.4 points on a single probe were not detectable at this replication count, and are reported as null rather than as zero. Result: mean change −0.3 points; worst single probe −11.2 points; four probes degraded beyond tolerance, listed by name; one probe improved beyond the noise floor. Estimator: the maximum-over-history operator was debiased by simulation under a null of no forgetting, which reduced apparent forgetting by 0.6 points.
Nothing in that paragraph is hard. All of it is computable from data a continual-learning study already generates. What it costs is the ability to write a single reassuring number in an abstract, which is why it is not done.
A model that ships with that accounting is making a claim that can be checked, and therefore a claim that can be wrong, and therefore a claim worth something. A model that ships with a low forgetting measure on eight datasets is making a claim about eight datasets.
XII. What would show me wrong
If someone demonstrates that performance on a modest, well-chosen probe set predicts performance across the unenumerated remainder — that the sample really does stand for the population, with an estimated coverage and an interval — then the quantifier objection largely dissolves. That is an empirical question about the structure of capability space and it has an answer. I do not know what it is. Nobody has told me what it is, which is different from it being unanswerable.
If the estimator bias I describe turns out to be small relative to the effects being reported, then section VI is a technical footnote rather than a problem. That is checkable by simulation in an afternoon and I would like to see it done, in either direction.
And if the attribution dispute resolves — if degradation in these systems turns out to be one phenomenon with one mechanism after all — then mitigation results become claims about that mechanism, and section VII loses its force. I would treat that as good news, and I would expect the resolution to change which methods look promising, which is exactly why the question matters now rather than later.
XIII. Coda
The stability–plasticity constraint is stated with a universal quantifier and evaluated with an average over a convenience sample, using an estimator that is biased upward by noise and blind to the failures that matter. That is not a criticism of the methods. Replay, regularisation and parameter isolation are all reasonable engineering, and the trade they are negotiating is real.
It is a criticism of the claim. “No catastrophic forgetting” is currently a statement about a benchmark, made in language that sounds like a statement about a model. The distance between those two things is not a rounding error; it is the entire question of whether these systems can be deployed for years without silently losing the capabilities we depend on and did not think to measure.
The useful system is not the one that promises to learn forever. It is the one that can tell you, with an interval, what it has lost.
Sources
-
Guo, H., Zeng, F., Zhu, F., Wang, J., Wang, X., Zhou, J., Zhao, H., Liu, W., Ma, S., Wang, D.-H., Zhang, X.-Y., & Liu, C.-L. (2025). Continual Learning for Generative AI: From LLMs to MLLMs and Beyond. arXiv:2506.13045 (v4, 24 August 2025). — problem formulation; evaluation metrics (average and last accuracy, forgetting measure, backward transfer, zero-shot transfer); the three-paradigm taxonomy; benchmark suites; and the attribution results discussed in section VII.
Note on the title: this work is indexed as Continual Learning for Generative AI: From LLMs to MLLMs and Beyond, not as Continual Learning in Generative Models. It is a preprint and has not, so far as the record shows, been through peer review.