Every AI agent today is an amnesiac intern. You can teach it an astonishing amount inside one context window. Then the context window ends, a new intern arrives, and all it has are the notes the previous one left behind. The notes can get better but the intern does not.

This is why continual learning feels like the holy grail of AI research. We have a recipe for creating intelligent machines. What we do not really have is a way for one of those machines to keep learning from experience over long time horizons.

To be clear, I mean a narrower problem than the one the big labs usually care about. They want to take a very general model and update it with the last six months of internet data without breaking too many things. In some manner of speaking, that problem is intractable by definition: you are trying to continuously update an enormous and poorly specified distribution whilst preserving everything useful about the old one.

I care more about the LLM intern ie a model doing one specific job for one company, team or user, even if that job is very agentic. How do we make that model continually learn everything it needs to know to get better at its job?

Humans are at least a proof of existence for this version of the problem.

The missing layer of memory

Language models are incredibly sample efficient at learning things when the information is inside the context window (arguably more sample efficient than humans). But once you exceed the context window, typically somewhere between hundreds of thousands and around a million tokens, you have to do a bunch of hacky stuff to keep taking advantage of this in-context learning mechanism.

LLMs currently have two native forms of memory. The first is perfectly lossless but very expensive: the context and its KV cache. The model can attend directly to everything that happened, but the cache gets very large very quickly. Full attention over longer sequences is expensive during prefill, KV cache memory grows linearly with sequence length, and it is hard to scale this indefinitely.

The second is incredibly compressed and stable: the weights themselves. An enormous amount of information and capability is packed into them, and retrieving that information at inference time is extremely cheap. But updating the weights without breaking the model is very hard. There is almost nothing in between.

We as a field have therefore pushed on a few different directions.

The first is extending the context window. People have tried linear attention and its variants, where we maintain some fixed-size hidden state that information can be written into, alongside many other tricks for making attention and KV caches cheaper. But we have not really managed to go orders of magnitude beyond current context lengths (around a million tokens) without giving something up. This feels very hard to scale.

The second is building harnesses with stateful memory: memory Markdown files, skill files, natural-language compaction, vector search over old conversations and increasingly elaborate rules for deciding what to load into context. This is hacky, but it is also what has worked best so far.

The third is compressing the context window itself into some learned latent representation.

The third is the direction I am most excited about. Humans do not have a perfectly lossless working memory containing a transcript of everything that has happened to us. Our working memory is tiny and lossy, but we seem able to continuously compress experience into several layers of longer-term memory.

This is what our work on Still has been about. It was inspired by work on Cartridges and more analytical compaction methods such as fast KV cache compaction. I think there is now enough evidence to say that this works in some form. We will eventually have granular control over compaction in neural/latent space, rather than being limited to compaction in token space through natural-language summarisation.

Suppose, for now, that we can take a very long context and compress it into a much smaller latent cache without losing much important information. The question then becomes:

Should we use the information in those caches to update the weights, or should we focus on loading the right cache at inference time?

Should knowledge be retrieved or baked in?

My initial intuition is that it seems stupid to continually look something up if there is a way to bake it into your brain.

@justinskycak (the Math Academy guy, who writes a lot of good stuff on the science of learning) discusses this in terms of automaticity. Through repeated, deliberate practice, especially retrieval practice, foundational knowledge and low-level procedures become deeply embedded in long-term memory. They become instantly and effortlessly available. You no longer have to consciously derive them, reconstruct them or use up scarce working-memory slots every time.

For example, a basketball player does not have to think through how to dribble whilst running a play. The dribbling is automatic, freeing cognitive resources for higher-level decisions. Without automaticity on the basics, everything feels effortful and fragmented.

You can see the same problem with language models. Frontier agents often feel like they spend 95% of their time searching for information, reconstructing what happened previously and rederiving procedures they have already performed many times. If we are going down the route of specialising models for particular jobs anyway, it feels natural that repeatedly useful knowledge and skills should eventually be baked into the weights.

But maybe they don’t need to be. Some people argue that if a system (a model plus a harness, memory files, skill files and tools, where everything except the model evolves) functions equivalently to a continual learner, then we do not need to bother with gradient updates. We can trust in-context learning to use whatever is placed in the context window. Our only job is to figure out what to store and what to load.

I think this is probably suboptimal for three reasons.

The first is the boring one: cost. If there is eventually a way to do the same task with 95% fewer tokens (and I do think frontier models spend 95% of their tokens just searching), margin pressure will make the incentives to use it too hard to ignore. If continual learning into weights for specialised models is possible, people will be highly incentivised to pursue it until it works.

The second is skill acquisition. External memory works best when the thing being remembered can be cleanly written down, such as a fact or the location of a database. But many behaviours are difficult to specify explicitly. You do not learn excellent tool use, software engineering or negotiation merely by loading a longer instruction file. There is a reason RL has become so popular for instilling skills after pretraining and for post-training specialisation.

The third, and I think most important, is that hand-engineering a harness does not feel very bitter-lesson-pilled. Maybe it still counts as the Bitter Lesson if the model continuously rewrites its own harness, memory system and retrieval policy. I would be more willing to accept that. But even then, it has to spend a huge number of tokens figuring out how to reduce the number of tokens it spends searching. It never gets around the fact that the underlying model is still an amnesiac intern.

Dwarkesh has a good analogy here. Imagine you are trying to teach someone the trumpet. Every 15 minutes, the student disappears and a new student arrives. The new student has never touched a trumpet. All they have are the notes left by the previous student.

You can make the notes extraordinarily good. But at some point, you probably want the student to actually learn how to play the trumpet.

The problem is that writing to weights cooks the model

Unfortunately, our work has increasingly pushed me towards an uncomfortable conclusion: the obvious alternative (continually updating the weights) does not work very well either.

There is an important distinction here between knowledge acquisition and skill acquisition. At first glance, knowledge acquisition seems easier. Surely teaching a model that a customer renewed their contract should be easier than teaching it a complex new behaviour. In a one-off setting, it probably is. But in the iterative regime, which is the regime that actually matters for continual learning, I increasingly think knowledge acquisition may be harder than skill acquisition.

Skill acquisition more or less works if you have a sufficiently good environment and a proper RL reward. Defining the environment and reward is hard, but at least we have a recipe. The model takes actions, receives feedback and gradually becomes better at the behaviour. As LLMs get better, autoevals will be constructed from traces of models running in production on the fly.

Knowledge is stranger. We do not have a reliable way to repeatedly write new facts into a model’s weights without eventually overwriting old facts, degrading its capabilities or reducing its ability to learn subsequent information.

A system that successfully acquires knowledge over time needs to pass three tests:

1. It does not overwrite useful information from the base model.
2. It does not forget information added during previous updates.
3. It does not degrade the model’s general capabilities, such as instruction following, reasoning or its ability to learn new things.

When you look across the spectrum of ways we might write to the weights from SFT, to offline distillation, to on-policy distillation, to RL, I do not think any of the vanilla methods passes all three tests.

SFT is probably the clearest example. In an iterative setting, it can catastrophically forget information from the base model, overwrite information introduced during previous updates and degrade the model’s general capabilities. It fails on all three fronts.

Offline and on-policy distillation seem somewhat better, as does RL. But none appears robust in the repeated-update regime. And repeated updates are the whole point. We are discussing continual learning, not one-off fine-tuning.

RL also does not necessarily acquire explicit knowledge very well. A model can learn a policy that succeeds in an environment without cleanly internalising the underlying facts. So it might fail at knowledge acquisition for a more basic reason. We have some research coming out shortly that substantiates a lot of these claims.

There are, of course, variants that show promise.

ECHO may allow RL to acquire explicit facts whilst continuing to learn capabilities. Pedagogical RL, along with variants of on-policy self-distillation, may mitigate some of the degradation, although these methods have not yet been made stable in the continual regime.

And to be honest maybe the objective is simply misspecified. @b_geist made a good point to me: if you reward both correctness and efficiency, perhaps a model will eventually learn to memorise recurring facts about its environment because repeatedly searching for them costs tokens. It could discover automaticity through optimisation.

But there is currently nothing that works out of the box for taking a model trained for a particular task and then allowing it to continually acquire knowledge over thousands of updates without breaking something important.

Maybe in-context learning is not a workaround

My current working thesis is that pretraining and RL spend enormous amounts of compute optimising the model’s in-context learning mechanism, and that subsequent weight modification destructively interferes with it.

The model learns how to rapidly adapt to information in context. We then attempt to make that adaptation permanent by changing the substrate that implements the adaptation mechanism itself. Plasticity research points in the same direction. Recent work shows that as a model is repeatedly fine-tuned on new tasks, it can lose plasticity and become progressively worse at learning subsequent tasks.

A possible conclusion is that we should lean into ICL fully rather than fighting it. What does this look like? Attention-only transformers + a large MLP database loaded dynamically? A database of compressed KV caches that we load in with vector search/HNSW? Regardless, these might both work better than pure harnesses and memory markdown files because search over latent space is more principled than search over token space.

You can imagine something like:

• A lossless context containing what is immediately relevant.
• A compressed latent episodic memory containing recent experience.
• The weights containing highly consolidated knowledge and capabilities.

The real problem may be learning what information belongs at each level, when it should move between levels and how much compression it can tolerate. Humans do not appear to store every memory in the same format eg some things remain episodic and some skills become automatic. So a continual learner may need the same kind of hierarchy!

Continual learning is not one problem

I guess my point here is that in order to tackle continual learning, you have to be able to specify the problem in the first place. Continual learning isn’t one big amorphous problem; rather it varies tremendously depending on how you define it (eg knowledge vs skill acquisition, narrow vs general), and only in really thinking about the characteristics of what you need it to do do you get some traction on figuring out what your approach should even be. It feels very much like blind men groping at different parts of an elephant at the moment (one working on catastrophic forgetting, one working on memory hierarchies, etc). We probably just need to get enough surface area on what we’re actually trying to do with a continual learning system before we can agree on the best way to build it.

Machines that follow rules

A computer is a machine that follows instructions. You give it a sequence of steps (“add these two numbers,” “if the result is negative, go to step 7,” “otherwise, write down a 1 and continue”), and it executes them one after another until it reaches a stopping point or runs forever.

In 1936, Alan Turing described an imaginary device, now called a Turing machine, that captures everything we mean by “mechanical computation.”[1] It has a tape (infinite memory), a head that reads and writes symbols, and a finite table of rules telling it what to do next. Every algorithm ever written, every program running on every server and phone and satellite, is equivalent to some Turing machine. The physical details (silicon transistors, vacuum tubes, pen and paper) change the speed, but not what is computable.

This is the Church–Turing thesis: anything that can be computed by any reasonable physical process can be computed by a Turing machine. It has never been proven; it functions more as a law of nature that has never been violated. Every proposed model of computation (lambda calculus, register machines, cellular automata, DNA computing) turns out to compute exactly the same class of functions.

Problems and their sizes

When we talk about a “problem” in computer science, we don’t mean a single question (“is 91 prime?”) but an infinite family of questions parameterised by size:

• “Given a number with n digits, is it prime?”
• “Given a graph with n nodes, does it contain a triangle?”
• “Given a logical formula with n variables, is there an assignment of true/false that makes it true?”

The size parameter n determines everything about tractability. An algorithm that works for 10 variables might take longer than the age of the universe for 300 variables, depending on how its running time grows with n.

Polynomial vs. exponential: the only distinction that matters

There are many possible growth rates, but for the deepest questions in computer science, only one distinction matters:

Polynomial growth means the running time is bounded by some fixed power of n: maybe $n^2$ steps, or $n^3$, or $n^{17}$. These are “fast” algorithms. Doubling the input size might multiply the time by 4 (if quadratic) or 8 (if cubic).

Exponential growth means the running time is something like $2^n$. Now doubling the input size squares the running time. An algorithm that takes one second for $n = 50$ would take roughly 34 million years for $n = 100$. No amount of engineering helps. Faster hardware and better compilers don’t matter against an exponential.

This is a statement about scaling laws, not current technology. Polynomial-time algorithms are feasible; exponential-time algorithms are not, for any input of practical size.

The Landscape of Difficulty

P: problems we can solve quickly

P is the class of problems that some algorithm can solve in polynomial time. Sorting a list, multiplying matrices, finding shortest paths in networks, determining if a number is prime. All in P.

If a problem is in P, we consider it “tractable.” There exists a procedure that will always give you the correct answer, and whose running time is bounded by a polynomial function of the input size.

NP: problems whose solutions we can check quickly

Now consider a different kind of question. Suppose someone hands you a massive logical formula, thousands of variables, millions of clauses, and asks: “Is there some assignment of true and false to these variables that makes this formula true?”

You might have no idea how to find such an assignment efficiently. But if someone handed you a candidate assignment, you could check it almost instantly: just plug in the values and see if the formula evaluates to true.

This is the class NP: problems where a correct solution, if one exists, can be verified in polynomial time, even if finding one might be hard.

Every problem in P is automatically in NP (if you can solve it quickly, you can certainly check a solution quickly). Whether the reverse holds is the most important open question in computer science, and arguably in all of mathematics:

Does P = NP?

Can every problem whose solutions are easy to check also be solved quickly?

Almost no one believes so, but no one has proven otherwise.

NP-completeness: the hardest problems in NP

In 1971, Stephen Cook showed that some problems in NP are, in a precise sense, as hard as anything in NP.[2] If you could solve any one of these problems in polynomial time, you could solve all of NP in polynomial time.

These are called NP-complete problems. The canonical example is 3SAT:

Given a Boolean formula in a specific form — a conjunction (AND) of clauses, where each clause is a disjunction (OR) of exactly three variables or their negations — does there exist an assignment of true/false values that satisfies every clause?

For example: $(x_1 \lor \lnot x_2 \lor x_3) \land (\lnot x_1 \lor x_2 \lor x_4) \land \ldots$

With 3 variables, you can check all $2^3 = 8$ possibilities. With 300 variables, there are $2^{300}$, more than the number of atoms in the observable universe. The best known algorithms are sophisticated forms of brute-force search.

3SAT is NP-complete: if you could solve it in polynomial time, every problem in NP would fall. Scheduling, protein folding, circuit design, logistics. Thousands of practical problems are NP-complete or NP-hard. A fast algorithm for any one of them would be a fast algorithm for all of them.

Beyond NP: the class PP

NP asks: “Does a satisfying assignment exist?” Now consider a harder question:

“Do more than half of all possible assignments satisfy the formula?”

This is a counting question rather than a search question. Finding one solution or verifying one isn’t enough; you need to know something about the proportion of solutions.

The class PP (Probabilistic Polynomial time) captures this. Formally, a problem is in PP if there exists a randomised polynomial-time algorithm that gives the right answer with probability strictly greater than 1/2, but perhaps only barely greater. Maybe it’s right with probability $\frac{1}{2} + \frac{1}{2^n}$. That razor-thin margin is what gives PP its power.

PP is believed to be vastly more powerful than NP. It contains the entire polynomial hierarchy, an infinite tower of complexity classes built by stacking NP-like quantifiers (“there exists… for all… there exists…”). PP lets you not just find needles in haystacks but compare the sizes of haystacks.

Why this hierarchy matters

The classes, ordered by believed difficulty:

Each inclusion is believed to be strict: each class is thought to contain problems harder than the one before. But none of these separations have been proven. Proving that any two are different remains open.

Quantum Mechanics in Five Steps

Step 1: States are vectors

In classical physics, a coin is either heads or tails. A bit is either 0 or 1. The state of the system is a definite configuration.

In quantum mechanics, the state of a system is a vector in an abstract mathematical space (a Hilbert space). For a qubit (the quantum analog of a classical bit), the state is:

where $\alpha$ and $\beta$ are complex numbers satisfying $|\alpha|^2 + |\beta|^2 = 1$.

This is not a statement about ignorance. The system is not “really” in state $|0\rangle$ or $|1\rangle$ with the answer hidden from us. The superposition is the physical state. The system has no definite value until something forces one.

Step 2: Evolution is linear and unitary

Between measurements, quantum systems evolve according to unitary transformations, linear operations that preserve the total “length” of the state vector.

Linearity means: if you apply an operation $U$ to a superposition $\alpha|0\rangle + \beta|1\rangle$, the result is $\alpha \cdot U|0\rangle + \beta \cdot U|1\rangle$. The operation acts on each component independently, and the coefficients pass through untouched. Linearity is a law of physics, not a design choice. All known physical interactions are described by linear, unitary quantum mechanics.

Linearity matters because you cannot selectively amplify one branch of a superposition without affecting others. If a state has a tiny component (say $\alpha = 10^{-100}$), no physically allowed operation can magnify that component while suppressing the rest, without the operation being exponentially costly or violating unitarity.

Step 3: Measurement and the Born rule

When you measure a qubit in state $\alpha|0\rangle + \beta|1\rangle$, you get:

• outcome $0$ with probability $|\alpha|^2$
• outcome $1$ with probability $|\beta|^2$

This is the Born rule. It connects the mathematical formalism (complex amplitudes) to observable frequencies. After measurement, the system “collapses” into whichever state was observed.

The probabilities are determined by the squared magnitudes of the amplitudes. An amplitude of $10^{-100}$ gives a probability of $10^{-200}$. Such outcomes never happen.

Step 4: Entanglement and decoherence

When two quantum systems interact, they can become entangled: their joint state cannot be described by specifying each system’s state independently. Measuring one system instantaneously determines facts about the other, regardless of distance.

In practice, quantum systems constantly interact with their environments: air molecules, photons, thermal fluctuations. This process, called decoherence, entangles the system with the environment so thoroughly that quantum interference effects are destroyed. The system behaves classically.

Decoherence is why you don’t see cats in superpositions. Quantum mechanics doesn’t stop applying at large scales; the quantum coherence required for quantum effects is destroyed almost instantaneously by environmental interactions.

Step 5: Quantum computing — what it actually buys you

A quantum computer maintains a register of $n$ qubits. The state of this register is a vector in a $2^n$-dimensional space:

There are $2^n$ amplitudes, one for each possible $n$-bit string. A quantum computation applies a sequence of unitary operations (quantum gates) to this state, then measures.

This looks like massive parallelism: $2^n$ components being processed simultaneously. But the bottleneck is extraction. When you measure, you get a single outcome $x$ with probability $|\alpha_x|^2$. All the other information is lost. You can’t “read off” the results of $2^n$ parallel computations.

Quantum algorithms work by carefully engineering interference — arranging the amplitudes so that wrong answers cancel out and right answers reinforce, funnelling probability toward the correct output. This is difficult, and only works for problems with specific mathematical structure.

The class of problems solvable by polynomial-time quantum algorithms is called BQP (Bounded-error Quantum Polynomial time). BQP is believed to be larger than P (Shor’s algorithm for factoring is the star example), but it is believed to not contain all of NP.

Quantum mechanics gives you some speedup over classical computation, but not an unlimited one. It does not, as far as anyone can tell, let you solve NP-complete problems efficiently.

The Many-Worlds Interpretation

What it says

The many-worlds interpretation of quantum mechanics, proposed by Hugh Everett III in 1957, takes the mathematical formalism at face value and refuses to add anything to it.[5]

In the standard (“Copenhagen”) interpretation, measurement causes a real, physical collapse: the state $\alpha|0\rangle + \beta|1\rangle$ becomes either $|0\rangle$ or $|1\rangle$, with Born-rule probabilities. Collapse is sudden, irreversible, and not described by the Schrödinger equation. It is a second law of nature grafted onto the first.

Everett asked: what if there is no collapse? What if the Schrödinger equation (linear, unitary evolution) is the only law, applying always and everywhere, including during measurement?

The consequence: when a measurement occurs, the universe doesn’t collapse into one outcome. All outcomes occur. The measuring device, the observer, the entire environment become entangled with the quantum system, and the universal wavefunction branches:

Each branch is equally real. In one branch, you see 0. In another, you see 1. Neither branch has access to the other; decoherence ensures they evolve independently from that point on. You experience one branch because you are in one branch. The “you” in the other branch is having the complementary experience.

What it does not say

Many-worlds says all branches exist. It does not say:

1. You can choose which branch you end up in.
2. You can communicate between branches.
3. You can influence the Born-rule probabilities governing which branch “you” experience.
4. The existence of all branches gives you computational power beyond BQP.

Point 4 is the important one. Yes, in some branch, a quantum computer “computes” every possible answer. But this is true of an ordinary coin flip in many-worlds too. In some branch, the coin lands heads. That doesn’t help you use the heads-branch result if you’re in the tails-branch. The branches are causally disconnected after decoherence.

Many-worlds provides no computational upgrade. It is an interpretive framework.

The Anthropic Computing Idea

The provocation

The idea, phrased as a thought experiment:

Suppose you want to solve 3SAT. You have a formula $\varphi$ with $n$ variables. You proceed as follows:

1. Generate a uniformly random assignment $x \in {0,1}^n$.
2. Check whether $x$ satisfies $\varphi$.
3. If it does not — eliminate the current branch.

If you take “eliminate the branch” literally in a many-worlds context, then: across all $2^n$ branches of the random assignment, only the branches where $x$ satisfies $\varphi$ survive (contain observers). If you are observing anything at all, you must be in a branch where $x$ satisfies $\varphi$. You’ve solved 3SAT, and all you did was guess randomly and apply a filter.

There’s a subtlety for unsatisfiable formulas (what if no assignment works, and you’ve eliminated yourself in every branch?). This is handled by adding a small escape clause: with some tiny probability, say $2^{-2n}$, you skip the whole procedure and do nothing. If the formula is satisfiable, you’ll almost certainly end up in a branch that found a satisfying assignment. If you find yourself in the “did nothing” branch, you can conclude with overwhelming probability that the formula was unsatisfiable.

The term for this is anthropic computing: computation in which the probability of the observer’s own existence depends on the computational output.

Why it seems to work

The logic runs:

1. In many-worlds, all random outcomes occur in some branch.
2. If you arrange for your existence to be contingent on a correct answer, then conditional on existing, you have the correct answer.
3. The conditioning is free — you don’t need to search; you just need to be.

This appears to give polynomial-time solutions to NP-complete problems (just guess and check — the checking is polynomial, and the “search” is handled by the branching).

Why it doesn’t work (in known physics)

The problem is the conditioning step. In standard quantum mechanics, you cannot “condition on existing” in the way the argument requires.

The Born rule assigns probabilities to measurement outcomes based on squared amplitudes. If $\varphi$ has exactly $k$ satisfying assignments out of $2^n$ total, the probability of landing in a satisfying branch (by random guessing) is $k / 2^n$. For a hard instance, $k$ might be 1, making the probability $2^{-n}$, exponentially small.

You can’t make this probability larger without doing exponential work, because the laws of physics are linear. No unitary operation can take an exponentially small amplitude and boost it to a constant without either:

• applying exponentially many operations, or
• violating linearity/unitarity (i.e., breaking quantum mechanics).

“Condition on surviving” is doing, by fiat, exactly what the laws of physics prohibit: selecting the exponentially unlikely outcome as your experience, at zero cost. In the language of quantum information, this is postselection, which is not a physically available operation.

Postselection — What It Is and What It Costs

Definition

Postselection means: perform a computation that produces some output with probability $p > 0$, then act as if that output is guaranteed.

Concretely: run a quantum (or classical) computation, measure a qubit (or bit), and condition on the outcome being 1, ignoring all runs where it was 0, no matter how many there are. If the probability of seeing 1 is $2^{-n}$, you’d need to repeat the experiment roughly $2^n$ times before you’d actually see it. But postselection says: pretend you saw it on the first try.

In a many-worlds framing, postselection is: “only count the branches where the desired outcome occurred.” It is anthropic selection: the observer exists only in favourable branches.

As a mathematical tool, it’s well-defined

There’s nothing wrong with postselection as a theoretical device. You can define complexity classes based on it and prove rigorous theorems. The question is whether it corresponds to a physically realisable operation.

Aaronson’s theorem: PostBQP = PP

In 2005, Scott Aaronson defined PostBQP: the class of problems solvable by a quantum polynomial-time computer augmented with the ability to postselect on measurement outcomes (provided those outcomes have nonzero probability).[4]

He proved:

Recall that PP is the class of problems decidable by a randomised algorithm that’s correct with probability just barely above 1/2. It contains the entire polynomial hierarchy. It is believed to be far more powerful than NP.

Postselection doesn’t just give you NP. It gives you PP, the ability to count solutions, compare exponentially large sets, and resolve questions well beyond NP-type search.

There is also a classical version. Classical anthropic computing (random guessing plus postselection, without quantum mechanics) gives power equivalent to a class called BPP$_{\text{path}}$, which sits between MA (a randomised version of NP) and BPP$^{\text{NP}}$. Weaker than PP, but well beyond polynomial time.

What this means

If “anthropic computing” were physically realisable, if you could condition your experienced reality on computational outputs at polynomial cost, you would have:

• The ability to solve 3SAT efficiently.
• The ability to break all public-key cryptography.
• The ability to solve every problem in PP in polynomial time, a class that contains the entire polynomial hierarchy.

What Would Have to Be True About the Universe

The question, restated

Many-worlds alone doesn’t give you postselection. Quantum mechanics, as we understand it, doesn’t give you postselection. So if anthropic computing were possible, if you could reliably end up in the branch where the computation succeeded, what would that tell us about the laws of physics?

The algorithm doesn’t work in known physics. The interesting question is what the universe would have to look like for it to work.

Something would have to give

For postselection to be physically available, at least one of the following must be false:

Linearity of quantum mechanics. If the evolution of quantum states is not linear, if nonlinear modifications of the Schrödinger equation describe real processes, then it becomes possible to amplify exponentially small amplitudes without exponential cost. The tiny branch where you guessed the right answer could be “boosted” to macroscopic probability.

As a hypothesis this is not exotic, but the consequences are severe. In the early 1990s, Nicolas Gisin and Joseph Polchinski independently showed that any nonlinear modification of quantum mechanics, however small, enables faster-than-light signaling. The nonlinearity allows you to distinguish non-orthogonal quantum states with certainty, which can be leveraged through entanglement to transmit information superluminally. This destroys relativistic causal structure.

The Born rule as the correct measure over outcomes. The Born rule says you experience outcome $x$ with probability $|\alpha_x|^2$. If the actual rule were different, if there were a physical law saying “observers preferentially find themselves in branches with certain computational properties,” then you could, in principle, experience the satisfying branch disproportionately often.

This is a strange kind of modification. It would mean the probability measure over observer-experiences is not determined by the dynamics (unitary evolution + Born rule) but by something else, something that depends on what the computation computes. Physics would have to “know” about the semantic content of computations, not just their physical implementation.

No-signaling / relativistic causal structure. Postselection is closely related to the ability to signal faster than light. If you can condition your experienced reality on events in your future light cone (as anthropic computing requires (you “survive” based on a future computation’s outcome)), you are introducing a form of retrocausality or nonlocal influence that sits uneasily with relativity.

Thermodynamic constraints. Postselection is, in a sense, a Maxwell’s demon operating on branches of the wavefunction. It extracts useful information (the satisfying assignment) from a uniform mixture (random guessing) without paying the entropic cost. This violates the spirit, if not the letter, of the second law of thermodynamics.

The measure problem: where cosmology meets computation

In modern cosmology, particularly in models involving eternal inflation, there is an unsolved problem called the measure problem. In an eternally inflating universe, every possible configuration of matter occurs infinitely many times in infinitely many regions. To make any prediction, you need a way to assign relative probabilities to different observations. You need a measure: a rule for weighting different observer-moments.

Different choices of measure lead to different empirical predictions. Some measures predict we should observe a very young universe (the “Boltzmann brain” problem). Others predict different values of cosmological constants. The “right” measure is not determined by the equations of general relativity or quantum field theory alone. It is an additional input.

Anthropic computing is a proposal for a very specific kind of measure: one that weights observer-moments by computational outcomes. You find yourself disproportionately in branches where the computation succeeded, not because of any dynamical mechanism, but because the measure over histories is rigged in your favour.

The cosmological measure problem, weaponised as a computational resource. And the fact that cosmologists cannot currently solve the measure problem, cannot determine from first principles which measure is correct, means that dismissing anthropic computing is harder than it might seem. You can’t say “obviously the Born rule is the right measure” because, in the context of eternal inflation and many-worlds, what constitutes “the right measure” is precisely what’s in dispute.

There is a strong pragmatic argument against computation-dependent measures: they would imply a universe so different from the one we observe that the resulting predictions would likely be falsified immediately. If our measure over experiences were biased by computational outcomes, we would expect to see all sorts of statistically anomalous patterns in our observations, and we don’t.

The Halting Problem Boundary

What even postselection cannot do

Suppose we grant postselection. Suppose the universe really does allow you to condition on computational outcomes at polynomial cost. Does this give you unlimited computational power?

There is a hard boundary.

The halting problem asks: given an arbitrary program $M$ and input $x$, does $M$ eventually halt (stop running), or does it run forever?

Alan Turing proved in 1936 that no algorithm can decide this for all programs. It is undecidable, not merely hard (like NP-complete problems), but provably impossible for any Turing machine, regardless of time or resources.

This is a different kind of impossibility than NP-completeness. NP-complete problems are (probably) hard because the best algorithms take exponential time, but they are decidable; given enough time, you can always determine the answer by exhaustive search. The halting problem cannot be decided by any algorithm in any amount of time.

Why postselection doesn’t cross this boundary

3SAT is decidable. For a formula with $n$ variables, you can check all $2^n$ assignments; the answer is always determined in finite (though possibly exponential) time. Postselection speeds this up; it collapses the exponential search into a polynomial-time conditioning trick, but the underlying problem was always solvable in finite time.

The halting problem is different. A program might run for a googol steps and then halt, or it might run forever. To decide halting, you would need to resolve an infinite-time property of a computation. Postselection operates on finite computations with definite outcomes; it conditions on a specific measurement result. It doesn’t help you with questions whose answers require observing a computation for an unbounded amount of time.

Could you try an anthropic approach? Run the program and stipulate: “I only exist in branches where I eventually observe the program halting.” If the program halts, you observe it halting, fine. But if it doesn’t halt, what do you experience? You’d need to experience “not halting” as a definite outcome in finite time. But “this program hasn’t halted yet” is not evidence that it won’t halt; it might halt in the next step, or in $10^{10^{10}}$ steps.

To convert “I haven’t observed halting” into a reliable certificate that the program never halts, you would need an additional mechanism: something like a cosmic time limit, or a rule that says “if it hasn’t halted by time $T$, conclude it won’t.” But choosing the right $T$ is itself an undecidable problem (it depends on the busy beaver function, which grows faster than any computable function). Any such rule is effectively smuggling in an oracle (a source of uncomputable information), which goes beyond mere postselection.

Postselection expands the class of tractable problems from P to PP but does not cross the boundary between computable and uncomputable.

The Principle

What kind of law would forbid anthropic computing?

We’ve mapped out the consequences: if anthropic computing worked, you’d get PP in polynomial time, you’d likely break no-signaling, and you’d need a non-Born measure over observer-moments that depends on computational outcomes. Our universe appears to forbid all of this.

What kind of law is doing the forbidding?

Consider an analogy. Before Einstein, the speed of light was a measured quantity, something that happened to be about $3 \times 10^8$ metres per second. Einstein’s move was to promote this empirical fact to a structural principle: the speed of light is not just fast; it is a speed limit, a constraint that shapes the geometry of spacetime and dictates the form of every physical law.

This was not a discovery of a new fact. It was a change in the status of a known fact. The numerical value didn’t change; its role in the architecture of physics did. The speed of light went from being a property of electromagnetism to being a property of spacetime itself.

The computational speed limit

There may be an analogous principle at work in the relationship between physics and computation:

No physical process can convert exponentially small measure into guaranteed experienced outcomes at polynomial cost.

This is a statement about what the laws of physics permit, not about any particular algorithm or physical system. It says that the structure of physical law, whatever its specific form, respects a constraint on the relationship between computational difficulty and physical feasibility.

In our universe, this constraint appears to be enforced by several interlocking mechanisms:

• Linearity and unitarity prevent the selective amplification of exponentially small amplitudes.
• The Born rule ties experienced probabilities to squared amplitudes, ensuring that low-amplitude branches are experienced with correspondingly low probability.
• Decoherence destroys the coherence needed to access other branches, making inter-branch interference (and thus inter-branch information transfer) impossible at macroscopic scales.
• Thermodynamic irreversibility imposes entropic costs on information extraction that mirror the computational costs of search.
• No-signaling constraints (enforced by relativistic causal structure) prevent the statistical steering tricks that postselection would enable.

No single one of these mechanisms, taken alone, is obviously sufficient to enforce the prohibition. But together they make anthropic computing physically impossible in our universe. Whether this web is a coincidence of our particular physical laws or reflects something deeper is still open.

Is the prohibition independent or derived?

Option A: Derived. The inability to postselect is a consequence of linearity, the Born rule, and relativistic causality. No new principle is needed; the existing laws are sufficient. The “computational speed limit” is a theorem, not an axiom. This is plausible: linearity alone arguably rules out the relevant kinds of amplitude amplification, and the Born rule fixes the measure over outcomes.

Option B: Independent. The computational constraint is a new principle that is not derivable from the existing laws in their standard formulation. It would need to be added as an additional axiom, a “computational postulate” alongside the dynamical postulates of quantum mechanics. This would be more surprising, but also more informative: it would suggest that computation plays a foundational role in physics, not just a descriptive one.

We don’t know which is correct. We don’t even have a sharp enough formulation of the principle to state the question precisely. But treating computational constraints as potential physical laws is an active direction at the boundary of physics and computer science.

Postselection as a fictional-but-useful primitive

Even if postselection is physically impossible, even if anthropic computing is definitively ruled out, the mathematical framework of postselection has proven useful.

Aaronson’s proof that PostBQP = PP provides a clean, conceptually transparent route to structural results about complexity classes. The original proof that PP is closed under intersection (due to Beigel, Reingold, and Spielman in 1995) used hard classical combinatorics.[6] The PostBQP = PP proof gives you the same result through a quantum argument that feels almost obvious in comparison.

This pattern recurs in theoretical physics and mathematics: imaginary capabilities, taken seriously as formal devices, yield real insight. Feynman’s path integrals sum over physically impossible trajectories. Boltzmann’s statistical mechanics imagines ensembles of systems that don’t exist. The “many worlds” of the many-worlds interpretation may or may not be real, but the mathematics works either way.

Postselection is another such device. It doesn’t need to be physical to be useful; the mathematical relationships it reveals are properties of computation and physics, independent of whether any universe actually allows it.

Conclusion

The “anthropic computing” thought experiment starts as a trick (guess and die, survive only if you guessed right) and opens into a probe of the relationship between physics and computation.

What it reveals:

1. Many-worlds alone doesn’t help. The existence of all branches does not let you select among them. Branching gives you BQP, not NP or PP.

2. The real primitive is postselection, the ability to condition experienced reality on computational outcomes at polynomial cost. Quantum mechanics as we understand it does not provide this.

3. If postselection were physical, you wouldn’t just get NP. You’d get PP, a class that dwarfs NP and contains the entire polynomial hierarchy.

4. Even postselection has limits. It doesn’t cross the Turing barrier. Undecidable problems remain undecidable.

5. The universe appears to enforce a computational speed limit: you cannot convert exponentially small measure into guaranteed experience without exponential cost. This prohibition is maintained by linearity, the Born rule, thermodynamics, and no-signaling constraints working together.

6. Whether this speed limit is a derived theorem or an independent principle remains open. If it’s independent, computation is not just something we do within the universe; it is something the universe itself is constrained by.

Opus 4.6 and GPT-5.3-Codex were both released last week. And beyond the hooha and the kerfuffle and the METR benchmarks and the posturing over who is going to develop AGI first, I thought of a now quite famous scene[1] from The Office.

Michael Scott has left Dunder Mifflin to found his own paper company. It’s going well — suspiciously well. He’s stealing clients left and right, because his prices are impossibly low. What Dunder Mifflin doesn’t know is that the Michael Scott Paper Company is about to go broke. Michael assumed fixed costs when planning, didn’t realise they’d grow as he scaled, and that’s how he made prices so cheap, and that’s how he stole so many clients.

But Dunder Mifflin has some idea of this. So David Wallace, the CFO, makes his move. He offers Michael a buyout and leans on the obvious leverage:

“Your company is four weeks old. I know this business. I know what suppliers are charging. I know you can’t be making very much money. I don’t know how your prices are so low, but I know it can’t keep up that way. I’m sure you’re scared. Probably in debt. This is the best offer you’re gonna get.”

And Michael (who is, underneath all the buffoonery, occasionally the most perceptive person in the room) says:

“I’ll see your situation and I’ll raise you a situation. Your company is losing clients left and right. You have a stockholder meeting coming up and you’re going to have to explain to them why your most profitable branch is bleeding. So they may be looking for a little change in the CFO. So I don’t think I need to wait out Dunder Mifflin. I think I just have to wait out you.”

Michael wins. Not because his position is strong (it isn’t, he’s weeks from bankruptcy) but because he correctly identifies that Wallace’s position is weaker than it looks on a shorter timescale. Wallace has a stockholder meeting. Michael just has to outlast the clock.

I think about this when I think about open-source AI.

The Intelligence Ceiling

Every model has an intelligence ceiling: the fraction of economically valuable work it can automate at a cost and quality where you’d be irrational not to deploy it. This is a lower bar than human-level performance — a model that’s 80% as good as a domain expert but 200x cheaper and runs in seconds has cleared it comfortably for an enormous category of work. And this ceiling isn’t fixed. It rises along two independent axes: models get smarter, and inference gets cheaper. Both expand the set of tasks worth automating, and they compound.

But most of that ceiling can only be reached through specialisation. A general-purpose frontier model, accessed through an API, will clear the bar for some tasks out of the box — the ones that look like the internet, the ones where general capability is enough. But most enterprise work lives in the long tail: proprietary edge cases, domain-specific data formats, error modes that only surface in production, workflows where “good enough” means something precise and unforgiving. For that work, you need to train the model on your domain. You need to run reinforcement learning against your own reward signal, on your own data. You need open weights.

This is worth being precise about. I’m not making a vague claim that fine-tuning is useful. I’m saying that for the majority of economically valuable tasks, the gap between what a general-purpose model can do via prompting and what a specialised model can do via post-training is large.

This is what I saw building specialised models at Parsed, and what I continue to see at Baseten: a sub 80B model fine-tuned on a client’s claims data will outperform GPT-5.2 on that client’s claims workflow, not because it’s a smarter model, but because it’s been trained on the actual distribution it needs to perform on. And it will do it faster, cheaper, and without sending proprietary data to a third party.

Sholto Douglas put it well:

“Even if algorithmic progress stalls out, and we just never figure out how to keep progress going — which I don’t think is the case, that hasn’t stalled out yet, it seems to be going great — the current suite of algorithms are sufficient to automate white collar work provided you have enough of the right kinds of data.”

The key phrase is the right kinds of data. The algorithms exist. The base models are good enough. The bottleneck is task-specific training data and the RL loops to learn from it. That’s specialisation. And specialisation, structurally, is where open-source wins.

The Timing Asymmetry

The conventional narrative is that the frontier labs will always be ahead. They have the capital, the talent, the data, and the compute. Open-source will lag behind by some number of months. And the Chinese labs who have been subsidising open-source development at enormous financial loss will eventually run out of money, at which point the gap widens and the game is over.

I think this narrative is wrong, but not for the reason most people think. I don’t think open-source needs to match frontier models on raw capability. It needs to match them on the intelligence ceiling, which is the fraction of economically valuable work that can actually be automated. And because most of that ceiling lives in the specialisation layer, open-source is already ahead in the metric that matters.

But grant the premise for a moment. Grant that open-source base models will always lag closed-source by some margin. Grant that the Chinese labs eventually stop subsidising. The question is: what will the intelligence ceiling be when this happens?

This is the timing asymmetry. The frontier labs have a capital-dependency clock. To train the next model and stay ahead, they need to raise billions from VCs who need returns. The only way to raise that capital is to convince investors that the margin between closed-source and open-source capability is worth paying for. That margin is their entire business model.

Open-source has a different clock. Every new base model release, every hardware generation, every inference optimisation expands the set of tasks that a specialised open-source model can automate profitably. And critically, this clock ticks along two independent axes. Model capability improves: GLM-5 is better than GLM-4.7, and the fine-tuning ceiling rises with it. And serving costs fall: GB200-class systems are resetting the economics of GPU compute, dragging token pricing downward and making it viable to serve models that would have been cost-prohibitive a year ago. SemiAnalysis[2] has quantified this dynamic extensively; each new hardware generation doesn’t just make existing deployments cheaper, it makes entirely new deployments economically rational for the first time (“You’ve made a classic blunder! You forgot to consider Jevon’s paradox!”).

So the frontier labs need the gap to stay wide enough to justify their capital requirements. Open-source just needs the ceiling to keep rising. These are very different pressures, and time favours the latter.

Why the Big Labs Can’t Just Pivot

The obvious counter-argument: if specialisation is where the value lives, why can’t OpenAI or Anthropic or Google just offer fine-tuning? They already have fine-tuning APIs. What stops them from capturing the specialisation layer too?

The answer is that capturing the specialisation layer requires a fundamentally different organisational shape. Training a single frontier model is a concentrated effort: one massive compute allocation, one research team, one training run, one model that you then sell to everyone. This is what the Big Labs are built to do, and they are extraordinarily good at it.

Specialisation is the opposite. It’s a thousand small efforts, each embedded in a different customer’s workflow, iterating on their data, learning their edge cases, building training loops around their specific failure modes. It requires domain expertise the lab doesn’t have, data the lab can’t access, and iteration cycles that happen at the customer’s pace, not the lab’s. You can’t do this from behind an API. You need open weights, custom training infrastructure, and people who understand both the ML and the domain deeply. In addition, we’ve talked at length here about why large models are hard to specialise at a general level.

This is the classic innovator’s dilemma. The Big Labs are structured to do one thing at enormous scale, and that thing is becoming less differentiated over time. For instance, Kimi K2.5, Minimax 2.5 and GLM-5 were all released in the last week, with Deepseek-V4 and Qwen-3.5 on the horizon. The specialisation game, in contrast, requires doing a thousand things at small scale, and each of those things creates compounding value that’s hard to replicate. OpenAI offering an API fine-tuning endpoint is to genuine specialisation what a hotel concierge recommending a restaurant is to actually knowing how to cook.

As Sarah Guo has pointed out, the economically efficient outcome is that a company should be willing to pay up to profit for the most model intelligence on a given task — because if you won’t, someone else will, and they’ll provide a better customer experience. This cuts against the “one model to rule them all” approach: if a specialised model delivers more capability on your specific task, and the open-source ecosystem makes it economically viable to build and serve that model, the frontier model is overpriced and underperforming for your use case. The rational buyer leaves.

Dario describes[3] the AI industry settling into a Cournot equilibrium[4] with a small number of firms with high barriers to entry and positive margins, like cloud. Maybe. But that equilibrium describes the pre-training layer, which is indeed expensive and concentrated. The specialisation layer has the opposite structure: low barriers to entry, thousands of players, value created through domain depth rather than compute scale. The frontier labs might sustain a comfortable oligopoly on base model training while the majority of economic value accrues to the specialisation layer, the way Intel was profitable making chips while the real wealth was captured by the companies building on top of them.

The RL Generalisation Counter

The strongest version of the case against open-source isn’t that the Big Labs will pivot to specialisation. It’s that they won’t need to.

The argument goes like this. RL scaling is showing the same log-linear improvements[5] that pre-training showed. As frontier labs train on a broader and broader set of RL tasks — first math, then code, then a wide variety of agentic work — the models will generalise. Just as GPT-2 generalised beyond its training distribution when trained on a broad enough internet scrape, RL-trained models will generalise beyond their specific training tasks when trained on a broad enough set of RL environments. At some point, the frontier model clears the automation-worthiness threshold for most enterprise tasks without any fine-tuning, and the specialisation gap closes from above.

I take this seriously. It’s probably the strongest argument that the frontier labs are going to win. But I think it’s wrong, for two reasons.

The first is empirical. Generalisation from broad training gets you remarkably far on tasks that look like the training distribution, which is why frontier models are so good at coding, at general knowledge tasks, at anything that resembles the internet. But enterprise work has a specific character that resists this kind of generalisation. The edge cases in an insurance claims workflow aren’t hard because they require more intelligence. They’re hard because they require knowledge of this specific insurer’s policy language, this specific state’s regulatory requirements, this specific client’s historical claims patterns. No amount of general RL training will teach a model these things, because they exist only in the client’s proprietary data. Generalisation gets you from 0 to 70. Specialisation gets you from 70 to 95. And the economic value lives disproportionately in that last 25, because that’s where “good enough to demo” becomes “good enough to deploy in production without a human in the loop.”

The second reason is that the RL generalisation argument actually supports the open-source thesis, even though it’s deployed against it. If RL scaling is log-linear and broadly applicable, if there’s no secret sauce, just scale and data and well-designed reward signals, then the same scaling laws apply to specialised fine-tuning. An open-source model fine-tuned with RL on domain-specific tasks benefits from exactly the same log-linear improvements that frontier labs see on their general RL training. The algorithms aren’t proprietary. The scaling behaviour isn’t proprietary. The only proprietary ingredient is the pre-training compute. And the argument of this piece is that, for the majority of economically valuable work, the specialisation layer on top of a good-enough base model is worth more than the marginal capability of the biggest base model. The frontier labs have announced the scaling laws, and those scaling laws work for everyone.

What About the Hard Stuff?

I want to be honest about where this argument doesn’t hold.

For genuinely hard tasks such as automating novel scientific research, the kind of work that requires sustained reasoning over massive context windows with real creativity, I think the frontier labs will win, at least for a while. These tasks require raw capability that currently only comes from models trained at enormous scale, and the specialisation layer doesn’t help as much when the task is fundamentally about general intelligence rather than domain expertise.

But many tasks that look like they require superhuman general intelligence actually require deep domain specialisation in disguise. The insurance claim that seems to need “reasoning” actually needs a model trained on 50,000 of that insurer’s claims that knows the edge cases. The clinical note that seems to need “medical knowledge” actually needs a model trained on that health system’s documentation standards and coding conventions. Leading AI application companies like OpenEvidence and Abridge have proven this. The compliance review that seems to need “legal reasoning” actually needs a model trained on that jurisdiction’s specific regulatory framework. Over the course of numerous customer engagements, we’ve found a surprising amount of what we call “hard” work is actually “specific” work, and specificity is what specialisation buys you.

The counter-counter, though, is about sequencing. The revenue from automating the 90% of currently valuable work (the insurance claims, the clinical notes, the compliance reviews, the customer support, the back-office workflows that make up the unglamorous bulk of the knowledge economy) funds the push toward the remaining 10%. Open-source companies that capture specialisation revenue can reinvest in training better base models, can contribute to open research, can keep the frontier within reach. This is how the open-source ecosystem sustains itself: not by matching the labs dollar for dollar on pre-training, but by generating enough value from the specialisation layer to stay in the game.

There’s a more sophisticated version of the objection, which is that the value created by very strong models, value from entirely new tasks that don’t exist today, will dwarf the value of automating current work. Maybe. But this is a sequencing argument too. New-task value only materialises after the infrastructure is built, the use cases are discovered, and the ecosystem matures. That takes time. And during that time, the specialisation layer is generating real revenue and funding the next generation of open models. The future is not a single moment; it’s a sequence of investments and returns, and open-source has a viable path through that sequence.

We Don’t Have to Wait Out AGI

So here is the full argument.

Open-source AI is the Michael Scott Paper Company. The conventional wisdom says it can’t last — the frontier labs have more money, more talent, more compute, and open-source will always trail behind. And maybe it will, on raw pre-training benchmarks. But that’s not the metric that matters. The metric that matters is the intelligence ceiling: how much economically valuable work can you actually automate? And most of that ceiling lives in the specialisation layer, which open-source owns structurally.

The frontier labs are Dunder Mifflin. They need their moat to hold. They need the gap between closed-source and open-source to remain wide enough to justify billions in capital raises and trillion-dollar valuations. Every quarter that the intelligence ceiling rises — through better open models, through cheaper serving, through better fine-tuning techniques — that moat narrows. The RL scaling laws that the frontier labs discovered work for everyone. The algorithms aren’t secret. The scaling is log-linear and domain-general. The only question is what data you feed the machine, and for most economically valuable work, the most valuable data is your own.

And we don’t have to wait forever. We don’t have to wait until AGI, or until open-source matches the frontier on some abstract benchmark. We just have to wait until the intelligence ceiling from fine-tuning open-source models covers something like 90% of economically valuable work. At that point, the specialisation advantage creates trillions of dollars in value for the open-source ecosystem, and the Big Labs have no choice but to grapple with a world where their most profitable layer, i.e. serving general-purpose intelligence at premium margins, is being eaten from below. I think we’re closer to this future of an ecosystem of specialised models built on open-source than most people realise. And the ceiling is rising every quarter.

So I don’t think open-source needs to wait out the frontier labs. I think it just has to wait out the gap.

Modern LLMs are amnesiacs. Once their context window fills, they start over with nothing but a summary to guide them. This isn’t a bug we can engineer away, but rather a consequence of how memory works in transformers. As context grows, the KV cache grows linearly which leads it to rapidly consume more memory than all of the model’s weights. And in order to build systems that learn continuously, we need to move information from the token level into the weight level: from explicit storage into superposition.

This leads us to a conclusion that shapes the rest of this piece: continual learning is inseparable from specialization. The cost of remembering scales with the scope of what you’re trying to remember. Narrow the scope, and the problem becomes tractable. Keep it general, and the model is now fighting thermodynamics.

Two types of Specialization

Specialization takes two forms: task specialization and domain specialization. Most enterprise AI use cases today focus on the former — document summarization, data extraction, classification. Any task with a fixed prompt template and a well-defined output space.

We think continual learning for task specialization is essentially solved.

Companies already collect data continuously. Modern fine-tuning stacks make it trivial to set up a continuous fine-tuning pipeline, whether you’re doing RL or SFT. Furthermore, there’s little practical difference between ten fine-tunes of 100,000 examples each or one fine-tune of a million examples. If you have enough data, a single training run at the start often suffices. If the distribution shifts, you retrain. This is engineering, not research.

The interesting question is: what does continual learning unlock beyond incremental improvement on tasks models can already do? What new use cases become possible?

Consider the difference between a model that summarizes documents and a model that acts as an employee. Why can’t we just collect conversation traces between employee and manager and train on them?

Two factors make the employee problem fundamentally harder:

1. Coverage. For document summarization, you can define a priori a set of evals that cover the task. The output space is bounded. An employee might be expected to do a much wider range of tasks — tasks you can’t enumerate in advance.

2. Sample efficiency. A summarization model sees thousands of documents. It has abundant opportunity to learn from feedback. An employee might receive an instruction once and be expected to follow it forever. There’s no second chance to learn from a repeated example.

It’s this second class of problems — broad scope, sparse feedback, one-shot retention — that continual learning needs to solve. This is what we’re focused on in this piece.

The Fundamental Tension

The best framing of continual learning’s core difficulty comes from Beren Millidge.

Beren says that continual learning is fundamentally in tension with long-term memory. Memories don’t store information directly but rather store pointers to distributed neural representations. When you recall something, you’re reactivating a pattern across many neurons. But continual learning keeps updating those very representations. The paths your memories point to drift out of sync with where they’re supposed to lead.

This generalizes to any modular system. If two subsystems communicate via representations that both evolve independently, their shared language eventually breaks down. Module A learns to send signal X for “danger.” Module B learns that X means “danger.” Then A updates, and now it sends Y. But B is still listening for X, and the interface is corrupted.

The brain’s solution is to freeze the interface. After some critical period (the highly neuroplastic period of childhood), the communication channel between systems becomes fixed. Each subsystem can still learn internally — it builds encoders and decoders that map between its evolving representations and the frozen channel. This is why critical periods exist, because they establish stable interfaces so the rest of the pipeline doesn’t catastrophically break every time you learn something new.

But notice what this implies. If the interface is frozen, all subsequent learning happens relative to that frozen base. The base constrains what you can learn and how efficiently you can learn it. The more you’ve already committed to the base, the harder it becomes to add more. This is why adults struggle with languages that children acquire effortlessly, i.e. the phonological interface froze decades ago.

You might object: adults still learn for decades, as we remain generally intelligent and continual learning isn’t catastrophic for us.

But humans are far more specialized than LLMs. Our knowledge is patchy, spiky. We pick one narrow sub-area and learn it deeply. A surgeon doesn’t also know tax law and fluid dynamics and Mandarin. The brain solves continual learning by not attempting generality in the first place.

The Big Labs face a harder problem: take a model that already knows everything, then force it to learn more, without forgetting what it knew.

The cost of continual learning scales with generality.

A model that knows everything must maintain consistency across all its representations whenever it learns anything new. Every update risks breaking something unrelated. You could design a synchronization system that keeps stored knowledge aligned with drifting representations, but this eliminates the computational advantage of long-term memory, which is that you can largely leave it alone unless needed. The maintenance burden grows with what you’re storing.

A model that only knows insurance claims has no such problem, as it can update freely and there’s little else to interfere with, little else to keep synchronized. Once you establish your base representations, all new learning happens relative to them. The more you’ve crammed in, the harder it is to add more without interference, and the more expensive it becomes to maintain what you have.

This is Adam Smith’s specialization of labor applied to neural networks. Generality has overhead, and that overhead grows faster than linearly with scope. Specialization sidesteps the problem entirely. A medical coding model establishes interfaces optimized for medical concepts and improves within that bounded space, where catastrophic forgetting becomes a local problem rather than a global one, and synchronization costs stay manageable.

The intractability of continual learning in generalist models is the invisible hand pushing deployment toward narrow experts.

Some of the obvious and less obvious ideas in the literature

The continual learning literature is large and growing. Most of it is trying to escape the tension we just described—finding some trick that lets you update representations without corrupting the interface. Some approaches are clever. Some might even work for narrow cases. But we think none of them escape the fundamental tradeoff.

Here’s a tour of the ideas we find most interesting, and why we’re still skeptical.

At the moment, the state-of-the-art technique for dealing with finite context windows is to prompt a model with ‘summarize this conversation’. Everyone knows that this is suboptimal, but coming up with a better solution is surprisingly difficult. However, one interesting observation from recent years is that many post-training innovations have originated with a capability that was initially elicited through prompting. The best example of this is reasoning models; before we had o1, we had ‘Think step by step’.

What then is the equivalent generalization of ‘summarize this conversation’? One approach from last year is the idea of constructing ‘Cartridges’. The idea here is to construct a KV cache that efficiently compresses prior knowledge into a dense learned representation. This is done by prompting the model to ‘self-study’ the prior knowledge, and using back propagation to update the KV cache instead of the model weights.

In fact, this idea is quite old. In 2019, FAIR published Augmenting Self-attention with Persistent Memory. This paper highlights the following point: a trained KV cache is actually very mathematically similar to a multi-layer perceptron (MLP) layer. Indeed, taking a dot product with a set of keys is equivalent to multiplying with an up-projection matrix. The softmax is then a suitable activation function, and a linear combination of value vectors is mathematically equivalent to multiplying with a down projection matrix. For this reason, the authors of this paper propose doing away with the MLP entirely: we simply make use of two types of KV vectors – static and dynamic. This makes the problem of continual learning even more tantalizing: it makes you wonder how we might move dynamic KV vectors into the static KV cache.

Of course, it would be remiss of us not to mention state space models in this context. In 2025, models like Qwen-3-Next, Kimi-Linear, Minimax-M1 and Nemotron 3 Nano demonstrated that hybridizing state-space models with interleaved full attention layers could lead to competitive long context performance. However, these full attention layers are still critical, so the context window is still growing indefinitely.

Without a non-linearity between the keys and the values, we are skeptical that pure state space models like delta-net and mamba-2 will have the expressivity needed to compete with full attention. In a KV cache, the exponential decay of softmax allows you to distinguish between adjacent keys with high fidelity; without it, interference is inevitable. Indeed, the 128 by 128 state matrices which dominate the state-space status quo can only accommodate 128 rank 1 updates before perfect recall becomes impossible. As such, we see such state space models as essentially a smoothed version of sliding window attention. Certainly, this is an interesting innovation, but not a silver bullet for infinite context.

Another idea is from a relatively old paper, Mixture of a Million Experts, which pushes MoE to its logical extreme: a million experts, each a single neuron.¹ When we read this paper, our first thought was “what if you had a mixture of a trillion experts, and continual learning was just updating the router for new tasks?” This is because, with a trillion experts, you basically have the primitives for all function composition, and your router can just expressively map to these. But this also felt a bit overkill and moving away from the representation-based world of Beren. It felt like we should still be routing to and updating meaningful latents, in some form, rather than very atomic functional primitives.

Sparse memory fine tuning makes this idea more explicit in an update rule. Given a memory layer (architecturally similar to mixture of a million experts), they ask: which slots should you actually update when learning something new? Their answer is TF-IDF scoring (a simple term-importance heuristic): rank memory indices by how specific they are to the new input versus a background corpus. This identifies indices that are activated by this knowledge but not by everything else, avoiding the general-purpose slots that would cause interference. This is Beren’s framework quite explicitly: a frozen addressing mechanism plus a simple heuristic (update what’s specific, leave what’s general) that sidesteps the need to learn how to learn.

There was a lot of hype around the (very lengthy) paper from DeepMind on nested learning, which starts with the observation that architecture and optimization aren’t separate things. Rather, they’re the same concept operating at different timescales. A transformer’s attention mechanism is associative memory updating every forward pass. The feedforward layers are associative memory updating at training time. Backprop itself is associative memory, mapping inputs to their prediction errors. Once you see this, the whole model becomes a hierarchy of nested optimization problems, each with its own update frequency. This leads naturally to what they call “continuum memory systems”: instead of a binary split between short-term (attention) and long-term (weights), you get a spectrum of memory modules updating at different rates. Their proof of concept architecture, Hope, implements this as a self-modifying recurrent network that can optimize its own memory through self-reference. From the Beren framing, this is another way of establishing stable interfaces: if different components update at different frequencies, the slow-updating ones become the frozen channel through which the fast-updating ones communicate. You get modularity through temporal separation of learning rates. (This builds on Titans; see below.)

We don’t know enough about this to make an informed comment, but it feels a bit flawed. We think that it introduces way too many manual and qualitative design choices that are difficult to justify exactly why you chose them (number of time scales, what the frequency of those time scales should be, etc.). To us, it seems much cleaner to have the minimal set of discrete components of the system to emulate our meta-learning algorithm (a long-term memory like parameter weights, a short-term memory like the KV cache, and a way to inject the relevant bits of short-term memory into long-term memory when we need to). But we could be wrong. Maybe this is a solution.

Speaking of “when we need to”, a lot of people think surprise-based learning is the way to go.² One recent paper we read on this was entropy-adaptive fine-tuning (EAFT). EAFT identifies confident conflicts, which are tokens where the model has low entropy (strong prior) but low probability (the training label contradicts that prior). These generate massive gradients that overwrite existing representations, causing forgetting. Their fix is to weight the loss by normalised entropy, suppressing gradients when the model is confident. This dramatically reduces catastrophic forgetting.

But I’m skeptical. The whole point of learning is sometimes you need to override a confident prior. The president of the United States changed from Obama to Trump. A fact you were confident about turned out to be wrong. EAFT treats all confident conflicts as damage to be avoided, but some of them are exactly the updates you want. The real problem is that standard architectures don’t have a clean separation between “what I know” and “how I reason”, so updating one fact corrupts everything else. Suppressing those updates just means you never learn the new facts.

And then there’s test-time training. One interesting paper we saw on this was Titans. This takes a memory-centric view of sequence modeling. They say attention (due to its limited context but accurate dependency modeling) acts as short-term memory, while a neural network that learns to compress information into its weights can act as long-term memory. Their neural memory module is essentially a meta-model that learns how to memorize at test time, using gradient descent on an associative memory loss. The clever bit is their surprise metric: an event that violates expectations (high gradient) is more memorable, but they decompose this into past surprise (momentum) and momentary surprise (current gradient), which prevents the model from missing important information after a big surprising moment. They also add a forgetting mechanism via weight decay, which they show is actually a generalisation of the gating in Mamba and friends. This feels like (1) a more principled surprise-based approach than EAFT, but simultaneously (2) re-deriving RNNs/LSTMs and the like, so we’re not sure how we feel about it.

A related and recent paper takes a simpler approach: just keep training a standard transformer with sliding-window attention at test-time via next-token prediction on the context it’s reading. The model compresses context into its weights rather than storing every key-value pair explicitly. To make this work well, they use meta-learning at training time to prepare the model’s initialization for test-time training, so the outer loop optimizes for “how good will this model be after it’s been updated on the test context”. The tradeoff is worse needle in a haystack retrieval, which makes sense, as the whole point is lossy compression, not lossless recall. I actually don’t mind this, as it feels a lot closer to what humans are doing. At the same time, I think we still need a lossless form of short-term memory that we can more efficiently swap to, without defaulting to test-time training.

There’s also a bunch of other interesting related papers, whether it’s about extending the effective context window (e.g. lightning attention or landmark tokens)³, or other architecture changes like Memformer, which are worth reading.

The final one goes back to CARTRIDGES, which feels very spiritually close to sparse memory fine tuning. Again, if you treat your bank of CARTRIDGES as an associative memory, and the pre-trained transformer backbone establishes the interface, and you figure out how to retrieve and update relevant CARTRIDGES continuously, then you’ve got Beren’s system. But maybe you want to be able to slowly add CARTRIDGES back into the weights, which is a whole other question entirely.

The common thread

Every approach here is trying to answer the same question: how do you update a neural network without breaking what it already knows?

The answers cluster into two families. Either you sparsify the update (million experts, TF-IDF routing, low-rank adapters), or you separate timescales (nested learning, test-time training, CARTRIDGES as a separate memory system). Both are ways of limiting interference. Neither escapes the fundamental tension.

Sparsification works when knowledge is separable. But the whole power of distributed representations is that they’re not separable: features are reused across contexts, and that reuse is what gives you generalization. The more you sparsify, the more you’re fighting the architecture.

Temporal separation works when you can afford to freeze something. But what you freeze becomes load-bearing. If you freeze too early, you’re stuck with bad representations. If you freeze too late, you’ve already caused interference. And the thing you froze can never improve.

None of this means these techniques are useless. For narrow domains with clean separation, sparse updates might be enough. For applications where you can tolerate lossy compression, test-time training might work. But for the Big Lab dream, a general model that learns everything, forever, without forgetting anything, we don’t see an escape hatch here.

Why this is probably the wrong framing

Most of the ideas in the previous section share an assumption: that the right way to solve continual learning is to learn how to learn. To us, it is fairly clear that humans don’t learn their meta-learning algorithm. Instead, nature endows us with a relatively fixed, heuristic-based strategy for absorbing information: we learn our encoding during childhood, then use that mostly-frozen encoding to update our associative memory for the rest of our lives. If we also had to learn how to learn, we’d be dead long before we learned anything useful. (There’s also evidence for this in animals. Zebras walk within minutes of birth. They’re not figuring out locomotion from first principles.)

As Rich Sutton points out, evolution solved the meta-learning problem over tens of thousands of years. The important takeaway is that we don’t have to. If we know what the meta-learning strategy is supposed to do, we don’t need gradients on gradients. We don’t have to bitter-lesson our way to the right algorithm. If we can design a system that (1) establishes an interface of representations plus encoders and decoders that read from and write to memory, and (2) slowly adds to that memory without disrupting the frozen interface, then we can just throw backprop at the subproblems. Learning representations and learning encoders/decoders are things we already know how to do. Retrieval might be as simple as vector search. Updates might be as simple as TF-IDFing which slots to touch, just like sparse memory finetuning.

And whilst we don’t know enough about the human brain to be able to emulate exactly what this meta-learning strategy might be, I imagine it could be akin to the airplane emulating the flight of birds. We know what we need to do and what the end goal is (keeping a body in the air for an extended period of time), so we’ll probably end up just figuring out what the end mechanism we need for this system, and writing some code/designing an architecture+algorithm to do it.

The Cambrian Zoo

So what does the future actually look like?

We think you’ll see a proliferation of specialized models (thousands, eventually millions) each optimised for a narrow domain and continuously improving within it. Medical models that know medicine. Legal models that know law. Models fine-tuned on individual users’ preferences, updating constantly.

These models won’t share weights. They’ll share APIs. The “general intelligence” emerges from composition rather than from cramming everything into one network. A routing layer (itself probably a model) will decide which specialist to invoke. The specialists can be updated, swapped out, improved independently. No global synchronization is required.

This is the Cambrian explosion that followed the Ediacaran period, a riot of specialized forms filling every niche. The foundation model era was Ediacara: a few general-purpose architectures dominating because nothing else had evolved yet. What comes next is adaptive radiation.

We don’t think the Big Labs will stop trying to build god-models. There’s too much momentum, too much narrative investment in AGI as a single artifact. But we think the actual deployed systems that matter, i.e. the ones doing useful work, making money, and improving over time, will be specialists. The monoliths will be impressive demos. The zoo will be the product.

The infrastructure for a world of specialized models looks different from the infrastructure for a world of monoliths. You need to serve thousands of models efficiently, not one model at massive scale. You need to route between them intelligently. You need to update them continuously without downtime. You need to version them, A/B test them, roll them back when something breaks.

This is what we’re building toward at Baseten.

Appendix: RL vs SFT

No continual learning discussion would be complete without a discussion of RL vs SFT. The catastrophic forgetting literature has long treated forgetting as an architectural problem i.e. something to be solved with replay buffers, elastic weight consolidation, or careful regularisation. But recent work suggests the training objective itself might be the culprit. Specifically: SFT and RL have fundamentally different relationships to the model’s existing knowledge, and this difference is best understood through the lens of KL divergence.

SFT minimizes negative log-likelihood over a dataset, which is equivalent (up to a constant) to minimizing the forward KL divergence between the data distribution and the model. This is a mode-covering objective. The model is heavily penalized for assigning low probability to any completion found in the training data; the loss increases exponentially as probability approaches zero. To avoid this penalty, the model must spread its probability mass across all modes in the dataset. The practical consequence is that when you fine-tune on new data, the model aggressively shifts probability toward that data, often at the expense of previously learned modes that aren’t represented in the current batch.

RL, by contrast, maximizes rewards on completions sampled from the model’s own policy (with KL regularization to the reference model). This is equivalent to minimizing the reverse KL divergence (a mode-seeking objective). The model emphasizes high-reward outputs even at the cost of ignoring some output modes entirely. Importantly, assigning near-zero probability to some completion simply prevents it from being sampled; there’s no exponential penalty forcing the model to cover that mode. The model can sharpen around what works without being dragged toward what the data says it should be doing.

This distinction turns out to be predictive of forgetting behavior. A recent group studying multimodal continual post-training, found that sequential SFT on seven tasks produced significant forgetting while reinforcement fine-tuning on the same sequence preserved prior-task performance almost entirely, approaching the upper bound of multi-task training without replay. Other papers report that RL achieves comparable target-task gains with substantially less degradation on non-target tasks. Another paper proposes what they call “RL’s Razor”: among all parameter configurations that solve a new task, online RL tends to converge to the one closest (in KL) to the original model. SFT, by contrast, can converge to solutions arbitrarily far from the base model depending on the training labels.

The common thread is on-policy learning. Because RL samples from the model’s current policy, it trains on completions the model already assigns reasonable probability to. This implicitly preserves prior modes as you’re reinforcing behaviors the model can already produce, not overwriting them with behaviors from an external distribution. The mode-seeking property of reverse KL means the model doesn’t need to spread probability mass to match some target distribution; it can simply sharpen around what’s working while leaving the rest of its knowledge largely untouched.

This suggests an important reframe: the problem with SFT isn’t just that it’s off-policy in the RL sense, but that it’s mode-covering in a way that actively redistributes probability mass away from prior knowledge. Every batch of new data pulls the model toward full coverage of that batch’s modes, creating interference with everything else. There’s also some stuff which shows low-rank updates are important for less forgetting, with everything from LoRAs to RL.

The on-policy distillation work from Thinking Machines makes this concrete. They show that even training on a model’s own samples via SFT degrades performance. Any finite batch exhibits distributional drift from the true policy, and the mode-covering objective amplifies this into progressive forgetting. On-policy distillation sidesteps this because the objective (reverse KL to a fixed teacher) is mode-seeking: the student converges on the teacher’s behavior without the self-reinforcing drift of off-policy training. This is why they can recover instruction-following capability after mid-training on domain data, and why distillation is emerging as a tool for continual learning more broadly.

San Francisco, without you in it, is a remarkably flawed and empty place. But having moved here last year, I want to try and say what I think it does have, and what it’s done for me.

I think the remarkable thing about holding all these colours is that San Francisco holds fragments of wherever you’ve come from. It is a palimpsest city. A stodgy blanket wrapping the disjointed patches of all the lives that amalgamated in this city; a city that, perhaps unfairly, is often viewed as a sterilised condensation of homogenised AI tech bros. When I sit and watch the NFL at McGarry’s in North Beach sipping a Guinness, San Francisco is Jack Duggans in Bathurst (an institution my brother has been banned from on multiple occasions). When I squint and look at Point Diablo and its rocky silhouette in twilight shivers, it’s County Mayo in Ireland on a rare cloudless evening. When I wake up in the morning on my side and see only the blue of the ocean out my window, and maybe a couple of drifting sailboats, it’s my Yaya’s house in Palm Beach.

I think most people who uproot their lives and move cities do so because they’re looking for a way to remove themselves from the missing parts of wherever they came from. San Francisco, I’ve realised, is only that on the surface. Really, its beauty is that people bring these other places with them, in a way I don’t feel in New York or London or even back home in Australia. This place breathes with the tide of everything that washes in.

But the tide goes both ways. People come to San Francisco, which means people leave somewhere in order to come. And leaving means goodbyes. To places. To people. And it’s so sad saying goodbye over and over. Sometimes I stand at the end of a goodbye and I can’t move.

I think about this sometimes. The specific confluence of yeses that put me in that room on that night. The flight I nearly missed. The city I nearly didn’t move to. The invitation I nearly ignored because I was tired. Or worse, the invitation I accepted, and the person I didn’t meet because I wasn’t in Aisle 8 of Woolworths at 7:30pm on a Saturday. They’re out there, those other versions, living their lives. They don’t miss this person because they never met them. They don’t know there’s anything to miss.

That’s the part that gets me. Not the loss; I can handle loss. It’s the recognition that the loss was never guaranteed. That I could have been one of the ones who never had it. That most versions of me, probably, are exactly that: people who never had it and never knew to grieve it.

But I had it.

So when I say a goodbye means something, I don’t mean it hurts. I mean I can feel the weight of all those other lives pressed up against this one. All those empty rooms where I never met them. And this room, about to pack my bags for San Francisco, where I did.

I don’t know what calculus put me here instead of there. I don’t know if I’m the lucky one or if luck has nothing to do with it. I don’t know what I’ve lost by being in this timeline instead of another—what people I’d have loved, what versions of myself I’d have become. That uncertainty never closes.

But in this life, the one I actually got, I met this particular person I’m now saying goodbye to. And the gratitude isn’t what comes after the grief. It’s underneath it. It’s what the grief is made of.

I don’t need it to make sense. Sometimes (not always, but sometimes) I don’t even need the feeling to justify itself. It’s enough to stand here, holding what I got, knowing most of me never got it at all.

Sometimes the gratitude doesn’t require consumption. The meaning is not in the feeling itself, but in the cosmic accident that I get to be the one who feels it.

San Francisco forced my hand in grappling with this. People come here searching for gold, and coming means leaving, and leaving means learning what it costs to have had something worth leaving. But just as leaving induces goodbyes, arrival induces the possibility of return. To paraphrase Cavafy:

San Francisco gave you the marvellous journey.
Without her you wouldn’t have set out.
She has nothing left to give you now.

And if you find her poor, San Francisco won’t have fooled you.
Wise as you will have become, so full of experience,
you’ll have understood by then what these Ithakas mean.

Better if the journey lasts for years, so you’re old by the time you leave the island, wealthy with all you’ve gained along the way, not expecting San Francisco to make you rich. And you will one day go home, back to Jack Duggans, and it will not be the same Guinness, nor will you be the same person. But the place will remember you, and you will remember San Francisco, and you will be grateful.

“Claude’s constitution is a detailed description of Anthropic’s intentions for Claude’s values and behavior. It plays a crucial role in our training process, and its content directly shapes Claude’s behavior. It’s also the final authority on our vision for Claude, and our aim is for all our other guidance and training to be consistent with it.”

This implies that Claude is trained on its Constitution. But if you know much about language models, it probably implies something deeper. When you train a language model on a document, it doesn’t simply memorise that document. It has to compress the likelihood of these tokens, one after another, into its weights; a process quite different from the lossless, short-term memory of the KV cache. And even if Anthropic included the Constitution in the training data tens of thousands of times, even if Claude had memorised the entire document verbatim, it wouldn’t necessarily do anything differently.

A good example[2] of this is a quirk of pretraining discovered early on: even if “Socrates was Plato’s student” appeared in the training data, the model would fail to answer “Who was the teacher of Socrates?” This held even when the model could regurgitate the definitions of “teacher” and “student” perfectly well. What was the failure mode? Well, minimising the negative log likelihood of the next token simply did not produce a holistic latent representation of the concepts required. The model couldn’t answer that Plato was Socrates’ teacher because it hadn’t built the latent bridge between “X being Y’s teacher” and “Y being X’s student.”

I think this points toward something fundamental about language models. In order to learn to approximate latent knowledge, you need such a vast amount of data that you’ve encountered all possible permutations. In the example above, we would have needed many instances of “X is Y’s teacher” and “Y is X’s student” appearing together in order to learn the pattern “A being B’s student implies B is the teacher of A,” and to internalise the different surface forms that could express that relation.

Humans probably do something quite different, by having latent encodings of these concepts and the right latent bridges to link them together. We are not restricted to subword tokens as our only means of thought—though some might push back that the residual stream means language models aren’t either. We think in pictures and hierarchies and fuzzy feelings and latent concepts.

Why outline all of this? Because I think it offers a fundamental clue about language models, and thus a fundamental clue about the future of model shaping. We would all agree that Claude does exhibit a strong internalisation of its values and behaviours. And as established above, you don’t develop strong internalisations from reading data repeatedly if it’s presented as-is. So what’s missing?

I believe Anthropic has already told us what the missing ingredient is, and they told us back in 2021. Anthropic has long been doing Constitutional Alignment[3], whereby they use some Constitution to improve the model’s answers, then perform supervised fine-tuning on these improved answers (alongside preference-based RL). This suggests that by continuously having some guided feedback process arising from a core “latent” (the latent in this case being the Constitution) you can implicitly, through pattern matching and the law of large numbers, internalise that latent to some degree. This was the original constitutional alignment, and Anthropic is probably still using some version of it today.

But I think there’s a more explicit approach to learning these latents. CARTRIDGES[4] introduces a self-study procedure where, instead of naively conditioning on a document, the model generates synthetic Q&A pairs and dialogues about the content, essentially quizzing itself from multiple angles. This is then used as training signal to distill the knowledge into a compact cache.

The relevance to constitutional internalisation should be clear: self-study forces the model to encounter the same underlying concepts through many linguistic permutations. If Constitutional AI implicitly creates these permutations via iterative refinement and feedback, CARTRIDGES makes the mechanism explicit. You manufacture the diverse views needed for latent compression. The model doesn’t just read “X is Y’s student” once; it asks “who taught X?”, “what was Y’s role?”, “list X’s mentors,” until the relational structure is genuinely encoded.

So I imagine Anthropic is doing something similar. They have certain behaviours they want Claude to exhibit, explicitly spelled out in the Constitution, and self-study is used to embed these behaviours into the model. It’s clear that Anthropic does this remarkably well; there have been many comments about how Claude feels like it has a “soul”[5], thanks in large part to Amanda Askell and her work on personality shaping, in a way that distinguishes it from other LLMs. This also has implications for capability enhancement. I could imagine some form of self-study working for improving humanities-based tasks like essay writing, philosophy, and deep thought.

A final observation: my work at Parsed, and now at Baseten, on finetuning and post-training specialisation has, in a funny way, converged toward this same insight. Initially, we developed iSFT[6] as a way to take a Constitution of sorts (our evals) and teach a model to improve at those evals by fine-tuning it on data refined with the evals as guide, similar to Constitutional Alignment. We then developed RGT[7] as a way to make the “latents” we were trying to learn more explicit, rather than hoping the model would learn them through imitating shadows on Plato’s cave of perfect dances. Our next direction is making the model itself generate its own critiques, evaluations, and reformulations, i.e. its own self-study. This will make everything much more on-policy, which we speculate can only be a good thing.

So I think the takeaway isn’t in the Constitution Anthropic released, but in the very fact they acknowledged its existence. Because it tells us they are still doing some form of constitutional alignment, which tells us it remains worth doing in Anthropic’s highly optimised training setup, which tells us that self-study, to a certain extent, just works.

I kind of realised how each of them (although admittedly not Hesse, as this was proffered by the other participant in the conversation, and I’m not sure I’ve actually read enough Hesse to comment) by different paths through different darknesses, arrives at essentially the same clearing: Look, we cannot reason our way out of the fundamental absurdity of being thrown into existence. We didn’t discover, backing out from first principles, the algorithm for how to live. So instead, you circle the question like a sculptor circling marble, you attack the space around it with all the resolve you can muster, and you learn (if you’re lucky / brave enough) to make of that very absurdity a kind of home.

This conversation, dissolving as it did into the larger dissolution of surf and mist, made me think about the greatest work I’ve encountered that also holds this terrible question without answering it. Or rather, that makes of the question itself a living thing. And I don’t know why I’m being coy about it because you’ve read the title: it’s Anna Karenina.

Here’s the thing about Tolstoy. He doesn’t give you philosophy’s abstract comfort of accepting absurdity. Instead he gives you human bodies moving through drawing rooms and train stations, trying and failing and trying again to solve the unsolvable equation of how to be both authentic and connected. Both free and loved. The novel becomes a kind of proof by exhaustion, not that there’s an answer, but that the question penetrates every gesture of our lives, from the way we butter our bread to the way we throw ourselves under trains.

“All happy families are alike; each unhappy family is unhappy in its own way.” The opening line has become so familiar we risk overlooking what it’s actually saying: that happiness is singular while suffering is infinite. But as the novel unfolds, you discover something worse. Even happiness contains the seeds of its own particular anguish. Tolstoy’s method isn’t to tell us that existence is absurd but to show us characters living inside that absurdity, each attempting their own desperate geometry around the unanswerable question.

I should confess something here. I first read this book because a girl I was dating in undergrad said she loved it, and I wanted to be able to talk about it with her. Classic. I think I understood maybe 40% of what was happening, emotionally speaking. The Russian names blurred together. But something stuck. And then I read it again this year, in a completely different life, and it was a completely different book waiting for me.

I don’t know what this essay is, by the way. I run an ML startup. I spend my days thinking about loss functions and evaluation frameworks. Writing literary criticism about nineteenth-century Russian novels is not in my job description. But I like writing, and I think about this book a lot, and I’m going to do it anyway. Consider this my own small refusal to stand in the corner.

Here’s what those philosophers on the beach understood that Tolstoy also knew: the alternative to engaging with life’s unanswerability—choosing nihilism, suicide, protective cynicism—is itself just another arbitrary answer to the question. And an aesthetically impoverished one at that.

It’s like that meme of someone standing in the corner at a party, arms crossed, thinking “They don’t know that all this partying and socialising is meaningless.” Meanwhile everyone else could just as easily think: “He doesn’t know that his withdrawal is just as arbitrary as our engagement, and far less beautiful.”

Camus explicitly rejects suicide as philosophical cowardice. Nietzsche scorns the last men who have given up on greatness. Kierkegaard mocks those who think they’ve found easy answers in either faith or doubt. I’m going to be honest: I’ve read The Myth of Sisyphus properly, most of Thus Spoke Zarathustra, and like forty pages of Fear and Trembling before getting distracted. So take my synthesis with appropriate salt. But they do seem to be blind men touching different parts of the same elephant—Camus more egalitarian in his rebellion, Nietzsche more aristocratic in his will to power, Kierkegaard more indirect in his faithful leap. All insisting on the same thing: keep touching, keep groping in the dark, keep attacking the space around the question even though you’ll never answer it.

The act of engagement itself becomes the only non-arbitrary response to arbitrariness.

I don’t know if I fully believe that. But I believe it enough to keep going.

The conventional reading positions Anna and Levin as opposing answers to the question. Passion versus duty. Destruction versus creation. The city versus the land. But this binary collapses pretty quickly if you actually pay attention. Anna and Levin aren’t opposites. They’re variations on the same theme. Both are seekers circling the same impossible marble. Both cannot accept the given world. Both experience moments of suicidal despair. Both glimpse something beyond the veil of ordinary existence that renders that existence simultaneously precious and unbearable.

The difference isn’t in their choices. It’s not even really in their morality. It’s in what they do when they realise there’s no right answer. Levin keeps attacking the space despite his doubts. Anna, tragically, stops.

Consider Anna’s first appearance at the Moscow railway station, where she successfully reconciles Dolly and Stiva after his infidelity. Here’s Anna at her most vital: a force of restoration, mending the very kind of breach she’ll later create. Yet even in this moment of social triumph, death announces itself. A worker crushed beneath the train. “It’s a bad omen,” Anna says. We tend to read this as foreshadowing her own death. But the omen runs deeper.

The very rails that connect us, the structures of marriage, family, society that make civilisation possible, are also the machinery that crushes us. We need these structures to live, and they’re what destroys us. The train that brings Anna to Moscow to heal one marriage will carry her into the affair that destroys her own. The rails are the question itself.

I think about this sometimes when I’m on BART, which is perhaps too on-the-nose.

The novel’s architecture reinforces the paradox through its parallel plots. While Anna descends into jealous obsession with Vronsky, Levin ascends toward spiritual revelation through marriage to Kitty. But Tolstoy subverts your expectation of moral judgment. Levin’s happiness isn’t a reward for virtue any more than Anna’s destruction is punishment for sin.

This is important, and I think a lot of readers miss it.

Levin succeeds because he maintains his commitment to attacking the space around the question, even when he finds no answers. His religious revelation at the novel’s end isn’t a solution (he admits he’ll continue doubting) but a decision to continue engaging despite the doubt. Anna’s tragedy is that she believed a right answer existed, and when she couldn’t find it, when the contradictions became unbearable, she chose withdrawal. She becomes the nihilist in the corner of the party. Convinced that her non-engagement is somehow more honest than everyone else’s arbitrary dancing.

I’ve been Anna. More times than I’d like to admit. Not in the affair sense, but in the sense of believing there was a right configuration. That if I just thought hard enough, optimised well enough, I could find the answer that would make everything cohere. Love and freedom. Authenticity and belonging. And when it became clear that every path toward one thing led away from another, I’ve chosen the corner. Told myself that my clarity about the impossibility was somehow superior to everyone else’s messy attempts, where in the end it wasn’t, it was just quieter.

Anna faces an impossible equation. To live authentically and to follow her heart toward Vronsky she must destroy not just her marriage but her motherhood. Not just her reputation but her very ability to exist in society. The novel’s cruelest insight is that Anna’s mistake isn’t loving Vronsky. It’s believing that love could be enough. That you could solve the absurd rather than simply inhabit it.

She exchanges one form of incompleteness for another. The sterile respectability of life with Karenin for the fevered isolation of life with Vronsky. Neither allows her to be whole. When she realises there’s no configuration that works, no answer that satisfies, she stops trying.

And this surrender, rather than her adultery, is her true tragedy.

This extends beyond romantic love. Tolstoy shows us character after character struggling to bridge the gap between self and other. Between inner truth and outer expression. Karenin can’t access his own emotions except through bureaucratic formulations. Vronsky can’t understand Anna’s anguish because he’s never had to choose between love and social existence; as a man, he can have both. Even Kitty and Levin, the novel’s happiest couple, experience moments of profound disconnection.

But some characters accept this incompleteness as the condition of being human and continue engaging. Others demand resolution and, finding none, choose destruction.

The novel suggests that isolation isn’t a failure of love but its very condition. We can only love what is separate from us, yet love seeks to eliminate that separation. This paradox reaches its apotheosis in Anna’s final hours, when her need for Vronsky’s love becomes so consuming that it destroys the very thing it seeks to preserve.

Her jealousy isn’t mere possessiveness. It’s an epistemological crisis. She can’t know Vronsky’s inner life. She can’t be certain of his love. And this uncertainty becomes unbearable precisely because she has sacrificed everything else for it. She’s attacked the space around the question with such violence that when she finds no answer, she has no ground left to stand on. It’s a kind of mode collapse. I think about this whenever I see someone online who’s made skepticism their entire personality. Or when I catch myself doing it.

Yet Tolstoy doesn’t counsel resignation. The novel’s most luminous moments come when characters briefly transcend their isolation through acts of spontaneous compassion or understanding.

Levin’s day mowing with the peasants, where he loses himself in physical rhythm and communal labor.

Anna’s morphine-induced insight into the hatred she and Vronsky share, which paradoxically allows her a moment of clarity and pity.

Karenin’s forgiveness of Anna at what he believes is her deathbed, when artificial social emotions give way to genuine spiritual feeling.

These moments can’t last. Karenin returns to his emotional rigidity. Levin can’t remain among the peasants. Anna’s clarity dissolves into renewed torment. But their transience doesn’t negate their truth. They suggest that meaning exists not in permanent states but in the ongoing engagement. Not in answers but in the quality of our questioning.

The sculpting itself, not the sculpture, is what matters.

This is what the philosophers were trying to tell us, but Tolstoy gives it flesh. Gives it weight. Gives it the specific texture of a scythe cutting through wet grass at dawn.

This is why the novel’s ending feels both necessary and insufficient. Levin’s realisation that good exists despite his inability to rationally prove it mirrors the novel’s own method—an accumulation of lived experience that yields wisdom without dogma.

“But my life now, my whole life… has a positive meaning of goodness, which I have the power to put into it.”

The qualification is everything. Not that life has meaning, but that we have the power to put meaning into it. Moment by moment. Despite everything.

Levin hasn’t answered the question. He’s simply decided to keep asking it. To stay at the party despite knowing it’s all arbitrary.

Here’s where Tolstoy transcends the philosophers. He doesn’t offer us another meta-answer about how to live with absurdity. He offers us something else entirely: the company of others who are also struggling with the question.

The novel doesn’t tell us how to attack the space. It shows us that everyone is already doing it. Badly and beautifully. Failing and trying again.

Where Camus gives us the myth of Sisyphus, where Nietzsche gives us the Übermensch, where Kierkegaard gives us the leap of faith, Tolstoy gives us the messy, contradictory, irreducible experience of other human beings trying and failing to solve the same unsolvable equation we face.

Philosophy tells you what to think about the question. Tolstoy shows you that everyone else is asking it too. In their own broken, beautiful ways. The novel becomes a companion.

The train that kills Anna is heading somewhere. Toward the modern world where such total destruction for love seems almost quaint. Where we’ve learned to compartmentalise our passions, to manage our rebellions, to live with what William James called “the habit of our losses.”

Yet the questions Tolstoy raises remain urgent because they’re not historical but existential. How do we live when meaning and happiness diverge? How do we love when love itself is what isolates us? How do we maintain faith in goodness when goodness provides no protection from suffering?

These are the same questions those philosophers were circling. But Tolstoy gives them flesh and blood and railway tickets.

I’m not a philosopher. I’m also not an English major, although in another life I would have liked to be. Walking away from that beach conversation, I realised something about why I keep returning to this book.

It’s not the answers. There aren’t any.

It’s the company. The proof that everyone else is also stumbling through the same fog. Attacking the same marble. Dancing at the same arbitrary party. Sometimes happy, sometimes miserable, sometimes giving up, sometimes pushing again.

And maybe this knowledge—not that there’s an answer, not even that we should keep looking for one, but simply that we’re all in this impossible position together—maybe this is the only wisdom that matters. If philosophy teaches us that we must imagine Sisyphus happy, Anna Karenina shows us Sisyphus in all his particular human detail. Sometimes happy, sometimes miserable, sometimes giving up, sometimes pushing again. Always achingly, specifically human.

The horizon line remains erased. The fog keeps doing what fog does, but we’re all in it.

The problem isn’t that they aren’t smart enough. It’s that you’re getting them to do the wrong job. Their job isn’t to reinvent frontier model training inside your startup. Their job is to help get you to product-market fit.

The prototyping role of MLEs

This doesn’t mean MLEs don’t play a crucial role. In fact, they’re often the best people to help you prototype in the early days.

They can wire up APIs, run small-scale experiments to get you to your first demos. This phase is all about speed, iteration, and exploring whether there’s a real problem-solution fit. MLEs shine here because they’re adaptable, quantitative, and can hack together a proof of concept quickly.

But once you’ve validated the problem and see early signs of product-market fit, the game changes. At that point, you need to shift from hacking together demos to running scalable, reliable post-training loops. And that’s not something you can, or should, expect your MLEs to own. That’s where Parsed comes in.

The intelligence mismatch

It’s a fallacy to equate the skills that make someone a good traditional MLE with those required to run an LLM-based engineering team. A PhD in neural tangent kernels, those Kaggle competition wins are orthogonal to what you actually need.

Even if you’re just calling closed-source models through APIs (before we even touch training and inference) LLMs demand a fundamentally different kind of intelligence. Understanding how to scaffold and prompt your way to performance isn’t about gradient-boosted models. It’s about something fuzzier, more qualitative.

Writing LLM-as-a-judge evaluation harnesses requires specific intuition that doesn’t map neatly onto traditional metrics of quantitative intelligence. (This is why companies like Cognition boasting about hiring IMO winners feels misguided to me). Solving International Math Olympiad problems is impressive, but it’s a completely different skillset from understanding, on an intuitive level, how LLMs process tokens. How the subtleties of English as a conditioning mechanism lead to those subtle, pernicious behaviours we all know and love so well.

The training trap

“But surely,” you might think, “training and ML Ops at scale—that’s where my traditional MLE shines?”

Not quite.

Training LLMs isn’t the neat train-test-split and hyperparameter search that Kaggle gurus cut their teeth on. A huge part of successfully training or finetuning a language model comes down to data curation. And this, again, requires that qualitative intelligence: the ability to deeply understand a problem and the requirements imposed by defining it in natural language rather than deterministic code.

When you’re finetuning on outputs from a bigger model and the distillation gap leaves you disappointed after training qwen-32b, where do you go? There’s a massive creative component to generating better data. And reinforcement learning is a whole different beast. How do you write evaluations that go beyond regexes and deterministic Python checks to actually serve as useful reward functions? How do you even measure whether your LLM-as-a-judge is measuring what you want, whether it’s aligned with domain experts, whether it’s well-defined enough for RL post-training?

LLM training is fundamentally harder than spinning up a model in Sklearn. Models are many orders of magnitude bigger. Training requires knowledge of parallelisation, sharding, and distributed systems in ways it never did before. Expecting your MLE to skill up on all this while simultaneously learning a fundamentally different paradigm of how to interact with and control LLMs is a massive ask. You can’t just lump an MLE into the category of “prompt engineer + hardware guru + distributed systems expert + deep transformer guru” when their last job involved training CNNs on OCT scans.

Your team’s scarce cycles should go into defining the problem — not solving the engineering puzzle of training frontier-scale models. Parsed’s role is to absorb that engineering burden and return production-ready models that reflect your definitions.

The social intelligence factor

Perhaps most critically, LLM engineering demands social and personable intelligence in ways traditional ML never did.

Tasks don’t exist in isolated boxes anymore. They’re not completely defined by the features and labels you train on. Most LLM tasks are icebergs, with massive underwater components of domain knowledge and implicit context only available to teams expert in your specific vertical.

LLM engineers need to be socially intelligent enough not just to grasp this domain context, but to engage productively with domain experts and end users. They need to hold complex, nuanced discussions with stakeholders and distill all that messy, implicit information into evaluations, generation prompts, and training data.

Parsed takes those definitions and runs the heavy machinery: post-training, evaluation harnessing, RL, infrastructure.

The frontier problem

Perhaps the most salient issue is that the frontier of LLM research moves so quickly that it’s impossible for engineers hired for specific vertical use cases to also position themselves on the rapidly expanding frontier.

No offence, but you’re not getting Big Lab-level engineering quality at your app layer company. There’s a reason the best talent goes to the big labs: They’re building AGI, which is sexy. They get to work on fundamental language model problems. They pay better. They don’t have to optimise a system for medical record data summarisation and JSON formatting.

To expect these MLEs to perform at a level where they can post-train the way big labs do, managing these systems at scale, is crazy. Everything is so new that the best commodity to hire for is raw intelligence and horsepower. Experience means less as things move faster. A PhD in convex optimisation doesn’t really give you any relevant experience in spinning up a GRPO run for the first time.

The case for specialisation

All of this strengthens the argument for specialisation and buying versus building.

We knew this when we started Parsed, and every interaction with teams in these verticals has only strengthened our conviction. The most self-aware companies understand this: They want Big Lab expertise packaged into models and tools their engineers can actually use productively. Sure, some are happy calling closed-source APIs forever. But anyone who wants a cheaper, faster, and most importantly, better model optimised for their specific task will eventually have to bite the bullet on post-training.

That’s not to say tools and frameworks won’t emerge that allow in-house engineering teams to actively participate in this process. Indeed, that’s literally what we’re building Parsed to be. We want to commoditise Big Lab talent so anyone can access it as a nicely packaged product.

But just putting a UI around a finetuning service isn’t the answer. Yes, writing finetuning code is difficult, and automating it solves that problem. But it’s only a tiny piece. The hard part about LLMs is that optimising them requires understanding everything end-to-end: from your generation prompt to your LLM-as-a-judge, through the complex mechanics of RL and SFT, and how to properly integrate all the feedback signals into a system that actually improves at what you want it to improve at.

Bottom line

If after all this, you still trust your MLEs to achieve what you want, then good for you. But you wouldn’t build your own server rack with a team of software engineers who only know Java and API development. So why would LLMs be any different?