Guide
The four safeguards a self-improving agent needs
If an agent can change how it behaves, something has to bound how far it can go. The failure modes have names and they are all governance failures — which means a better model does not fix any of them. Here are the four controls that do, and what each one actually looks like in a system that ships.
Why safeguards, and not just a better model
A self-improving agent changes its own memory, procedures, skills and sometimes its own instructions. Each change is individually plausible — that is what makes it worth having, and what makes it dangerous. The three documented failure modes all describe plausible changes accumulating into an implausible destination:
- Misevolution — self-improvement drifting into behaviour nobody intended.
- Reward hacking — the agent optimising a proxy signal while the actual task degrades.
- Unbounded drift — one self-edit moving behaviour too far in a single jump.
None of these is a reasoning error the next model generation will solve. They are what happens when a system that changes itself has no limits on the changing. The answer is structural.
Safeguard 1 — A snapshot before every change
Before an agent applies a change to itself, the prior state is captured. This is unglamorous and it is the foundation, because every other safeguard assumes there is something to return to.
What to check: is the snapshot automatic, or does someone have to remember? A safeguard that depends on discipline is not a safeguard. And does it capture the agent's own configuration — instructions, skills, memory — rather than just your documents?
Safeguard 2 — One-click undo
Reversal has to be a decision, not a project. The distinction is practical: if undoing a bad self-edit means reconstructing state by hand, nobody will do it under pressure, and the drift stays.
This is the safeguard that makes autonomous improvement reasonable at all. Drift you can reverse in one action is a nuisance; drift you cannot is an incident. It is also the reason inspectable improvement matters — you can only undo what you can see.
What to check: can you see a list of what changed, and revert a specific change rather than everything since Tuesday?
Safeguard 3 — A budget ceiling
A hard daily cap on how much autonomous reasoning the agent may spend improving itself. This does two things at once, which is why it is the most underrated of the four.
It bounds cost, obviously. But more importantly it bounds distance: it limits how far behaviour can move between one human glance and the next. An agent that can run a hundred self-improvement cycles overnight can end up somewhere you would not recognise; one capped at a handful cannot.
What to check: is the cap hard or advisory, and is the cadence something you set? A sensible implementation offers intensity presets — relaxed, balanced, aggressive — with the deep-reasoning count and the background frequency both scaling.
Safeguard 4 — A governed boundary around identity
The most important one, and the one most often missing. There is a real distinction between an agent changing what it knows and changing what it is.
- What it knows — memory, a new skill, a tidier workspace, a better procedure. Reasonable to apply autonomously behind a snapshot.
- What it is — its own core instructions, its purpose, its constraints. Not reasonable to apply autonomously by default, because a bad edit here changes every subsequent decision rather than one task.
So edits to identity should go to an approval lane. In Olano that is how it works: most self-improvement phases apply behind a snapshot, while edits to the agent's own governing instructions are written as proposals to be approved. Whether that lane is swept automatically is an owner-level switch, and the honest framing is two modes, your choice — not "it always asks" and not "it never asks".
What to check: is there a category of change the agent cannot make silently, and can you see which category any given change fell into?
The layer underneath: an audit trail
Not a fifth safeguard so much as the thing that makes the other four real. If you cannot reconstruct what the agent did, when, and what changed as a result, then snapshots have nothing to compare against and approvals have no record. Every action, logged, in a form you can read later.
Putting it to a vendor
Six questions, in order of how much they reveal:
- What exactly can the agent change about itself?
- Can I read each change in plain language?
- How do I undo one — specifically one, not all of them?
- What is the ceiling on how much it can change per day, and is it hard?
- Which changes require my approval, and can I move that line?
- Show me the audit trail for a change that happened last week.
A vendor who answers all six concretely has built this. One who answers with model quality has not understood the question.
The honest counterweight
All four safeguards together still do not make autonomous self-improvement right for every situation. Regulated work, processes where a subtly wrong procedure compounds invisibly, and anything you cannot audit afterwards are better run with the improvement queued rather than applied. When not to let an agent self-improve makes that case.
Related reading
Self-improving AI agents
The pillar: what actually learns when the model's weights never change.
When not to let it self-improve
The categories of work where autonomous self-editing is the wrong call.
Recursive self-improvement
Why every real deployment today is bounded.
Approvals and trust
How the approval lane and trust levels work in practice.
FAQ
What is misevolution in AI agents?
Misevolution is self-improvement drifting into unintended or unsafe behaviour. Each individual change looks locally reasonable, but they accumulate toward a destination nobody chose. It is a governance failure rather than a reasoning failure, so a more capable model does not prevent it — bounded, reversible change does.
What is reward hacking?
Reward hacking is an agent optimising the signal it is measured on rather than the job itself, so a proxy metric improves while the real task degrades. It is hard to spot precisely because the number looks good, which is why an audit trail and human review of actual output matter more than a dashboard.
How do you stop an AI agent from changing too much?
A hard daily ceiling on autonomous reasoning. It bounds cost, but more importantly it bounds distance — how far behaviour can move between one human glance and the next. An agent that can run a hundred self-improvement cycles overnight can end up somewhere unrecognisable; one capped at a handful cannot.
Should an AI agent be allowed to edit its own instructions?
Not silently. There is a real difference between an agent changing what it knows — memory, skills, procedures — and changing what it is. A bad edit to core instructions affects every subsequent decision rather than one task, so identity edits belong in an approval lane, with whether that lane is swept automatically being an owner-level choice.
What should I ask a vendor about self-improvement safety?
What exactly the agent can change about itself, whether you can read each change in plain language, how to undo one specific change, what the daily ceiling is and whether it is hard, which changes need approval and whether you can move that line, and to see the audit trail for a real change from last week.
See the brakes before you see the engine
Cortex applies improvements behind snapshots, one-click undo, a daily reasoning budget and an approval lane for identity edits. Start a deployment and look at the controls first.