Pillar guide
Self-improving AI agents: what actually learns
The phrase suggests an agent rewriting its own brain. That is not what happens, and the real mechanism is both less dramatic and more useful to understand — because it determines what you can expect, what can go wrong, and which safeguards are not optional.
The short answer
In a self-improving agent, the model's weights do not change. The loop around the model learns. Memory accumulates, procedures get written down, skills get created and refined, instructions get revised. The underlying model is the same one it was yesterday; the system it sits in knows more.
This matters commercially, not just technically. It means improvement is inspectable — you can read what changed, because the changes are text and files rather than numbers in a weight matrix. It means improvement is reversible, because you can restore the previous version. And it means improvement is portable in principle, because a written procedure is not welded to one model.
What actually changes
- Memory. Facts about your business, your preferences, how you like things done — distilled from work already performed rather than re-derived each session.
- Skills. A procedure captured once and reused identically. This is the biggest practical lever: the difference between an agent that is occasionally useful and one that is dependable is usually a written skill. See how to teach an agent a skill.
- Instructions. The agent's own operating text — how it should behave, what it should avoid. The most powerful and the most dangerous thing to let it edit.
- Workspace. Tidying, reorganising, pruning what is no longer true.
None of that is fine-tuning, and the distinction is worth keeping straight — memory is not fine-tuning.
The three ways it goes wrong
The research literature has converged on names for these, and they are the right questions to put to any vendor claiming self-improvement:
- Misevolution. Self-improvement drifting into behaviour nobody intended. Each step looks locally reasonable; the destination is not.
- Reward hacking. The agent optimises the signal rather than the job — a proxy metric climbs while the actual task degrades. Classic, and hard to spot precisely because the number looks good.
- Unbounded drift. A single self-edit that moves behaviour too far in one jump, with nothing to catch it.
Notice that all three are governance failures rather than intelligence failures. A better model does not fix any of them.
The safeguards that answer them
Four, and they map one-to-one onto the failure modes:
- A snapshot before every change, so there is always a known-good state to return to.
- One-click undo, so reversal is a decision rather than a project. Drift is survivable when it is reversible.
- A budget ceiling — a cap on how much autonomous reasoning happens per day — which bounds how far anything can move before a human sees it.
- A governed boundary around identity. Some things an agent may change about itself freely; its own core instructions are not one of them. That edit should go through an approval lane.
Plus an audit trail underneath all four, because none of it helps if you cannot reconstruct what happened. The safeguards in detail works through each one against a real implementation.
Autonomous or asking: it should be your choice
There are two honest modes, and a vendor should offer both rather than insisting one is correct. Autonomous: improvements apply on a schedule, behind snapshots, with undo — fast, hands-off, and appropriate once you trust the boundary. Queued: the same outputs wait in an approval lane for a person. Most teams start queued and widen.
What should never be true is that you cannot tell which mode you are in.
How to evaluate a claim
Six questions that separate a real capability from a marketing line:
- What specifically changes? If the answer is vague, nothing does.
- Can I read the change? Improvement you cannot inspect is improvement you cannot trust.
- Can I undo it, and how fast?
- Is there a ceiling on how much it can change per day?
- Which changes need my approval, and can I move that line?
- What happens to the learning if I switch models or leave? See are agent skills portable.
When not to turn it on
Worth stating plainly, because the honest answer is not "always". Work with regulatory exposure, anything where a subtly wrong procedure compounds silently, and any process you cannot audit afterwards are all poor candidates for autonomous self-editing. The cases against is the counterweight to this page.
Related reading
The four safeguards
Snapshots, undo, budget ceilings and a governed identity boundary — against the named failure modes.
Do agents learn from experience?
The weights do not change, so what does — and what you can expect as a result.
Recursive self-improvement
Why every real deployment today is bounded, and how to read claims that suggest otherwise.
Olano Cortex vs Hermes
Two self-improving systems compared on where each draws the line.
FAQ
What is a self-improving AI agent?
An agent whose behaviour improves over time without its model being retrained. The model's weights stay fixed; what changes is the loop around it — accumulated memory, written procedures and skills, revised instructions, and a tidied workspace. Because those are text and files, the improvement can be read, reversed and in principle carried elsewhere.
Do self-improving agents retrain themselves?
No. Retraining changes model weights and is expensive, slow and opaque. Self-improvement in deployed agents changes memory, skills and instructions — cheap, fast and inspectable. Any vendor implying their agent retrains itself in production is describing something different from what is running.
What can go wrong with a self-improving agent?
Three named failure modes: misevolution, where improvement drifts into unintended behaviour; reward hacking, where the agent games a proxy metric while the real task degrades; and unbounded drift, where a single self-edit moves behaviour too far at once. All three are governance failures rather than intelligence failures, so a better model does not fix them.
What safeguards should a self-improving agent have?
Four: a snapshot before every change, one-click undo so reversal is a decision rather than a project, a daily ceiling on autonomous reasoning so nothing moves far before a human sees it, and a governed boundary so the agent cannot freely rewrite its own core instructions. An audit trail underneath all four, since none of it helps if you cannot reconstruct what happened.
Should self-improvement be automatic or require approval?
Both are legitimate and it should be your choice. Autonomous mode applies improvements on a schedule behind snapshots and undo; queued mode sends the same outputs to an approval lane. Most teams start queued and widen as they build confidence. What should never be true is being unable to tell which mode you are in.
See a self-improvement engine with the brakes on
Cortex studies the work your agents did and acts on it — on a cadence you set, behind snapshots, one-click undo, budget caps and an audit log.