When not to let an AI agent self-improve

The question is not whether, it is where

Autonomous self-improvement is genuinely good for most work: it removes the tax of re-explaining things, and behind snapshots and undo the downside is bounded. That is why it ships on.

But "most" is not "all", and the exceptions share a property — they are the cases where a wrong change is not obviously wrong. When drift is visible, autonomy is cheap because you notice. When drift is silent, the same autonomy is how a small error becomes policy.

Four categories to queue rather than apply

1. Work with regulatory or professional exposure

Anything where a procedure has to be defensible to somebody else later — financial advice, clinical adjacency, legal process, regulated communications. The problem is not that an agent will write something outrageous; it is that it will write something slightly off, apply it consistently, and produce a hundred defensible-looking outputs that share one flaw. Queue the change so a human owns the procedure.

2. Processes where errors compound invisibly

Classification and routing are the common examples. If an agent quietly adjusts how it categorises enquiries, nothing looks broken — the queue still empties. Six weeks later a category has been systematically misrouted and the evidence is buried in a thousand correct-looking decisions. Anything feeding a downstream process that trusts its input belongs here.

3. Anything you cannot audit afterwards

If the output leaves your systems and you have no record of what was sent — a channel you do not log, an integration that reports nothing back — then self-improvement removes the only thing keeping it honest. Fix the audit gap first; the autonomy question is secondary.

4. The first few weeks of any deployment

Not a category of work but a phase, and the one most often ignored. Early on you do not yet know what good output looks like for this agent, so you cannot tell a useful self-edit from a bad one. Let it queue, read what it proposes, and turn autonomy on once you have opinions. This is temporary, and skipping it is the most common way teams end up distrusting a system that was fine.

What to turn off first

Autonomy is not one switch, and the useful move is usually narrowing rather than disabling. In rough order of what to restrict first:

• Edits to the agent's own instructions. The highest-leverage change it can make, and the one whose effects reach every future decision. Queue this even when everything else is autonomous.
• New skills that touch external systems. A procedure that only reads is low-risk; one that writes, sends or pays is not.
• Anything affecting classification or routing. Per category 2.
• Memory about people. An agent recording an inference about a customer, and then acting on it forever, is a different category of mistake from one misremembering a price.

What is almost always safe to leave autonomous: tidying its own workspace, refining how it words things, and remembering facts you have stated explicitly.

The test worth applying

One question, for any category of self-change:

If this change were subtly wrong, how long before somebody noticed?

Hours: leave it autonomous, the feedback loop is doing the work. Weeks: queue it. Never: do not just queue it — fix the visibility problem, because you have one whether or not the agent is improving itself.

What this is not an argument for

Turning it all off. An agent that cannot retain what it learns makes you the memory, which is the most expensive possible configuration — you pay the re-explanation tax forever and get none of the compounding. The queued mode exists precisely so that the answer to "is this safe?" can be "not yet" rather than "no".

The safeguards that make the autonomous mode reasonable are in the four safeguards.