Field Note

The hard part of AI delivery is recovery

A mountain ridgeline at sunset
A mountain ridgeline at sunset.

AI makes change cheap. Great. Now what happens when it is wrong?

That is the less exciting question and the one I care about. Generation gets the demo. Recovery gets production.

Happy paths are cheap now

Models are very good at producing the visible path through a system. A shopper can browse. A service agent can summarize an order. An integration can transform a payload. A feature can pass the first demonstration.

The expensive behavior usually starts after that:

  • The dependency times out after accepting the request.
  • A retry creates a second transaction.
  • The data import partially succeeds.
  • The new field is missing on older records.
  • A release changes behavior nobody was monitoring.
  • The rollback restores code but not the data already changed.

These are not unusual edge cases. They are normal conditions in a system with networks, state, vendors, and people.

AI did not create that complexity. Faster delivery gives us more opportunities to collide with it.

Recovery has four parts

I think about recovery as four separate capabilities.

Detection: Can we tell that the system is wrong before a customer explains it to us?

Containment: Can we stop the impact without shutting down everything around it?

Reversal: Can we return code and state to a known condition, or compensate for what already happened?

Learning: Can we turn the failure into a test, control, or architectural change?

A rollback button only addresses part of the third item. It is useful. It is not a recovery strategy.

Ask the agent for the failure design

When an AI system proposes an implementation, ask it to produce the failure model with the code.

What can fail? Which failures are retryable? What is the idempotency boundary? Which state changes happen before the external call? How is partial completion detected? What telemetry proves the job finished? How would an operator repair one bad record without replaying everything?

The first answer will often expose how much the implementation assumes.

This is not unique to AI. Plenty of human-designed systems have discovered idempotency in production. The difference is that AI can generate enough implementation detail quickly that the design feels more complete than it is.

Working code is persuasive. Sometimes too persuasive.

Practice the ugly path

Recovery plans that have never been exercised are architecture fiction.

Before expanding an AI-assisted delivery model, run controlled failures. Break the dependency. Corrupt a non-production payload. Interrupt an import. Roll back the deployment. Restore from a known state. Hand the incident to somebody who did not generate the implementation.

Then watch where the process depends on hidden context or the original conversation.

If only the person who prompted the change can recover it, the organization does not own the system yet.

Speed should buy smaller risk

The best operational use of faster implementation is not necessarily bigger releases. It is smaller changes, earlier tests, and cheaper experiments.

AI can make it practical to test an integration contract before the full program depends on it. It can generate fixtures, failure cases, runbooks, and observability scaffolding alongside the feature. It can help turn an incident into a regression test while the evidence is still fresh.

That is where speed becomes resilience instead of volume.

The real promise is not software appearing without effort. It is learning faster without betting the whole system on every lesson.

If recovery is still heroic, AI did not fix delivery. It just made the next incident arrive faster.

Field Notes

More Field Notes.