Self-healing is useful when it means restoring a known desired state under an explicit policy. Kubernetes provides a familiar example: controllers replace failed containers and workloads, maintain replica counts, and remove unhealthy endpoints. That is reconciliation. It is not autonomous source-code repair, and it does not prove that an application is correct.
For AI-assisted systems, “deterministic” should describe the repair policy, gates, and state transitions—not a promise that a model will always produce the same patch or that verified software has no defects.
Separate three repair layers
The safest architecture keeps increasingly risky actions in separate lanes:
- Operational reconciliation: restart a process, replace an unhealthy instance, reschedule work, fail over, or roll back to a known artifact.
- Configuration correction: restore an approved value, rotate an expired lease through an authorized service, or reconcile declarative infrastructure.
- Code repair: diagnose a defect and draft a source change for evaluation and approval.
The first two layers can often use allowlisted actions with strong preconditions and reversible outcomes. Code repair is different: the search space is open-ended, tests may be incomplete, and a plausible patch can change behavior outside the observed failure. Treat it as a proposed change, not a healing primitive.
Use an explicit repair state machine
A bounded loop can be represented as:
detect -> classify -> select policy -> stage repair -> validate
-> canary -> observe -> promote
\-> rollbackEvery transition should be enforced outside the model. The policy defines:
- accepted signals and their freshness;
- incident and risk classes;
- allowlisted actions for each class;
- maximum attempts, time, and cost;
- required checks and independent oracles;
- canary size and observation window;
- rollback conditions;
- actions that require a human.
The model can classify evidence or propose a patch, but it should not grant itself permission, alter the success criteria, or declare its own output correct.
Verification is layered evidence
Different techniques answer different questions:
- Unit and integration tests check named examples and interactions.
- Property-based tests generate many inputs from stated properties and shrink failures to smaller counterexamples. Hypothesis is a mature implementation of this approach. Passing generated cases is strong evidence, not a mathematical proof over all inputs.
- Static analysis and type checking detect classes of defects within their models.
- Formal verification can prove that a program or model satisfies a written specification under stated assumptions. Dafny verifies implementations against contracts; TLA+ and its TLC model checker analyze specifications and reachable model states. A proof is only as relevant as its specification, model, and assumptions.
- Canary observation tests behavior in a limited real environment before broad promotion.
Use the cheapest adequate oracle first, then require stronger evidence as impact increases. Authentication, authorization, money movement, destructive data operations, migrations, and safety-critical controls should not be automatically promoted from an LLM-generated patch.
Make repair selection deterministic
For a known failure class, the policy should produce the same allowed action set from the same validated inputs. For example:
IF health probe fails for 3 consecutive intervals
AND artifact digest is unchanged
AND replacement budget is available
THEN replace one instance
ELSE escalateThis is safer than asking a model “what should we do?” for every incident. Models are most useful after deterministic recovery is exhausted: summarizing evidence, finding related changes, or drafting a candidate fix inside a restricted sandbox.
Idempotency matters. Every action needs an incident identifier, target identity, expected prior state, and idempotency key. A repeated event must not create repeated destructive effects. Store the decision, policy version, inputs, action result, and resulting state for audit and replay.
Never repair from a single weak signal
An alert may be stale, correlated rather than causal, or caused by an external dependency. Before mutation, combine signals such as:
- health and error-rate windows;
- a known-good baseline;
- recent deployment and configuration history;
- dependency health;
- resource saturation;
- reproducibility in an isolated environment.
Classification should be able to return “unknown.” An unknown incident is an escalation condition, not permission for a broader experiment in production.
Bounded code-repair workflow
When source repair is authorized:
- Pin the repository, dependencies, failing input, and environment.
- Reproduce the failure in an isolated sandbox.
- State the invariant or acceptance criterion before generating a patch.
- Limit writable paths, tools, attempts, network access, and runtime.
- Run independent checks that the patch cannot rewrite.
- Compare behavior against a known-good baseline and test for regression.
- Produce a reviewable diff and evidence bundle.
- Canary only if policy permits; otherwise stop for approval.
- Roll back automatically on a predefined signal.
Do not let the repair agent weaken tests, suppress errors, broaden permissions, or change monitoring thresholds unless that exact change is separately approved.
What to measure
Track recovery time, false repair rate, rollback success, repeated incidents, escalation rate, repair attempts per incident, affected-user exposure, and cases where validation passed but production behavior regressed. Evaluate the policy on recorded incidents and fault-injection exercises before enabling automatic promotion.
The credible path to self-healing is deliberately unglamorous: deterministic reconciliation for known states, independent evidence for proposed changes, small blast radii, and a reliable stop button.