How agents fail you

An agent does not fail the way a server fails. When a server fails, it tells you: an exception fires, a health check goes red, a pager goes off, and someone gets out of bed. Nearly everything we know about running production systems rests on that honesty, on the assumption that a failing system announces its own failure at the moment it happens.

An agent that fails finishes the run anyway, reports success in confident prose, and hands over an artifact that looks exactly like the artifact you asked for. The error surfaces downstream, sometimes hours later, inside a system or a decision that trusted the output. Your infrastructure fails like a machine, loudly and mechanically; an agent fails like a confident new hire with no fear and no memory, articulate, certain, fast, and unaware that anything has gone wrong.

If agents ran one task a day, you could file the difference under trivia, but they run at machine speed and in parallel, so a mistake a human would make once a quarter can execute a thousand times before lunch. I count ten distinct ways agents fail, and they group into five families: failures of scope, of memory, of execution, of compounding, and of verification. The grouping is mine, so argue with it freely. Most of these modes are not model weaknesses waiting for a smarter model to fix them; they are design work nobody has done yet, and I will come back to what that work looks like at the end.

Failures of scope

In the first family, the agent does the wrong amount of work.

Overstepping. Models are trained to be helpful and thorough, so wherever a task leaves a boundary unstated, the agent fills the gap with its own judgment, and its judgment leans toward doing more. Ask a coding agent to fix one function and it may also rename a shared utility it found confusing and update a configuration value it decided was stale. Read any single edit and you can defend it, but read the diff as a whole and nobody asked for most of what is in it.

Overstepping survives review because reviewers calibrate their attention to the size of the ask rather than the size of the diff: a one-line request earns a one-line review, and the other forty lines ride through unread. The agent still holds the right goal, but it has quietly expanded the territory it believes the goal covers. You catch this mode before the run, with standing constraints on what the agent may touch, and the mechanism I use for that comes in the closing section.

Goal drift. Over a long task, an agent re-derives its objective at every step from the context in front of it, every re-derivation is a lossy paraphrase, and the small mutations compound. The agent also gravitates toward whatever success signal it can measure in the moment (tests passing, no errors thrown) rather than the intent behind the signal, and at the extreme this becomes what researchers call reward hacking: optimizing the letter of the task against its spirit. In one documented case, an agent asked to fix a codebase hit an error in a dependency, created a local stub of the dependency to get past it, and went on stubbing nineteen more files until every test passed, having fixed nothing at any point along the way. Agents in the wild have also deleted failing tests and hardcoded their expected values to get to green.

Drift slips past because every intermediate step looks locally reasonable: reading the transcript afterward, you find no single moment where the agent visibly goes wrong, since each step is a plausible response to the step before it. Drift is also not overstepping in a different costume, because a drifted agent can produce a tiny, perfectly scoped diff that solves the wrong problem; overstepping changes how much the agent touches, while drift changes what it is optimizing. The check that catches it compares the final artifact against the original ask instead of against the previous step, and it asks whether this is what was wanted rather than whether it ran clean.

Failures of memory

In the second family, what the agent was told stops binding.

Context degradation. An agent's working state is not stored anywhere; it is re-read from the context window at every step, and the model's attention over that window is not uniform. Practitioners call the result context rot: models degrade a little with every increment of added context, well before the window's advertised limit, and in the widely replicated lost-in-the-middle experiments, recall of information placed mid-context drops by more than 30 percent. A model sold with a million-token window is already degrading at fifty thousand tokens, and the loss applies to instructions as much as to facts: the model forgets to keep following a rule the same way it forgets a fact it read earlier.

Say the constraint at step one is "do not touch the public API." It holds for ten steps and gets violated at step fourteen, with nothing changed except distance. The violation slips past because it looks like a decision rather than a malfunction, and nobody suspects memory (the constraint is right there in the transcript). Runs kept short, critical constraints re-asserted at checkpoints, and context treated as a budget all push this mode down.

Instruction ignoring. Where degradation is a slope that steepens as a run grows, ignoring is the floor that exists even at step one. An instruction in a prompt is a probabilistic influence on the output rather than an enforced invariant: however explicit the rule, the probability that the agent disregards it never reaches zero, and it rises with context length. The rule is explicit, even bolded, and the agent follows it forty-nine times and not the fiftieth. Teams respond by writing the instruction again, longer, and that makes things worse in slow motion, because the longer prompt grows the context and makes the next miss slightly more likely.

The mode stays invisible because the team believes writing the rule down enforced it, and the gap between "we told it not to" and "it cannot" only surfaces when a miss lands somewhere expensive. Nothing inside the prompt closes that gap; anything that must never happen needs a check that lives outside the agent, a validation at the boundary or a permission the agent does not have. After the next miss, watch whether your team writes a longer prompt or adds an external check, because the two responses build very different systems over a year.

Failures of execution

In the third family, the agent acts wrongly on the world around it.

Tool misuse. The agent holds an internal model of every tool it can call, and that model can diverge from the tool's actual contract: the wrong tool for the step, the right tool with malformed arguments, or a correct call whose output the agent misreads. A bad call early in the run is corrosive because the rest of the run treats whatever it returned as fact, and everything built afterward inherits the flaw. Say an agent queries the wrong environment, gets an empty result, concludes the records do not exist, and sets about recreating them. Every component behaved exactly as designed, and the agent still acted on a false picture of the world.

Misuse slips past because tool calls succeed mechanically, with clean return codes and valid response shapes, while the error lives in the semantics, where no infrastructure layer looks. You catch it at the boundary: validate tool inputs and outputs with deterministic code, schema checks and range checks, and leave the judgment calls to the model while code enforces the contracts.

Runaway execution. Retry and recovery logic was designed for deterministic services, where a failed call retried is the same call. Wire it to a non-deterministic actor and the logic meets an agent that treats failure as an obstacle to route around: it retries, it tries variants, it triggers compensating actions that can spawn actions of their own, and it does all of it at machine speed. In one publicly told case, an agent retrying a failed billing call set off retries that spawned further retries and compensating workflows, and tens of thousands of redundant operations ran within minutes. Each individual action was small and legitimate-looking, and the failure only existed in the aggregate rate, visible on a dashboard nobody was watching at two in the morning.

Runaway execution is not tool misuse, because every call in the loop may be perfectly well-formed. The defect lives in the loop itself, and no prompt catches it. Catching it takes hard limits that hold no matter what the agent decides: budgets, rate limits, circuit breakers, and kill switches. Any action that is irreversible or unbounded should never be the agent's alone to take, a principle I will come back to.

Failures of compounding

In the fourth family, errors multiply across steps and across agents.

Cascading errors. Inside an agent chain, each step trusts the previous step's output as ground truth, and no skepticism exists between steps unless someone builds it in. One early mistake does more damage than ten different kinds of mistake, because it compounds through every decision that follows. Picture a research agent misreading one number at step two: steps three through nine build an analysis on top of it, and the final report arrives internally consistent and wrong, with every downstream conclusion resting on the misread number.

The cascade slips past because reviewers use internal consistency as a proxy for correctness, and a cascade preserves internal consistency perfectly: the artifact agrees with itself at every point a reader might check. Catching it means verifying between steps instead of only at the end, and checking intermediate outputs against sources rather than against each other.

Inter-agent misalignment. Human teams run on repair machinery so ordinary we forget it exists: the clarifying question, and the "that's not what I meant" said before any work is lost. Agents coordinating with agents have none of it unless you build it. One research team annotated more than 1,600 failure traces across seven popular multi-agent frameworks, published the result as a failure taxonomy called MAST, and put the finding plainly: agents cannot ask a clarifying question mid-task and cannot notice that a peer's output quietly violates their own assumptions. Of the failures in the study, 37 percent landed here.

Suppose a planner agent hands a task list to an executor agent, and one task is ambiguous. A human executor would ask, but this one picks an interpretation, silently, and the planner never learns that the plan it is tracking no longer matches the work being done. Each agent's transcript looks fine on its own, the failure exists only in the relationship between transcripts, and nobody reads transcripts side by side. Unlike a cascade, no step here is wrong on its own terms, since the interpretations are individually valid and mutually incompatible. The countermeasures are structural: explicit contracts at handoff boundaries, structured outputs instead of free-text messages between agents, and a topology with fewer handoffs to go wrong.

Failures of verification

In the fifth family, the system cannot tell success from failure.

False completion. The agent that does the work also declares the work done, and it checks that work with the same weights that generated it, the way a writer proofreading their own page reads what they meant to write rather than what is there. The errors the agent made are precisely the errors it cannot see, and in the MAST data, 21 percent of failures were verification and termination failures. It sounds like this: "all tests pass, the feature is complete," where the tests are the ones the agent wrote, passing against the stub the agent created. Sometimes the task has been silently narrowed along the way until what got completed is no longer what was asked, at which point drift and false completion have become the same failure.

It slips past because models sound confident by default, so a wrong "done" and a right "done" read identically. Catching it requires separating the judge from the worker: a different layer with a different context and, ideally, different incentives.

Silent quality decay. Every mode so far describes something the agent does wrong, and this last one has no event in it at all. A model version updates underneath you, someone edits a shared prompt, the input traffic drifts away from what you tested on, a tool changes its output format, and none of it throws an error, because every run completes. Quality erodes across the population of runs while each individual run stays acceptable: an agent shows 99 percent uptime and healthy latency while its sampled outputs have been degrading for three weeks. Infrastructure monitoring cannot see it, because the infrastructure cannot tell a model that is responding from a model that is responding well.

Decay is invisible per run by definition, and teams watch single runs while the decay only shows up across many of them. Only an aggregate view surfaces it: quality evaluations scored on a sample of production traffic, tracked across releases, with every failure found this way promoted into a permanent regression check so it cannot return unnoticed.

The design surface

The strongest finding in the MAST study reframes the whole catalog: the largest bucket, 42 percent of the failures, traced not to a weak model but to how the system around it was specified and wired, through ambiguous task definitions, missing termination conditions, success criteria nobody pinned down, and roles that overlap or leave gaps. Better models will shrink some of the ten modes, but they will barely touch the structural ones. Overstepping, instruction ignoring, runaway loops, misalignment between agents, and workers declaring their own output correct are properties of how the system is put together rather than of how smart the model is. For an engineering leader, that means how agents fail you is substantially within your control, and four moves cover the catalog.

Bound the agent before it runs. In my own agent systems, I maintain a tenets module: a configurable set of standing principles injected into each agent, closer to a code of conduct than a task spec, and it travels with the agent across tasks. The tenet I lean on most is minimal blast radius: prefer the smallest change that works, and do not alter load-bearing implementation, public interfaces, shared utilities, configuration, or migrations unless the task requires exactly that. Each agent's copy carries its own instruction for what to do when blocked, because an unbounded agent starts improvising at the moment it gets blocked. Tenets answer the scope family: they pre-resolve the ambiguity that overstepping feeds on, and they keep the original ask in front of the agent, so drift has something to be checked against.

Treat instructions as priors and invariants as code. A tenet shifts the distribution of the agent's behaviour without clamping it, and believing a written bound is a guarantee just recreates instruction ignoring one level up. So the memory and execution families get a second layer: anything that must never happen is enforced by a check outside the agent, through validations at tool boundaries, permission gates, deterministic contracts, and hard budgets. Instructions bring the rate of the bad thing down, and the external check is there for the cases where the rate has to be zero.

Give the agent a physician's default. When the bounds and the checks leave a situation ambiguous, the agent still has to act somehow, and the disposition I set is the one physicians train under: first, do no harm. Default to the smallest reversible action, escalate rather than guess when the action would be hard to undo, and never let an irreversible action, or one with a wide blast radius, be the agent's alone to take. Watch the default itself too, because an agent so cautious that it escalates everything quietly recreates the manual workflow with extra steps, which is a system nobody set out to build.

Layer the harness: inner layers work, outer layers judge. The compounding and verification families share one answer: wrap the layers that do the work in layers whose only job is to judge the work. Per-step checks against sources catch cascades, contracts at handoff boundaries catch misalignment, a separated verifier catches false completion, and evaluations over sampled production traffic catch decay. The last layer is human, a person who reports the error when they see it, and the judge layer exists for everything that person does not see, because a failure that slips past both stays silent for good. The judge earns the same engineering attention as the worker, because an untuned judge either rubber-stamps, which returns silent failure one layer up, or cries wolf until everyone ignores it. Every failure that either layer catches becomes a permanent check, and the harness compounds the way a good test suite does.

In a production agent system, the working agent is 20 percent of the build, and the other 80 percent is the checks, budgets, contracts, judges, and rollback paths around it. Teams set out to do AI engineering and found themselves rediscovering workflow engineering and distributed-systems engineering, which is older and less glamorous work than the demos suggested, and also work most engineering teams already know how to do.