Agent Incident Responder
When an agent misbehaves at 3am in a market you do not speak, this is what finds it and contains it.
target under 30 min
Mean time to containment
70%+
Incidents caught pre-report
100%
Postmortems yielding new evals
The problem
Agent failures are not like service outages. They are quiet, semantic and local: a tool loop that burns budget, a policy drift that only shows up in one language, a jailbreak that spreads through a community overnight. Traditional alerting sees none of it.
As soon as agents take real actions, mean time to containment becomes the number the business cares about. Building the response capability before the first serious incident is the difference between a fix and a headline.
Agent design
- A detector bank watching semantic signals - refusal spikes, tool-loop depth, sentiment collapse per locale - rather than only latency and errors.
- A triage agent that clusters related reports across languages, so the same incident in six markets is one page, not six.
- Containment actions expressed as reversible controls: disable a tool, tighten a policy, force human review for a segment.
- A postmortem writer that assembles the trace timeline and proposes the eval item that would have caught it.
Guardrails
- Containment actions are reversible by design and always logged with the operator identity.
- The agent proposes containment; a human commits anything that degrades service for real users.
- Every closed incident must produce a new evaluation item, or it cannot be closed.
Detection is a language problem
The teams awake at the time of an incident often do not speak the language it is happening in. Detectors have to work on behavioural shape rather than keyword lists, or coverage collapses to English office hours.
Closing the loop
An incident that does not become an eval item will happen again. Making that a hard gate on closure is the cheapest quality mechanism I know of.