Carousel · AI Agents · Oct 9, 2026
Your AI agent works in the demo. Production is a different story.
In short
AI agents are distributed systems that can lie: they call tools, retry, wait on slow dependencies, and sound confident when they're wrong. Five ways they fail in prod, each with the SRE fix: confident wrong answers, retry storms and infinite loops, prompt injection via tools, silent drift, and cascading failures. Most teams already have observability, but only about half run evals. That gap is where the bugs live.
Checklist: what to do about it
- Confident wrong answers → evals plus strict output validation.
- Retry storms and infinite loops → step budgets, backoff with jitter, circuit breakers.
- Prompt injection via tools → least-privilege tools, sanitised inputs, human approval for writes.
- Silent drift → golden-set evals in CI plus per-conversation tracing.
- Cascading failures → timeouts, fallbacks, graceful degradation.
- Observability tells you it broke. Evals tell you it's wrong. Run both.
Sources
Stats source (from the caption): LangChain, State of Agent Engineering (1,340 respondents, Nov–Dec 2025). The carousel's numbers: 57% have agents in production, 89% have observability, 52.4% run evals.