This is the companion to our general Amazon Connect runbooks guide. Everything there still applies to an ACXD deployment — outages, numbers, Lambda failures, rollbacks. But ACXD adds a category of failure that deterministic contact flows simply don't have: the application can misunderstand a request, give a plausible but incorrect answer, breach a policy, or take the wrong action through a tool — and do all of it confidently, without throwing an error.
That changes what a runbook has to do. For a deterministic failure, the runbook helps you fix the thing. For an agentic failure, the runbook often has to help you detect something that isn't erroring, contain it fast (guardrails, fallback to human), and then roll back and investigate — because there may be no single line of broken code to point at.
New to ACXD? This assumes familiarity with the basics — applications, flows, nodes, builds, guardrails, and the build-test-deploy-observe lifecycle. If you need the grounding first, the concepts here build on ACXD guardrails and ACXD CI/CD with the SDK.
Why ACXD Needs Its Own Runbooks
Three properties of an agentic application drive the difference:
- Non-deterministic behaviour. The same input can produce different outputs. A failure may be intermittent and not reproducible on demand, which makes "can't reproduce, closing" a dangerous instinct.
- Silent failure. A hallucinated answer or an off-policy response doesn't raise an exception. If your only alerting is on errors and latency, you won't see it — detection has to come from guardrails, evals, analytics, and human reports.
- Action, not just answers. Agentic applications can do things — call APIs, update records, trigger processes. A wrong action has consequences a wrong answer doesn't, so containment urgency is higher.
The core operating principle: for consequential actions and sensitive answers, your containment can't depend on noticing the problem in real time. It has to be built in beforehand — deterministic guardrails, confirmation steps before consequential actions, and a clean fallback-to-human path. Runbooks assume those exist; if they don't, that's a design gap, not an ops one.
What an ACXD Runbook Should Include
ACXD runbooks use the same backbone as any runbook (trigger, severity, owner, diagnostics, resolution, rollback, escalation, communication, post-incident — see the general runbooks guide for the full template), plus a few AI-specific additions:
| ACXD-specific section | Why it's there |
|---|---|
| How to detect it | Because the failure may not error — name the signal (guardrail trigger log, eval score drop, analytics tag, human report, sentiment dip) |
| Example transcript | A sample of the bad behaviour so the responder recognises it and can confirm the scope |
| Build / version in play | Which deployed build the application is running, so you know what to roll back to |
| Containment before fix | How to stop harm immediately — tighten/activate a guardrail, disable a flow, or fall back to human — before root-causing |
| Blast radius check | How many contacts were affected and whether any wrong actions were taken that need reversing |
Which ACXD Runbooks to Write
On top of your standard Connect runbooks, write these ACXD-specific ones:
- Bad build / deployment rollback — a new build regressed quality or behaviour
- Guardrail breach / unsafe output — the application produced restricted, off-brand, or risky content
- Hallucination / wrong answer — fluent but factually incorrect responses
- Knowledge base problem — stale, missing, or incorrect source content producing bad answers
- Wrong or runaway tool action — the application called a tool it shouldn't have, or repeatedly
- Prompt / model change regression — a prompt edit or model update degraded performance
- Fallback-to-human — taking the application out of the loop entirely and routing to agents
- Containment escalation — when and how to disable the application in production
The one every ACXD operation must have: the fallback-to-human runbook. If the application is misbehaving and you can't fix it quickly, the ability to cleanly route all affected contacts to human agents — and know exactly how to flip that switch — is your ultimate safety net. Build and test it before go-live.
Worked Example Runbooks
Condensed examples showing the ACXD-specific shape. Full versions carry the complete section set.
RB-A1: Guardrail Breach / Unsafe or Off-Brand Output
Trigger: The application produced restricted, non-compliant, offensive, or badly off-brand content — spotted via guardrail logs, a human report, or escalated complaint.
Detect: Check guardrail activity logs for triggered rules; search analytics/transcripts for the pattern to gauge scope.
Contain first (before root cause):
- Activate or tighten the relevant guardrail (e.g. switch a rule to Override/Redirect/Mask) to block the output pattern immediately.
- If guardrails can't contain it, fall back to human for the affected flow (invoke RB-A7).
Resolve: Identify whether cause is a prompt, a knowledge source, a missing guardrail, or a model change; fix in a new build; test with trigger and non-trigger cases before redeploying.
Escalate & communicate: Notify the application owner, security/compliance, and Operations Lead; if customer-facing harm occurred, follow the incident-communications process.
Post-incident: Add the failing case to the eval set so it's caught automatically in future.
RB-A2: Hallucination / Confidently Wrong Answers
Trigger: The application is giving fluent but incorrect answers — reported by customers, agents, QA, or a drop in an eval/faithfulness score.
Detect: Pull affected transcripts; confirm whether the wrong answer is unsupported by the knowledge base (hallucination) or faithfully reflecting bad source content (a knowledge problem — go to RB-A3 instead).
Contain: If the topic is high-stakes, tighten the flow to confirm/escalate on that intent, or route it to a human, while you investigate.
Resolve: Depending on cause — strengthen grounding/prompts to force citation of knowledge, add a guardrail (LLM-judge) on that answer type, or correct the knowledge source. Rebuild and re-test against evals.
Post-incident: Capture the case in the eval set; check whether a recent prompt/model change introduced it (RB-A6).
RB-A5: Wrong or Runaway Tool Action
Trigger: The application called a tool/API it shouldn't have, with wrong parameters, or repeatedly — potentially changing records or triggering downstream processes.
Contain immediately:
- Disable the offending tool/data request or the flow that invokes it (or fall back to human for that journey).
- Assess blast radius: how many contacts, and what actions were actually taken that may need reversing in the downstream system.
Resolve: Fix the flow logic/tool contract so the action is correctly gated (and gated behind confirmation if consequential); redeploy via a new build.
Escalate: Owning team for the affected downstream system, to reverse any erroneous actions.
Post-incident: Review why a consequential action wasn't behind a deterministic confirmation — this is usually a design fix, not just an ops one.
RB-A9: Bad Build Rollback
Trigger: A newly deployed build has regressed behaviour or quality.
Steps:
- Confirm the regression correlates with the recent deployment (check deploy time vs when symptoms started).
- Redeploy the last known-good build to the affected environment — ACXD's immutable-build model supports one-call rollback via the SDK.
- Verify the previous behaviour is restored (run a quick smoke test / key eval cases).
- Record which build was restored, who approved it, and the trigger.
Post-incident: Investigate the regressed build in a non-prod workspace; add a test that would have caught it to your CI gate.
The Observability That Makes These Work
ACXD runbooks are only usable if you can actually see the failures. Make sure these are in place before go-live, because they're what the "detect" steps rely on:
- Guardrail activity logs — your primary signal for policy/safety breaches
- Conversation analytics & transcripts — to find and scope bad behaviour patterns
- A production eval set & scoring — to catch quality drift that no error log will show you (see using evals to deliver excellence)
- AI-specific metrics — hallucination rate, guardrail-breach rate, groundedness, tool-call success, containment quality (see new AI agent metrics)
- Build/version attribution — so every observed behaviour can be tied to the build that produced it
Runbooks and evals are two halves of the same loop. Every time a runbook is triggered by a real failure, the fix should include adding that case to your eval set — so the thing that caught you out once gets caught automatically forever after. That's how an ACXD operation gets more reliable over time rather than just reacting repeatedly.
Keeping ACXD Runbooks Alive
The general maintenance rules apply — owner, last-reviewed date, accessible storage, version control, game-day testing. Two ACXD-specific additions:
- Re-review after every model or prompt change. A model update can shift behaviour enough to invalidate a runbook's assumptions. Treat significant prompt/model changes as a reason to re-check the relevant runbooks.
- Keep example transcripts current. The "what the bad behaviour looks like" examples age as the application changes — refresh them so responders are matching against reality.
Conclusion
Operating ACXD well means accepting that some failures won't announce themselves with an error — they'll be fluent, confident, and wrong. ACXD runbooks exist to make those failures detectable, containable, and recoverable: detect through guardrails, evals, and analytics; contain through guardrails and fallback-to-human; recover through immutable-build rollback. Write the fallback-to-human and rollback runbooks first, build the observability that the detect steps depend on, and feed every real incident back into your eval set.
Pair this with the general Amazon Connect runbooks guide, ACXD guardrails, and AI agent metrics for the full operating picture.