A traditional IVR only does what it was scripted to do. If it misbehaves, it's because someone designed it that way. A self-service AI agent is different: it reasons, generates language, and takes actions autonomously. That autonomy is what makes it powerful — and it's exactly why it needs a proper risk assessment before it goes anywhere near a live customer.
An AI risk assessment isn't box-ticking. Done well, it's the document that lets you deploy with confidence: it names every plausible failure, rates how bad and how likely it is, records the control that mitigates it, and defines how you'll know if it happens in production. This report walks through what that assessment must cover.
How to use this report: Treat it as a template. Work through each of the six risk domains below, identify which risks apply to your specific agent and sector, document the controls you have (or need), and score the residual risk. The checklist at the end pulls it all together.
Start With Scope and Context
Before assessing risk, define what you're assessing. The same technology carries wildly different risk depending on what it's allowed to do. Capture:
- What the agent can do — informational answers only, or can it take actions (payments, account changes, cancellations)?
- Who it serves — general public, vulnerable customers, minors, businesses?
- What data it touches — personal data, financial data, health data, special-category data?
- The sector and regulation — financial services, healthcare, utilities all carry specific obligations.
- The channels — voice, chat, messaging; each has different risks (e.g. voice recording, deepfakes).
- The level of autonomy — fully autonomous, human-in-the-loop, or human-on-the-loop?
The higher the autonomy and the more sensitive the data and actions, the deeper the assessment needs to be. An agent that reads opening hours needs a light touch; one that moves money needs the full treatment.
The Six Risk Domains to Cover
A complete AI risk assessment for a self-service agent spans six domains. Work through each in turn.
1 Safety & Accuracy Risk
The risk that the agent gives wrong, harmful, or misleading information — the failure mode unique to generative AI.
Risks to assess
- Hallucination — the agent fabricates facts, policies, prices, or entitlements that don't exist.
- Unfaithful answers — it cites a real source but misrepresents what it says.
- Outdated information — it answers from stale training data or an un-refreshed knowledge base.
- Harmful advice — it strays into medical, legal, or financial advice it isn't qualified or permitted to give.
- Overconfidence — it states guesses as certainties, giving customers false assurance.
Key controls
- Ground answers in an approved, current knowledge base (RAG) rather than model memory.
- Guardrails that block out-of-scope topics and advice.
- Continuous hallucination and faithfulness monitoring (LLM-as-a-judge) on live traffic.
- Confidence thresholds with fallback to a human when uncertain.
2 Security Risk
The risk that the agent is manipulated, abused, or used as an attack surface into your systems.
Risks to assess
- Prompt injection / jailbreaking — a user crafts input that overrides the agent's instructions or guardrails.
- Data exfiltration — the agent is tricked into revealing another customer's data, system prompts, or internal information.
- Unauthorised actions — social engineering pushes the agent to perform actions the caller isn't entitled to.
- Tool/API abuse — the agent's connected tools are exploited beyond their intended use.
- Synthetic voice / deepfake attacks — cloned voices used to defeat voice-based verification.
- Insecure secrets handling — API keys or credentials exposed in prompts, logs, or config.
Key controls
- Adversarial testing (red-teaming) for prompt injection and jailbreaks, run continuously.
- Strict authorisation checks on every tool/action — the agent's request never bypasses backend access control.
- Least-privilege scoping of the agent's tools and data access.
- Multi-factor, risk-based identity verification; do not rely on voice biometrics alone.
- Secrets held in a secrets manager, never in prompts or logs.
3 Data & Privacy Risk
The risk of mishandling personal or sensitive data — often the domain with the sharpest regulatory teeth.
Risks to assess
- Excessive data collection — the agent gathers more than it needs for the task.
- Data leakage to the model provider — sensitive data sent to a third-party LLM without appropriate controls or agreements.
- Retention and logging — sensitive data (card numbers, health info) captured in transcripts or logs.
- Training on customer data — interactions inadvertently used to train models without consent.
- Cross-border transfer — data processed in a region that breaches residency requirements.
- Consent and transparency — customers not told they're talking to AI or how their data is used.
Key controls
- Data minimisation by design; redact sensitive data before it reaches the model where possible.
- Contractual and technical guarantees that provider won't train on your data.
- PCI-compliant capture for payment data; suppress sensitive data from recordings and logs.
- Clear AI disclosure to customers and a lawful basis for processing.
- Data residency controls aligned to your obligations (e.g. UK/EU processing).
4 Compliance & Regulatory Risk
The risk of breaching laws, regulations, or industry rules — and of failing to prove you didn't.
Risks to assess
- Sector-specific breaches — e.g. financial promotions rules, fair-treatment obligations, healthcare regulations.
- Vulnerable customer duty — failing to recognise and appropriately handle a vulnerable customer.
- Auditability gap — inability to reconstruct what the agent said, did, and why, for a regulator or dispute.
- Discrimination / bias — the agent treats customers unfairly based on how they speak, their accent, or inferred characteristics.
- Emerging AI regulation — obligations under frameworks like the EU AI Act for higher-risk AI systems.
- Record-keeping — required disclosures, confirmations, or consents not captured.
Key controls
- Full, immutable audit trail of every interaction, including the agent's reasoning and actions.
- Defined vulnerable-customer detection and escalation path.
- Bias testing across accents, dialects, and demographics.
- Legal/compliance review of scope and prompts before launch, and after material changes.
- Mapping of the agent against applicable AI-regulation risk tiers.
5 Operational & Technical Risk
The risk that the agent fails, degrades, or behaves unpredictably in live operation.
Risks to assess
- Model drift / regression — behaviour changes silently after a model or prompt update.
- Availability and latency — model or tool outages, or slow responses that break the experience.
- Dependency risk — reliance on a single model provider or a backend that changes without notice.
- Cost overrun — token and inference costs spiral with volume or verbose behaviour.
- Cascading tool failures — one failing integration causes broad, hard-to-diagnose breakage.
- Change management — untested prompt or config changes pushed straight to production.
Key controls
- Version attribution and drift monitoring against a fixed evaluation set.
- Deploy via a tested pipeline with staging, canary, and automated rollback.
- Fallback behaviour and graceful degradation when a model or tool is unavailable.
- Cost and latency monitoring with alerting thresholds.
- A defined rollback plan and an owner on call.
6 Reputational & Experience Risk
The risk of damaging customer trust or brand, even when nothing technically "broke".
Risks to assess
- Tone and brand misalignment — the agent sounds off-brand, robotic, or inappropriate.
- Customer trapping — no clear route to a human, leaving customers stuck.
- Insensitivity — mishandling bereavement, distress, or complaints without empathy.
- Public failure — a screenshot of the agent saying something absurd or offensive goes viral.
- Accessibility gaps — the agent fails customers with disabilities or non-standard speech.
- Erosion of trust — customers lose confidence after repeated poor automated experiences.
Key controls
- Defined persona and tone guidelines, tested before launch.
- Always-available, easy escalation to a human.
- Sensitive-topic detection with compassionate handling or handoff.
- Accessibility testing and inclusive design.
- A rapid-response plan for public incidents.
How to Score Each Risk
For every risk you identify, score it consistently so you can prioritise. The standard approach is likelihood × impact, assessed twice — before controls (inherent risk) and after controls (residual risk).
| Element | What to capture |
|---|---|
| Risk description | The specific failure and how it would occur |
| Likelihood | How probable, given your design (e.g. Low / Medium / High) |
| Impact | Severity if it happens — customer harm, financial, regulatory, reputational |
| Inherent risk | Likelihood × impact before controls |
| Controls | What mitigates it, and whether the control exists today or is planned |
| Residual risk | Likelihood × impact after controls |
| Owner | Who is accountable for the control and its monitoring |
| Monitoring | The live metric or alert that tells you if the risk is materialising |
An illustrative row:
| Risk | Inherent | Control | Residual |
|---|---|---|---|
| Agent hallucinates a refund policy | High | RAG grounding + faithfulness monitoring + guardrail | Low |
| Prompt injection reveals system prompt | High | Red-team testing + input filtering + output checks | Medium |
| Customer can't reach a human | Medium | Always-on escalation path | Low |
The point of residual risk: No AI agent is zero-risk. The assessment's job is to show that residual risk is understood, owned, monitored, and accepted at the right level of the business — not that risk has been eliminated.
Governance: Who Owns It and When
A risk assessment is only as good as the governance around it. Cover these process points:
- Ownership. Risk or compliance typically owns the framework, but the assessment needs input from CX/design, engineering, security, data protection, legal, and the accountable business owner.
- Sign-off. Define who must approve deployment based on the residual risk level — higher residual risk should require more senior sign-off.
- Cadence. The assessment is a living document. Re-run it before launch, after any material change (new tools, new model, expanded scope), and on a regular review cycle.
- Human-in-the-loop policy. Decide which actions the agent can take autonomously and which require human approval — and record the rationale.
- Incident response. Define what happens when a risk materialises: who's alerted, how you pause or roll back the agent, and how you communicate.
The Reusable Checklist
Pull it all together with a checklist you can take into any self-service AI agent assessment.
- Documented what the agent can do, who it serves, and its autonomy level
- Identified data types, sector, channels, and applicable regulation
- Answers grounded in an approved, current knowledge source
- Hallucination and faithfulness monitoring in place on live traffic
- Guardrails block out-of-scope topics and prohibited advice
- Adversarial / red-team testing for prompt injection and jailbreaks
- Authorisation enforced on every action, independent of the agent
- Multi-factor identity verification; voice biometrics not relied on alone
- Secrets held securely, never in prompts or logs
- Data minimisation and sensitive-data redaction applied
- Provider will not train on your data; residency requirements met
- Clear AI disclosure and lawful basis for processing
- Full, immutable audit trail of what the agent said and did
- Vulnerable-customer detection and escalation defined
- Bias tested across accents and demographics
- Mapped against applicable AI-regulation risk tiers
- Version attribution and drift monitoring active
- Tested pipeline with staging, canary, and automated rollback
- Fallback behaviour and cost/latency alerting in place
- Persona/tone defined and tested; always-on human escalation
- Sensitive-topic handling and accessibility tested
- Public-incident response plan ready
- Owners assigned, sign-off level set by residual risk
- Review cadence and change-triggered re-assessment defined
- Incident response and rollback plan documented
Frequently Asked Questions
What is an AI risk assessment for self-service agents?
A structured evaluation of the ways an AI-powered self-service experience could cause harm — to customers, the business, or third parties — across safety, security, data, compliance, operational, and reputational domains, together with the controls that mitigate each risk and a plan to monitor them in production.
Why do AI agents need a different risk assessment from traditional IVR?
Traditional IVR is deterministic — it only does what it was scripted to do. Agentic AI generates responses and takes actions autonomously, introducing new risks such as hallucination, prompt injection, unsafe autonomous actions, and model drift that a traditional IVR risk assessment never had to consider.
Who should own the AI risk assessment?
It should be cross-functional. Risk or compliance typically owns the framework, but it requires input from CX/design, engineering, data protection, security, legal, and the business owner accountable for the customer journey.
How often should it be reviewed?
Before launch, after any material change (new tools, new model version, expanded scope or autonomy), and on a regular scheduled cycle — because model behaviour and threats both evolve over time.
Conclusion
Self-service AI agents earn their keep by acting autonomously — and that autonomy is precisely why they need a rigorous risk assessment. Cover all six domains: safety and accuracy, security, data and privacy, compliance, operational, and reputational. For each risk, score inherent and residual risk, name the control, assign an owner, and define how you'll monitor it live.
The goal isn't to eliminate risk — that's impossible with any AI system. The goal is to understand it, control it, monitor it, and accept it consciously at the right level of the business. Do that, and you can deploy AI agents that are not just impressive, but genuinely trustworthy.
Related reading: the new AI agent metrics to track in production, how to build a secure ID&V module, and whether your IVR is AI-ready.