Measuring Chatbot Quality: Resolution, Escalation, and Repeat Contact
How to define and compare chatbot containment, resolution, satisfaction, fallback, escalation, and repeat-contact metrics.
Conversation volume and session length show how a bot is used, but they do not establish whether it resolves requests. Include repeat contacts, satisfaction, and escalation outcomes in the review.
Define a resolution signal before choosing dashboard metrics.
The Problem with Surface Metrics
Session count tells you the bot was used. It does not tell you whether anything was resolved. Average response time tells you the bot is fast. It does not tell you whether the fast answer was the right one.
A bot that answers quickly and incorrectly is creating work for your human team, not reducing it. You need metrics that distinguish between "the bot responded" and "the bot resolved."
Core Chatbot Quality Metrics
Containment Rate
Containment rate is the percentage of conversations that are fully handled by the bot without human involvement. It is the headline number that most leadership teams will ask for, and it is useful — but only in combination with satisfaction data.
Formula: (Conversations without escalation / Total conversations) × 100
A containment rate of 70% means nothing if half of those "contained" users immediately filed a support email because the bot answer was wrong. Track containment rate alongside CSAT or repeat contact rate to know whether contained conversations are actually resolved.
Target setting: Use a baseline from your own query mix and review answer quality alongside containment. An appropriate target depends on the bot's scope and the cases it must escalate.
Resolution Rate
Resolution rate is stricter than containment rate. It measures the percentage of conversations where the user's problem was genuinely resolved, typically confirmed by a positive rating, no repeat contact within 48 hours, or a behavioral signal like completing a downstream action.
Formula: (Verified resolutions / Total conversations) × 100
When resolution requires evidence beyond containment, compare the two using consistent denominators. Investigate contained conversations that lack a resolution signal.
Fallback Rate
Fallback rate measures how often the bot failed to understand the user's input at all and responded with a generic error or "I don't understand." This is a direct signal of model or coverage quality — not user behavior.
Formula: (Fallback responses / Total bot turns) × 100
Target setting: Track fallback by query category and compare it with the baseline. Set thresholds based on the consequence and frequency of missed queries.
High fallback rates point to one of two problems: the intent classification is undertrained, or there are query categories in your user base that have no coverage in the knowledge base. Both are solvable with different interventions.
CSAT (Customer Satisfaction Score)
CSAT for chatbots is usually collected via a post-conversation rating prompt. It measures whether the user felt their experience was positive, regardless of whether an objective resolution occurred.
Formula: (Positive ratings / Total ratings collected) × 100
Measure CSAT separately for bot-resolved and human-escalated conversations. A blended score can hide differences, though separate scores still need query context and response-rate checks.
Compare bot and human CSAT within similar query categories and account for response rates. Different scores alone do not identify the cause.
Escalation Rate
Escalation rate is the inverse of containment rate, but it deserves its own attention because the reason for escalation matters.
Break your escalation data into categories:
- User-requested escalation ("I want a human")
- Hard trigger escalation (fraud keyword, urgent topic)
- Bot-initiated escalation (low confidence score, repeated failed turns)
- Unknown / no trigger logged
A high rate of unknown escalations usually means your escalation logic is not logging trigger reasons — fix the instrumentation first. A high rate of user-requested escalations may mean the bot is answering but not answering in a way that builds trust.
Repeat Contact Rate
Repeat contact rate measures the percentage of users who contacted support again within a defined window (commonly 48-72 hours) after a bot-resolved conversation.
This is the clearest proxy for false resolution — cases where the bot appeared to contain the query but did not actually solve the problem.
Formula: (Users who contacted again within N hours after bot resolution / Total bot-resolved conversations) × 100
Review repeat contacts to distinguish unresolved requests from unrelated new issues. Set an investigation threshold based on your baseline.
How to Read These Metrics Together
No single metric tells the full story. Here is how to interpret combinations:
- High containment, high CSAT, low repeat contact: promising signals; check query mix and whether unresolved users are missing from the data
- High containment, low CSAT: review transcripts for unanswered questions, unhelpful responses, or obstacles to escalation
- Low containment, low fallback rate: investigate escalation policy and confidence thresholds
- Low fallback rate, high repeat contact: inspect answer accuracy, completeness, and whether the user needed a later step
Building a Measurement Infrastructure
Most chatbot platforms expose these metrics in some form, but rarely in the combinations that matter. Plan for:
- A weekly review of fallback logs to identify new coverage gaps
- A monthly review of escalation trigger breakdown to catch miscalibrated rules
- A quarterly audit of articles linked to high-repeat-contact resolutions to find accuracy issues
Assign an owner to review these metrics and connect recurring failures to specific content or workflow changes.