How to Review Outcomes and Guardrail Insights

Supervisor’s Outcomes tab answers a question the old single resolution-rate number couldn’t be trusted to answer honestly: did the conversation actually get resolved, or did it just end? Instead of asking a model to judge “resolved vs. unresolved” from scratch, the transcript itself now decides the category and how confident that category is. The model (the judge) is only called in for the one thing the transcript can’t tell you on its own: what the customer’s last message was actually doing.

This article covers the Outcomes tab, the confidence tiers behind each row, the trend chart, how to filter the closed-conversation list, the tenant opt-in for mining handover replies into quick drafts, and the superadmin Guardrail Insights page that sits alongside it.

Where to find it

• Outcomes is a tab in Supervisor, next to the other conversation views.
• Guardrail Insights is a superadmin-only page, separate from Supervisor, that looks across every tenant’s guardrail activity.
• The handover mining opt-in lives in the Supervisor header, not inside the Outcomes tab itself — it’s a tenant-level setting, not a view filter.

How conversations are classified

Every closed conversation is assigned one outcome category. The category is derived deterministically from transcript facts — how the conversation ended, whether the customer replied again, whether a handover happened — not from a model’s overall impression:

• Confirmed resolved — The customer said, in substance, that their issue was handled.
• Assumed resolved — The conversation ended without complaint and without a follow-up question — most likely resolved, but nobody said so.
• Unresolved — The conversation ended with an open question, complaint, or unmet request still on the table.
• Abandoned — The customer stopped responding mid-conversation, before anything was resolved or explicitly left open.
• Unassessable — The judge call needed to classify the last customer message failed. The row stays visible rather than being silently dropped.

The important design choice here: confirmed and assumed are never added together into one headline resolution rate. Showing a single optimistic number was the exact credibility problem this taxonomy replaces, so the Outcomes tab always shows both counts side by side and lets you read them separately.

Rows that get reopened or superseded — a later close on the same conversation, or a reopen inside the 48-hour reopen window — aren’t deleted from the list. They stay, marked as retracted, so the historical counts don’t quietly change out from under you.

Conversations closed before this feature shipped don’t have a real confidence measurement, so they’re shown as assumed resolved at an unknown tier rather than being assigned a confidence they were never actually measured for.

Confidence tiers

Alongside the category, each row carries a confidence tier. The tier reflects how strong the evidence behind the classification is — a short, ambiguous last message earns a lower tier than a substantive one that clearly confirms or disputes resolution. Because the tier is computed from the same deterministic transcript signals as the category, it doesn’t change from one run to the next for the same conversation.

The judge is invoked only when there’s a last customer message to interpret at all — silent exits (the majority of abandoned conversations) never need a model call, and an unassessable row means that one judge call specifically failed, not that the whole classification pipeline broke.

Reading the trend chart

The trend chart plots the resolution picture over time. The most recent 48 hours of the chart is drawn dashed rather than solid — conversations that closed that recently can still reopen, so that segment of the trend is provisional and worth re-checking later rather than treated as final.

An info icon sits on the chart (and on every other widget in the tab — the KPI cards, the coverage note, the confidence filter, and the list) so you can hover for a short explanation of exactly what that widget is counting, without leaving the page.

Filtering the closed-conversation list

The closed-conversation list below the chart can be filtered by confidence tier, and each row shows:

• The outcome category (confirmed resolved, assumed resolved, unresolved, abandoned, unassessable)
• The confidence tier
• Whether the row is retracted (reopened or superseded after close)
• The conversation’s current status — closed conversations can live in either the done or archived inbox, and the list reflects which one so a deep link opens the right place
• The judge model used for that row’s classification, where one was called

Use the confidence filter when you want to sanity-check the lower tiers specifically — that’s where “assumed” resolutions and thin evidence live, and it’s the fastest way to spot conversations worth a manual look.

Handover knowledge mining opt-in

Handover mining reads what your own team wrote to a customer during a human handover and turns the best of those replies into quick-draft suggestions. Because it reads real teammate replies, it does not start on its own just because Supervisor is enabled for your tenant — it requires an explicit opt-in, toggled from the Supervisor header, in addition to the platform-side rollout flag.

A few things worth knowing about the opt-in:

• It’s a tenant-level consent switch, separate from any other Supervisor setting — toggling it doesn’t touch anything else.
• Mining only runs once your tenant has opted in and the platform-side flag for this capability is enabled — if your tenant hasn’t opted in, mining is skipped before anything is read, at no cost.
• Not every closed handover produces a quick draft. There’s a deliberate screening step before a suggestion is drafted at all, so the queue only ever contains items worth a reviewer’s time; conversations that don’t clear that bar are skipped.

Quick drafts in the review queue

When a handover reply clears the mining screen, it shows up in the review queue as a quick-draft suggestion — the same place quick drafts a person explicitly asked for appear. Mined suggestions carry a badge so a reviewer can tell at a glance that a suggestion came from something a human teammate actually said to a customer, rather than being requested directly.

Guardrail Insights (superadmin)

Separate from Supervisor, Guardrail Insights is a superadmin page that gives a cross-tenant view of every guardrail trigger — regenerations, forced fallbacks, forced handovers, and the other guardrail checks — in one normalized stream instead of scattered across different logs.

The page includes:

• A KPI row: total triggers, forced fallbacks, regeneration share, and triggers per 1,000 turns.
• A stacked history chart, bucketed by day — or by hour, automatically, when the selected date range is 48 hours or less.
• Filters for guardrail, guardrail family, outcome, tenant, and date range. The guardrail filter accepts multiple values at once.
• A paginated events table underneath the chart and filters.
• A floating detail pane for a selected event, showing the raw detail payload, with a drill-down link that opens the underlying tenant conversation in a new tab.

Because the drill-down needs to land on the right inbox tab, each event carries the conversation’s real lifecycle status — a closed conversation won’t open into an “active” list and show up empty.

If you toggle the guardrail or date filters and see gaps or ragged bars in the chart, that’s expected for very sparse periods — the chart fills in buckets with no events client-side so the stacked chart stays readable, rather than silently dropping days that had nothing to show.

Summary

• The Outcomes tab replaces a single model-judged resolution rate with a deterministic taxonomy: confirmed resolved, assumed resolved, unresolved, abandoned, and unassessable — confirmed and assumed are always shown separately, never summed.
• Each row carries a confidence tier; the trailing 48 hours of the trend chart is dashed because those conversations can still reopen.
• Handover knowledge mining requires an explicit tenant opt-in from the Supervisor header before any handover reply is read, and its quick-draft suggestions carry a badge in the review queue.
• Guardrail Insights, under Superadmin, gives a cross-tenant, filterable view of every guardrail trigger with drill-down into the source conversation.