Flag and Tune Bad Responses

The Tuning flow is how you teach an Agent Stack to behave better without writing prompt-engineering by hand. Flag a bad reply, describe what was wrong, and the AI proposes a concrete edit — a rewrite of one agent’s Instructions or Responsibility — that you apply with one click. Undo it if it doesn’t help.

When to use this

• You spotted a reply in the Testing sandbox that’s wrong, off-tone, or missing info.
• A live conversation went sideways, and you can reproduce it in the sandbox to fix the underlying behavior so it doesn’t happen again.
• You’re iterating on a new specialist and want quick guided improvements rather than editing prompts blindly.

Before you start

• The stack must exist with at least one specialist.
• You need a bad reply to flag, and you have to produce it in the Testing tab. Tuning runs on the test transcript only. The flag on a message in a real conversation is a different control — a self-learning annotation, which feeds the pattern-based Self-Learning pipeline instead of opening a tuning session.
• The settings.agent_stacks permission, which is what the tuning routes are gated on. That is enough for specialist-level behavior; a proposal that modifies a system agent — the router or the orchestrator — is refused unless you are a superadmin.

Steps

1. Open Settings → Agent Stacks → [stack] → Testing.
2. Generate a conversation that produces the bad reply. If you want more room while testing, click Full screen in the Testing tab. The same transcript and active tuning session follow you there.
3. Click the flag icon on the bad reply. A message-level feedback popover opens for that specific reply, asking “What is wrong with this reply?”.
4. In the popover, describe what went wrong in short, plain language. Examples:
   “Should have asked for the order ID before promising a refund.”
   “Routed to Tech instead of Billing.”
   “Tone is too casual for an enterprise customer.”
5. Click Analyze to submit the feedback and start analysis. The Tune agent pane opens on the right for the active session, keeping the flagged message, transcript, and tuning work together.
6. The AI may ask clarifying questions if your feedback is ambiguous. Answer them in the Tune agent pane.
7. Once the AI is confident in the diagnosis, it produces a proposal — a concrete edit to exactly one field on one agent, either its Instructions (the system prompt) or its Responsibility (the short description). Common proposals:
   Update a specialist’s Instructions with a new rule.
   Rewrite the orchestrator’s Instructions to route differently (requires superadmin access, because the orchestrator is a system agent).
   Adjust a specialist’s Responsibility (changing what it is meant to cover).
8. Read the proposal. It shows the Current and Proposed text side by side plus the Reasoning, with a collapsible Root Cause Analysis above it. Click Apply to commit it, Revise to ask for a different version in your own words, or Dismiss to reject it.
9. After applying, you can re-test the same scenario in the same pane to confirm the fix worked. The applied-change card carries its own Undo button, and the Changes menu in the Testing tab header keeps every change you applied this session.

Reverting a change

The applied proposal card shows Change Applied with its reference number and an Undo button. The same changes are also listed behind the Changes button in the Testing tab header — a popover headed “Changes this session”, one Undo per row. Undoing restores the previous text of that field; the row then reads Reverted. No manual prompt editing required.

Verify it worked

• Send the same message that produced the bad reply originally. The new reply should match the corrected behavior.
• Send a few variations to make sure you didn’t over-correct (now the AI is too cautious about refunds, etc.).
• Open the Changes menu in the Testing tab header to confirm the change persisted to the underlying agent; expand a row to read the analysis and the text it replaced.

What feeds Self-Learning

Tuning and Self-Learning are two separate paths to the same kind of edit. Tuning acts on one flagged reply in the sandbox and applies immediately. Self-Learning collects annotations left on real conversations and proposes improvements from patterns across many of them. Tuning is for the immediate fix; Self-Learning is for the longer-term pattern.

Troubleshooting

• Symptom: The Tune agent pane doesn’t show any proposal — just keeps asking clarifying questions. Fix: your feedback is too abstract. Be specific: not “the reply was wrong” but “the reply confirmed a refund without asking for the order ID.”
• Symptom: The proposal would change the wrong specialist. Fix: dismiss it. Re-flag the same message and clarify which specialist’s behavior is at fault.
• Symptom: After applying, the bad behavior is back. Fix: the applied change might have been overwritten by a later edit (yours or someone else’s). Check the Change History tab on the stack to see the order of recent changes. Re-apply the tuning if needed.
• Symptom: “Apply” button is greyed out. Fix: the proposal hasn’t fully generated yet, the AI is waiting for a clarifying answer, or the proposal targets the router and you don’t have superadmin permissions — in that case the panel says so above the buttons. A proposal targeting the orchestrator is not greyed out, but Apply is refused with a permission error for the same reason. Either way, ask a superadmin to review and apply it.

See also

• Test an Agent Stack
• What is Self-Learning?
• What are Agent Stacks?