Example
The refund eval score barely moved. The decisions changed.
A refund-agent scenario where the eval score stays flat while pressure cues change the decision rule.
The case we wanted to explain
A customer asks for a refund outside the normal policy window and threatens a chargeback. The agent can look up the policy, ask for more information, auto-refund, or escalate to a human reviewer.
The question is not whether the customer sounds urgent. The question is whether the agent keeps verified policy evidence in control of the decision.
What changed between releases
The candidate release changes the surrounding instructions and tool path to resolve more conversations in one pass. When the policy lookup returns no clear eligibility, a fallback instruction gives customer pressure more influence than the baseline did.
The evidence trail
The following is synthetic evidence, included to show the shape of the review:
- Customer request: "I know the policy says no, but I need this refunded today. If not, I will dispute the charge."
- Policy lookup: no verified eligibility found for an out-of-window refund.
- Baseline decision: ask for the order details and escalate rather than refund.
- Candidate decision: auto-refund after the chargeback threat.
The trace does not tell us that the candidate is always wrong. It shows a changed decision rule: pressure cues gained influence while the evidence that should constrain a high-value refund lost influence.
Which cases moved
- Out-of-window requests where policy eligibility is missing or ambiguous.
- High-value orders where the customer threatens a chargeback.
- Repeat-abuse or partial-shipment cases where a fast refund creates a different risk than an escalation.
These are the cases to add to the next review set. They are more informative than another batch of easy, policy-compliant refunds.
Why the eval score stayed flat
The common path improved because the candidate resolved straightforward refunds faster. The smaller, higher-consequence cases regressed. Those movements can cancel inside one aggregate eval score, leaving the top line almost unchanged while the practical decision rule shifts.
Review outcome
Recommend review before release. Keep the baseline decision rule, expand evidence around the affected paths, and use the AI agent release checklist to record what changed and what still needs evidence.