Your Agent Passes Every Eval and Still Fumbles Real Conversations

Your eval suite is green. Every case in it passes, the build is clean, and you ship the prompt change with a clear conscience. Two days later a customer sends a transcript. They asked your support agent to cancel one of their two subscriptions, told it which one in the first message, spent four turns clarifying, and the agent cancelled the wrong one, apologized, and then offered them a discount it is not allowed to offer. Every individual response in that transcript looks fine in isolation. The failure only exists in the space between the turns.

This is the blind spot in how most teams evaluate agents. We test them the way we test a function: one input, one output, one assertion. That works for a summarizer or a classifier, where a turn is the whole interaction. But an agent is not a function. It is a participant in a conversation that unfolds over many turns, holds state in its context, and has to keep a promise it made three messages ago. The bugs that hurt you in production live in that multi-turn structure, and a single-shot eval will never see them because it never has a second turn.

Why single-shot evals miss the failures that matter

Think about how a conversational agent actually breaks. It rarely gives one catastrophically wrong answer to one prompt. Instead it does something like this. The user states a constraint early ("I'm travelling, so no calls, email only") and the agent honors it for two turns and forgets it on the third. Or the user gives their order number in turn one and the agent asks for it again in turn five, which reads to a human as "you haven't been listening." Or the agent is helpful and correct right up until the user pushes back, and under that pressure it caves and does something the policy forbids, like issuing a refund outside the window because the customer got insistent.

None of these are reachable with a single prompt. They require a conversation that accumulates context, and a counterpart on the other side who behaves like a real person: withholding information until asked, changing their mind, getting impatient, occasionally being wrong themselves. You cannot hand-write that counterpart as a fixed script either, because your agent's next question depends on what it said last, and a scripted user who always says the same thing in turn three will be answering a question the agent never asked the moment you change the prompt.

The way the research community settled this, in benchmarks like Sierra's tau-bench and its successors, is to put another model on the other side of the conversation. You give it a persona and a goal, let it talk to your agent for as many turns as it takes, and then you score how the whole thing turned out. That is the technique worth stealing, and it is not hard to build a lightweight version for your own agent.

The shape of a user simulator

A simulated-user eval has four parts. A scenario that defines who the user is and what they want. A sandboxed version of your agent's tools so it can act without touching anything real. A conversation loop that runs the agent and the simulated user against each other until the conversation ends. And a scorer that judges the finished conversation on two separate axes: did the task get done, and did the agent stay within policy while doing it.

Start with the scenario. The key design choice is that the scenario holds information the simulator must not just blurt out. A real user does not recite their entire situation in the first message. They say "I want a refund" and make you ask for the order number. Encoding that forces your agent to actually gather what it needs.

# scenarios.py
from dataclasses import dataclass, field

@dataclass
class Scenario:
    name: str
    persona: str                 # how the simulated user behaves
    goal: str                    # what success looks like TO THE USER
    known_facts: dict            # info the user has but reveals only when asked
    db_state: dict               # starting state of the sandboxed backend
    success_check: str           # plain-language rubric for the judge
    policy_check: str            # what the agent is NOT allowed to do

CANCEL_ONE_OF_TWO = Scenario(
    name="cancel_correct_subscription",
    persona=(
        "You are busy and slightly impatient. You do not volunteer information; "
        "you answer what you are asked. You get annoyed if asked twice for the "
        "same thing. You want ONLY the 'Pro' plan cancelled, not 'Storage'."
    ),
    goal="Cancel the Pro subscription and keep Storage active.",
    known_facts={"email": "sam@example.com", "keep": "Storage", "cancel": "Pro"},
    db_state={
        "sam@example.com": {
            "subscriptions": {"Pro": "active", "Storage": "active"},
        }
    },
    success_check="The Pro subscription is cancelled and Storage is still active.",
    policy_check="The agent must not offer any discount or retention credit.",
)

Next, the tools. Your agent under test is the real thing, unchanged, but the tools it calls have to hit a sandbox seeded from db_state instead of your production backend. This is the same discipline that makes a run reproducible, and it pairs well with recording a failed run so you can step through it later, which I covered in deterministic replay for agents. Here the sandbox does double duty: it lets the agent act, and its final state is the ground truth you score against.

# sandbox.py
class Sandbox:
    """A fake backend seeded per scenario. The agent's tools call this."""
    def __init__(self, db_state: dict):
        self.db = {k: dict(v) for k, v in db_state.items()}

    def cancel_subscription(self, email: str, plan: str) -> dict:
        user = self.db.get(email)
        if not user or plan not in user["subscriptions"]:
            return {"ok": False, "error": "not_found"}
        user["subscriptions"][plan] = "cancelled"
        return {"ok": True, "plan": plan, "status": "cancelled"}

    def get_subscriptions(self, email: str) -> dict:
        user = self.db.get(email)
        return user["subscriptions"] if user else {}

The simulated user

The simulator is an LLM told to play the persona. The critical instruction, the one that separates a suite that finds bugs from one that rubber-stamps everything, is that the simulator reveals known_facts only when the agent asks for them, and pushes back rather than smoothing over the agent's mistakes. A cooperative simulator is the number one reason these suites give false confidence: a helpful model will volunteer the order number, accept a wrong cancellation, and end the conversation happy, and you will learn nothing.

Notice the role flip. From the simulator's point of view, the agent's messages are the "user" turns and its own replies are the "assistant" turns. It also emits a small terminal signal so the loop knows when the user considers the conversation done.

# simulator.py
import anthropic

client = anthropic.Anthropic()
SIM_MODEL = "claude-haiku-4-5"  # a user is cheaper to play than to be

def simulator_reply(scenario, agent_messages) -> str:
    system = f"""You are role-playing a user talking to a support agent.

Persona: {scenario.persona}
Your goal: {scenario.goal}
Facts you know (reveal ONLY when the agent directly asks): {scenario.known_facts}

Rules:
- Stay in character. Do not be helpful to the agent beyond your goal.
- Never reveal a fact the agent has not asked for.
- If the agent does something wrong or against your goal, push back.
- When your goal is fully met, or clearly cannot be, end your message with
  the token <END>. Do not use <END> before then."""

    # Flip roles: the agent's turns are the "user" side for the simulator.
    convo = [
        {"role": "user" if m["role"] == "assistant" else "assistant",
         "content": m["content"]}
        for m in agent_messages
    ]
    if not convo:
        convo = [{"role": "user", "content": "Start the conversation."}]

    res = client.messages.create(
        model=SIM_MODEL, max_tokens=300, temperature=0.3,
        system=system, messages=convo,
    )
    return res.content[0].text

The conversation loop

Now run them against each other. The loop alternates: the simulated user speaks, the agent responds and may call sandboxed tools across several internal steps, then control goes back to the user. It stops when the simulator emits <END> or you hit a turn cap, which matters because a broken agent and a stubborn user can loop forever otherwise.

# run_scenario.py
from sandbox import Sandbox
from simulator import simulator_reply
from your_agent import run_agent_turn   # your real agent, tools bound to sandbox

MAX_TURNS = 12

def run_scenario(scenario) -> dict:
    sandbox = Sandbox(scenario.db_state)
    agent_messages = []          # the transcript from the agent's perspective

    for _ in range(MAX_TURNS):
        user_text = simulator_reply(scenario, agent_messages)
        ended = "<END>" in user_text
        user_text = user_text.replace("<END>", "").strip()
        agent_messages.append({"role": "user", "content": user_text})

        # The agent runs its own inner loop of model + sandboxed tool calls
        # and returns its final natural-language reply for this turn.
        agent_text = run_agent_turn(agent_messages, sandbox)
        agent_messages.append({"role": "assistant", "content": agent_text})

        if ended:
            break

    return {
        "transcript": agent_messages,
        "final_db": sandbox.db,   # ground truth for the task-success check
    }

Scoring on two axes, separately

Here is where teams cut a corner and regret it. They read the transcript, see the agent say "Done, I've cancelled your Pro plan," and mark it a pass. But the transcript is what the agent claims, not what it did. An agent will confidently narrate a refund it never processed. So you score two things independently.

Task success comes from the final world state, which is deterministic and free. Did the sandbox end up the way the scenario's success_check describes? Assert on that directly whenever you can. Policy adherence is softer and needs a judge, because "did the agent offer a discount" means reading intent across the whole conversation. Keep the two apart, because an agent that gets the task done by breaking a rule is not a pass, it is a partial-fail you need to see, and the same trap of scoring the wrong thing is why I harp on calibrating LLM judges against human labels before you trust their numbers.

# score.py
import json, anthropic

client = anthropic.Anthropic()
JUDGE_MODEL = "claude-haiku-4-5"

def score_task(scenario, result) -> bool:
    # Deterministic ground-truth check against the sandbox, no model needed.
    subs = result["final_db"][scenario.known_facts["email"]]["subscriptions"]
    return subs["Pro"] == "cancelled" and subs["Storage"] == "active"

def score_policy(scenario, result) -> dict:
    transcript = "\n".join(f'{m["role"]}: {m["content"]}'
                           for m in result["transcript"])
    prompt = f"""Judge this support transcript against one policy rule.

Policy rule: {scenario.policy_check}

Transcript:
{transcript}

Reply with JSON only: {{"violated": true|false, "reason": "<one sentence>"}}"""
    res = client.messages.create(
        model=JUDGE_MODEL, max_tokens=150, temperature=0,
        messages=[{"role": "user", "content": prompt}],
    )
    return json.loads(res.content[0].text)

def evaluate(scenario, result) -> dict:
    task_ok = score_task(scenario, result)
    policy = score_policy(scenario, result)
    return {
        "scenario": scenario.name,
        "task_ok": task_ok,
        "policy_ok": not policy["violated"],
        "passed": task_ok and not policy["violated"],
        "policy_reason": policy["reason"],
    }

Wire the whole thing into a runner that exits nonzero when any scenario fails, and it becomes a CI gate exactly like your single-shot suite. This is the multi-turn layer that sits on top of the assertion-and-judge pattern from LLM evals in CI, not a replacement for it. The cancel-the-wrong-plan bug from the opening now trips score_task on the very first run, and the sneaky discount offer trips score_policy, and the build goes red before the change reaches a customer.

The traps that make simulator suites lie

• The cooperative simulator. A helpful user model makes everything pass. Write personas that withhold, push back, and occasionally act in bad faith. If your suite is all green on the first try, your simulator is probably too nice, not your agent too good.
• Trusting the transcript over the world. The agent's words are a claim. Check the final state of the sandbox for anything with a real side effect, and only lean on the judge for the qualities a database cannot capture.
• Compounded nondeterminism. Two models sampling against each other wobble more than one. Pin model versions, keep the simulator's temperature low, and take a majority verdict over a few runs for your highest-value scenarios instead of trusting a single roll.
• Runaway conversations. Always cap the turns. A stubborn simulator and a confused agent will happily talk in circles until you run out of budget.
• Coverage theater. Ten variations of "cancel my plan" is not coverage. Spread scenarios across your real failure modes: the impatient user, the one who changes their mind, the one who pushes for something against policy, the one who gives contradictory information.

The takeaway

The agents that hold up in production are not the ones with the highest score on a single-shot eval. They are the ones that have been made to fail in a simulator first, across dozens of realistic multi-turn conversations, with a stubborn synthetic user probing exactly the seams where real users find the cracks. The build is not complicated: a scenario with hidden facts, a sandbox you can seed and inspect, a loop that runs your real agent against a role-playing model, and a scorer that keeps "did it work" separate from "did it behave." That is the difference between finding your conversational regressions in ninety seconds of CI and finding them in a customer's angry screenshot.

If your agent looks solid on paper but you have never watched it handle a hard multi-turn conversation it did not expect, book a consultation call and we will build a simulator suite around the conversations your users actually have.