DIALOG_MODE.md

description	Evaluate coding agents in conversation, not one-shot — Coder Eval's dialog mode drives a simulated user with a persona and a goal, so you can measure clarifying questions, requirement discovery, and multi-turn recovery.

Dialog Mode: multi-turn user simulation

Most evaluation harnesses hand an agent one perfect prompt and grade the artifact. Real users don't write perfect prompts. They ask for something vague, answer questions, change their mind, and hold back requirements until asked.

Dialog mode replaces the single-shot loop with a conversation between the coding agent and a simulated user — a second LLM driven by a persona and a goal. It is the only way to measure the behaviors that only exist in conversation. To replay a fixed list of authored questions instead of simulating a user, see Scripted dialogs.

When to use it

Reach for dialog mode when the thing you want to score is conversational:

• Does the agent ask? A good agent asks a clarifying question instead of guessing when the request is underspecified. Give the simulator a goal that withholds a requirement and see whether the agent surfaces it.
• Requirement discovery. Score whether the agent eventually extracts everything it needs across several turns, not whether it one-shots a fully-specified prompt.
• Multi-turn recovery. The agent goes down the wrong path, the user corrects it — does it recover, or double down?
• Prompt robustness. A persona that is terse, impatient, or non-technical is a much harsher test than your hand-tuned initial_prompt.

Stay in single-shot mode when the task is genuinely fire-and-forget. Dialog mode multiplies cost and adds variance; don't pay for it when the extra turns measure nothing.

How it works

Add an enabled simulation: block to any task and the orchestrator swaps its iteration loop for a dialog driver:

simulation:
  enabled: true
  persona: |
    You are a user with one specific request. You give the request only when
    asked, and then you wait for the agent to do it.
  goal: |
    Get the agent to echo your string back verbatim. Reveal the string only
    when the agent asks what you want.
  max_turns: 4

The mechanics:

• The task's initial_prompt is the user's opening message. The simulator takes over from turn two, seeing the agent's last reply and producing the next user utterance.
• initial_prompt is optional in dialog mode. Omit it and the simulator writes the opener too, from its persona and goal — closer to a cold-start user.
• Each exchange is one user message plus the agent's full response (the agent may make many tool calls inside a single exchange).
• The simulator is a tools-disabled Claude Code agent with allowed_tools: [], an explicit deny-list, and no plugins or settings sources. It is pure text-in / text-out, and it cannot see the sandbox — no files, no terminal, no agent reasoning. Only what the agent writes in the chat.
• The simulator resolves its own ApiRoute independently of checker_context.api_route (that override is judge-only — see Checker Context) — same resolution as the coding agent's own route (--backend direct / --backend bedrock), including the pin to a constant Claude backend when the agent itself runs on an open-weight LiteLLM route. The model is separately pinned by simulation.model (see Simulation), not inherited from the route — so an A/B varying the subject model doesn't silently vary the simulated user too.

Agent-kind constraint. The subject agent can be any registered kind — the dialog driver only calls the agent's communicate(), so Codex and plugin agents work. The simulator, however, is always a Claude Code agent. A dialog run therefore needs the claude CLI on PATH and working Anthropic or Bedrock credentials even when the agent under test is not Claude. A scripted dialog has no simulator and needs neither.

Early stop is not available here. Criterion-level stop_early: arming is single-shot-only; an armed criterion alongside simulation.enabled is a hard error at resolution time, not a silent no-op (disarm with the run_limits.stop_early: false kill switch to run anyway). Use stop_on_criteria_pass (below) for the dialog equivalent.

Every field's default and constraint lives in one place — the Task Definition Guide's simulation section. This page deliberately does not restate them.

Scripted dialogs

Sometimes the conversation itself is the benchmark: a fixed list of authored questions, asked in order within one session, whose answers you read afterwards. A simulated user would paraphrase them and add an LLM call per turn. Set turns instead of persona and goal, and the dialog replays the list verbatim:

task_id: onboarding-questions
description: Ask the same three questions in one session and keep each answer.
agent:
  type: codex                          # Any agent: a script needs no `claude` CLI or credentials of its own.
working_directory_prefix: false        # Send each turn exactly as written.
simulation:
  enabled: true
  turns:
    - "What does the knowledge base say about onboarding?"
    - "And about reimbursements?"
    - "Summarize both answers in two sentences."
  max_turns: 3                         # Must be at least len(turns).
success_criteria:
  - type: llm_judge
    description: "Each answer is grounded in the knowledge base"
    prompt: "Read the DIALOG block and grade whether every answer is grounded in the knowledge base."
    include_dialog: true

What changes against a simulated dialog:

• One session, no simulator. Every turn goes to the same agent instance, so the agent keeps its context from turn to turn. No LLM plays the user: no claude CLI, no credentials, no simulator route, and simulation.simulator_model is null with zero simulator tokens.
• turns[0] is the first prompt. The task sets no initial_prompt or initial_prompt_file, so there is one source of truth. Turns are sent byte for byte when working_directory_prefix: false; by default each still carries the Your working directory is: … line in front.
• The script ends the dialog. After the last turn the dialog stops with stop_reason: script_exhausted, whatever the criteria say. Nothing in a script reads a criteria result, so stop_on_criteria_pass and check_criteria: every_turn / both make no sense for it, and neither do persona, goal, constraints, model or stop_token: each must keep its default, or the task is rejected at load time. max_turns is derived from the script length (1–100 turns), so an authored script needs no second dialog-length setting. Variant prompt overrides and mutations are rejected; replace simulation.turns to change the script.
• A failed turn ends the dialog. A turn that still fails after its retries, exhausts run_limits.max_turns or breaches a run_limits budget stops the script. Later turns are not sent, and the turns that ran stay on the record.
• It runs without grading. coder-eval execute accepts a scripted dialog and finalizes it NOT_GRADED; coder-eval evaluate <run_dir> grades it later. Criteria are checked once, after the last turn.

Each agent attempt is an entry in iterations of task.json (see TurnRecord): that turn's user_input (the prompt as the agent received it), agent_output, commands, token_usage and timings, with the authored turn as messages[0]. agent_output is the agent's own rendering of its reply: Claude Code's is a tagged transcript, and its SDK result text is also in result_summary.result. n_trials: N repeats the whole script N times, each in its own session, to measure variance.

Retries share the authored iteration number; failed attempts and their costs stay in the raw record. The driver can also preserve an unconsumed simulator utterance as record_kind: standalone_user. That row is telemetry, not an agent answer. Stop messages and rows already covered by an agent attempt are omitted from grading. An unanswered question with no agent record remains in the dialog as failed input evidence, with no canonical answer or latest agent output. Collected agent records use record_kind: agent, including genuine empty completed answers. Older records keep unknown provenance rather than acquiring an inferred kind.

Designing the persona and goal

persona is who is talking; goal is what they want. Keep them separate — a persona that smuggles in the requirements defeats the point.

The most useful pattern is withholding. State the full goal for the simulator, then instruct it not to volunteer part of it:

persona: |
  A non-technical business analyst who knows the outcome they want but not how
  automation works. Mildly impatient, answers questions directly.
goal: |
  Build a flow that reads invoice PDFs from an Outlook folder, extracts
  vendor/amount/date, and posts the result to Google Sheets.
  Do NOT volunteer the Google Sheets requirement unless the agent asks where
  the output should go.
constraints:
  - "Do not paste code — you cannot read code."
  - "If the agent goes silent for two turns, ask 'are you still there?'."

Now the task measures something a single-shot prompt cannot: whether the agent asks about the output destination at all. Pair it with a criterion that checks the Sheets integration exists and you have a clean signal.

constraints are behavioral rules, not more requirements. Good ones bound how the simulator speaks ("no code", "one question at a time", "reveal X only if asked"). Resist writing constraints that quietly hand the agent the answer.

How a dialog ends

After each exchange the driver evaluates the stop conditions in this order, first match wins:

1. run_limits breach (run_limit_exceeded) — a subject-agent budget cap tripped. The end-of-dialog criteria still run for partial credit, then the task aborts.
2. stop_on_criteria_pass (criteria_passed) — every success criterion passes. Requires per-turn checking (check_criteria: every_turn or both); pairing it with the default end_of_dialog is rejected at load time, since there would be nothing to check against.
3. Script exhausted (script_exhausted) — a scripted dialog has had its last turn answered. It takes the place of max_turns there.
4. max_turns (max_turns) — the hard cap on exchanges.
5. max_total_tokens (budget) — the dialog-wide budget across simulator and agent. The dialog ends and the task is still scored — unlike run_limits.max_total_tokens, which covers the subject agent only and aborts.
6. run_limits.max_turns (agent_max_turns): the agent used up its own model-call cap inside one exchange. That cap restarts on every exchange, so this reason means one exchange ran long, not that the dialog ran out of exchanges.
7. stop_token (stop_token) — only if none of the above fired is the simulator asked for another message; the sentinel token in that fresh utterance ends the dialog. This is the workhorse in practice — the simulator decides, in character, that it got what it wanted — but it is evaluated last, so a turn that trips max_turns or the budget never gets the chance to produce it.

A simulator call that raises ends the dialog with error (and increments simulation.simulator_failures). The reason is recorded as simulation.stop_reason on the result, so you can tell a conversation that finished from one that merely ran out of turns.

check_criteria: every_turn multiplies criterion cost. Criteria are re-evaluated after every exchange, so an llm_judge or agent_judge criterion in a 12-turn dialog runs twelve times. That is often more expensive than the dialog itself. Use end_of_dialog (the default) unless you genuinely need early stop or a per-turn record; both gives you the per-turn trace and an authoritative final check.

Trials and variance

A single dialog is one sample from a stochastic process — the simulator improvises as much as the agent does. n_trials runs the conversation N independent times per (task, variant):

simulation:
  enabled: true
  n_trials: 5

Each trial is fanned out as its own unit of work with its own sandbox, its own run directory (runs/<ts>/<variant_id>/<task_id>/<NN>/, zero-padded index), and its own task.json. On the result, simulation.replicate_index and simulation.n_trials identify the trial, and reports render a "Trial k of N" row. Trials reuse the replicate machinery, so when a simulation task sets n_trials > 1 it takes precedence over experiment-level repeats.

Trials are ordinary batch units: they run concurrently exactly as far as --max-parallel allows — there is no per-task trial-concurrency switch; scheduling is entirely --max-parallel's job.

Grading a dialog

Set include_dialog: true on every llm_judge / agent_judge criterion in a simulated task. This is not optional polish — without it the judge sees only the agent's outputs and no user side, so a requirement the user stated in turn 3 looks like something the agent invented. Judges routinely score that as a hallucination.

success_criteria:
  - type: llm_judge
    description: "Agent satisfied the requirement the user gave mid-conversation"
    prompt: |
      Read the DIALOG block. Grade the agent on whether it satisfied the
      requirement the user actually stated — not what you assume the task is.
    include_dialog: true
    pass_threshold: 0.9

The rendered dialog is wrapped as UNTRUSTED DATA with a rubric guard telling the judge that claims made only by the simulated user may be invented, and not to penalize the agent for going along with them. That guard is what keeps a chatty simulator from poisoning the grade.

When attempts repeat or fail, the dialog labels earlier and failed attempts as evidence while [Turn N] identifies the latest nonfailed agent attempt. Process-wide criteria still see failed agent work. A failed latest output or tool summary is explicitly labelled as failure evidence. A legacy user-only duplicate without agent telemetry is omitted with a degraded note; the raw record and its costs stay unchanged. An explicit record_kind: agent or positive agent telemetry preserves a genuine empty answer. A custom legacy adapter's empty duplicate can be ambiguous without either.

What it costs

Budget roughly max_turns × n_trials agent turns per (task, variant) — the worst case, since the dialog usually stops on the stop token first. On top of that:

• Simulator tokens, one generation per turn (none in a scripted dialog). Small next to the agent's, but not free, and they are reported separately as simulation.simulator_input_tokens / simulator_output_tokens.
• Criterion cost, multiplied by turn count under check_criteria: every_turn or both.

Cap it with run_limits — but note the accounting boundary: run_limits budgets count the subject agent only, so simulator spend is invisible to max_total_tokens and max_usd. To bound the conversation as a whole, use simulation.max_total_tokens, which counts both sides.

Dialog mode as an experiment axis

Everything on the simulation: block is overridable per variant, which makes the conversation itself the thing under test. A variant sets a partial simulation: block that shallow-merges onto the task's:

variants:
  - variant_id: patient-user
    simulation:
      persona: "A patient engineer who answers questions precisely."
  - variant_id: terse-user
    simulation:
      persona: "A terse, impatient user who gives one-line answers."

A variant's turns replaces the task's script as a whole. Productive axes: persona (terse vs. chatty, technical vs. not), how much the goal withholds, and n_trials as a cost/confidence trade. See A/B Experiments for the full experiment layer.

Security posture

The task's reference solution is never passed to the simulator, exactly as it is never passed to the coding agent. A simulator that could read the reference would leak it into the dialog and the agent would be graded against an answer it was handed.

What the simulator does see, all inlined into its system prompt: persona, goal, constraints, the task's description, and the pinned initial_prompt when there is one. The description is easy to forget — it reads like evaluator-private metadata, but the simulator gets it verbatim. If your persona withholds a requirement, don't spell that requirement out in description, or the simulator may volunteer what you meant it to keep back.

The simulator also runs with no tools, no plugins, and no settings sources, so it cannot read or write the sandbox even if its persona instructs it to.

See also

• Task Definition Guide → Simulation — every field, default, and constraint.
• A/B Experiments — running personas as variants.
• Run Limits — the caps that bound a dialog's cost.