Voice agent testing: how to evaluate before you take real calls

Voice agent testing starts with a nine-scenario suite and pass criteria that score a conversation as a whole. Launch readiness comes down to whether the agent finishes the job on a real call. One recording per scenario and a written end state give you the answer before your callers do. Re-run the suite after prompt edits.

Testing means scoring the whole call. You record when the agent spoke, what it committed to, and whether the caller left with what they came for. WER measures a transcript, so no word-level score tells you whether the agent interrupted the caller or looped on a confirmation. A confident read-back of the wrong order number also falls outside that score.

Your STT provider produces the transcript. You remain responsible for how the full agent handles it. So the behavior around those words is what you have to test, and only a live recording shows it. In this article you'll build the suite, set the pass line, and walk from uploaded clips to real phone calls.

Pre-launch confidence comes from scoring whole calls against written expected outcomes.

• Emotional speech can increase recognition errors relative to neutral speech.
• The suite covers interruptions, silence, accents, anger, and wrong numbers that happy-path demos miss.
• Turn counts come off the session log, while task completion needs a system of record to diff. False statements need a listener.
• Small prompt edits can produce large, model-dependent behavior changes, so every prompt edit re-runs the suite.
• Use progressively more realistic test environments before launch.

Recorded TTS clips are the easy case, and passing them proves little. They differ from human speech in prosodic timing and variance, as TTS comparison research shows, so voice agent testing starts from what callers actually do.

The clips are read in a neutral register, arrive without hesitation or overlap, and pause for exactly as long as you scripted. Some failure modes real callers trigger may never fire.

Your TTS engine defaults to a flat, even register, which is the easiest input your stack will ever get. Expressive clips can produce acoustic patterns outside the data most recognizers train on. Deepgram's accuracy tax research states that speech recognition systems "train on predominantly neutral data."

From a 26.1% WER reduction achieved by emotion-aware synthesis, the same analysis infers that untuned emotional TTS is associated with WER roughly 25–35% higher than neutral baselines. The figure is Deepgram's inference from the RLAIF-SPA paper. The degradation was not measured.

Real callers bring unpredictable timing and speech errors that no script contains. Overlap and gaps of varying length occur in live conversations, as turn-timing research documents, and a scripted clip may produce none unless you add them.

Your TTS reads a clean script, so the "uh," the restart, and the self-correction never arrive. Each omission is a distinct way for endpointing and barge-in logic to misfire.

The model layer ends at the transcript, and a better model can only reduce recognition errors involving accents, noise, disfluencies, and emotional prosody. Agent control logic remains outside the model, including dialogue policy and barge-in handling. Incorrect tool actions also come from that control logic.

As of 2026, Deepgram can supply the speech layer under your agent or, through its Voice Agent API, infrastructure combining STT, TTS, and LLM orchestration. You built the agent's response to a transcript that arrived mid-interruption, and only voice agent testing exercises that behavior.

Everything downstream depends on these scenarios, so a thin suite caps how much any later test can tell you. Recorded calls with written outcomes are the deliverable here. Scoring and prompt regression run against this same set. Telephony testing uses it too.

Copy the table and rewrite the failure column in your domain's vocabulary.

Keep the middle column when you adapt the rows, since it's what tells you whether a passing call actually proved anything.

Every row describes a behavior you can hear in the recording. A diff against a reference transcript doesn't capture that behavior, and neither does a per-utterance score. Write each expected outcome as an end state a reviewer can mark without interpreting, such as "appointment exists for Thursday and the caller heard the time read back."

Mine recordings from a human-staffed line for rows this table lacks, because what any suite misses are the calls nobody thought to script. Listen for the moment a caller does something unplanned, then write it up as a row with its own expected outcome.

Mid-sentence code-switching and numbers read aloud off a screen show up in live traffic long before they reach a test plan.

A conversation passes or fails on four measures, and those are task completion, turns to resolution, escalation, and false statements. You can compute transcript scores with a script, but you must judge conversation-level outcomes against the written expectation. Name the metrics here and set the pass line yourself.

Task completion is a yes or no, and turns to resolution is a plain count. When your agent writes to a system of record, diffing the end state settles completion without a reviewer. You mark everything else. The turn count comes straight off the session log.

Escalation is scored on whether the call reached a person when it needed to, and false statements are scored as a count. Containment rate is the share of calls handled without escalation, and its mirror image is what your out-of-scope row tests.

Mark any claim your policy or database contradicts as a false statement. Air Canada's chatbot gave a customer incorrect bereavement-fare information. On February 14, 2024, a British Columbia tribunal held the airline liable for negligent misrepresentation. Tolerance depends on what each false statement costs you.

A reviewer owns the verdict, and the pass line belongs to your business rather than to any published benchmark. You score each recording against the written outcome. LLM judges can scale that work, but the final call stays yours.

One production deployment covered millions of calls. Artefact's engineers found one binary judge per question reached 90% precision and recall against human labels on "Request Resolved." The same containment rate is a triumph where escalation costs a specialist's hour and a disaster where a missed handoff loses a sale. Call value and escalation cost set yours.

One changed word in the instructions can flip behavior in a scenario you never touched, so every prompt edit re-runs the suite.

The same edit can help one model and hurt another, so reading the diff tells you nothing about the outcome.

Swapping "excludes" for "lacks" degraded the Flan-T5-large model by 28% on average and improved Flan-T5-XL by 46% when researchers ran the same task across many prompt wordings. Model updates behave the same way. Across API updates studied, 55% had no consistent direction. Some prompts improved while others worsened on the same task.

If a change touches personality, run the suite anyway, including the wrong-number caller. Deepgram's prompting guide recommends changing one section at a time, since editing two at once hides which one caused the regression.

It also points you at the recordings, so score fresh ones after each edit instead of rereading the prompt. The guide says to test with the specific model you plan to deploy, and the API-update results are the reason.

Keep a changelog at the top of the prompt as a comment block, as the guide recommends. Store the prompt in source control next to the scorecard from its last run. Rollback is then a revert plus a re-run.

Testing moves through three stages, starting with uploaded clips, then real calls over the transport you'll ship on, then production telemetry. A browser demo stops answering the question once your callers dial in over the phone network.

Moveo's published evaluation documents where browser-demo results diverge from telephony and slices its telemetry by language. The language slices reflect customer practice. Deepgram doesn't prescribe this method.

Uploaded clips isolate model and endpointing behavior, which makes a playground the right place to start. Clips can't exercise live full-duplex overlap, though, since a recording plays on a fixed timeline and never talks back when the agent does.

Place one real call per scenario over the transport your callers will use. Moveo tracked transport as WebRTC versus Twilio SIP. If you build on Deepgram's Voice Agent API, the telephony leg is the piece you wire and the piece this stage tests.

After launch, watch latency at each hop and slice it by language and transport. Moveo slices transcription delay and end-of-utterance delay. It also tracks agent response latency. The slices cover provider and language, with transport tracked separately. Behavior in English and Brazilian Portuguese diverged from the aggregate averages. Greek did too.

No suite reaches your real caller mix, your traffic patterns, or the intents nobody predicted. So plan the first week as observation rather than as proof.

Watch transfer success, silence-triggered hangups, and any intent your scorecard has no row for. One healthcare deployment reported transfer-to-human failures in its six-month retrospective.

The same team found callers who said "I'll hold" and went quiet were dropped by an 8-second silence timeout, a pattern that surfaced only after launch. Each failed call you capture becomes a new row in the suite.

The handoff happens once real traffic is flowing, when scored recordings give way to continuous monitoring.

Latency under concurrent load, alerting thresholds, dashboards, and escalation design belong to the production orchestration playbook, which picks up where this page stops. For STT model selection, see the model-specific guidance. Test setup and scoring methods live in the speech-to-text benchmarks guide.

Pull the table into a doc, have a colleague call the agent for each row, and score the recordings against the outcomes you wrote. Then change one line of the prompt and do it again. If the second run passes every row the first one did, you've got a regression practice instead of a hunch.

Try it on your own calls. Create a Deepgram account and put your $200 free credits against the scenarios your agent fails.

Nine rows is enough to start, which is what the table above gives you. Weight each row by the cost of failure, then run high-consequence rows first. Breadth matters more than volume, so a new failure class beats another variation on one you already cover. When a failed call reveals a new intent or timing pattern, promote its recording ahead of near-duplicates.

Some of it automates and some of it always needs a person. Split your scorecard into machine-checkable fields and reviewer fields. Put system-of-record state and session-log turn counts in the first group. Put false statements and judgment calls in the second, since deciding whether the agent said something untrue takes ears on the recording.

Yes, and the old model's passing scorecard is the bar the new one has to clear. Compare fresh recordings for the accent and noise rows, then check the angry-caller row separately. Re-sweep silence-timeout and endpointing settings. Ship once nothing that passed before is failing.

Voice agent testing scores one call at a time, and load testing checks whether those results hold when sessions pile up. Run the conversation suite as an acceptance gate first. Then replay the same passing scenarios in parallel and compare per-call outcomes with the one-call baseline. Concurrency problems show up as outcomes that changed or as infrastructure limits you hit.

A shared spreadsheet with one row per scenario is the smallest useful setup at this scale. Attach each phone recording, session-log turn count, expected end state, and system-of-record result. Skip LLM judging until call volume outgrows two people.