voice-eval-toolkit

A command line tool for testing voice agents

Test how a voice agent behaves when callers interrupt.

voice-eval-toolkit places scripted test calls to your voice agent, listens to both sides of the audio, and tells you how long the agent took to stop talking, how often it cut the caller off, and how long the caller waited for an answer.

It is a Python command line tool called qsv. It is for people who build voice agents and want numbers instead of "it felt fine on my call".

Two rows of audio, caller on top and agent below. The caller starts talking while the agent is speaking, and the agent goes quiet 200 milliseconds later.
A real run of the tool. The caller cuts in while the agent is talking, and the agent goes quiet 200 ms later. This call used the mock agent that ships with the repo, not a real voice agent.

What it tests

Barge-in stop latency

Time from the caller starting to speak until the agent's audio stops.

False interruptions

Times the agent started talking before the caller had finished.

Turn gap

Time from the caller finishing a sentence until the agent starts to answer.

Latency p50, p95 and p99

The usual percentiles over every turn in a run.

Word error rate on phone audio

How well the agent's own transcript matches what the caller said, at 8 kHz. This needs an agent that publishes its transcripts.

All timings come from the recorded audio, using voice activity detection on both channels. The agent is treated as a black box, so the same scoring works for any stack.

The problem

Voice agents often keep talking after the caller starts speaking. Some answer too slowly, and some jump in while the caller is still thinking.

These problems are hard to spot by calling your own agent a few times. They are even harder to check when many people call at once.

How it works

You describe a call in a short YAML file: what the caller says, when it pauses, and when it interrupts. The tool starts as many simulated callers as you ask for and sends their audio to your agent. It records what comes back and scores it.

Diagram. A scenario file feeds simulated callers, which talk through a transport to the voice agent. Both audio streams are recorded, passed through voice activity detection, turned into metrics, and written to a report or sent over OpenTelemetry.
Scenario in, audio through a transport, metrics out.
Two histograms. Turn gap values run from about 600 to 1000 milliseconds. Barge-in stop latency values run from about 140 to 260 milliseconds.
Turn gap and stop latency from 20 loopback calls. The mock agent has random jitter switched on so the charts have something to show. These are not real agent numbers.

Quickstart

git clone https://github.com/AQIB-NAWAB/voice-eval-toolkit
cd voice-eval-toolkit
python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"

# one call, interrupting the agent, against the built-in mock agent
qsv bargein --transport loopback

# twenty callers at once
qsv load --transport loopback \
  --scenario examples/scenarios/barge_in.yaml --callers 20

Each run writes report.json and report.md. You can also set pass or fail limits, so a slow agent fails your CI build.

qsv bargein, one call
Terminal output of a qsv bargein run against the mock agent, showing turn gap and barge-in stop latency.
One call against the mock agent. The numbers describe the mock agent's settings, not a real agent.
qsv load, 20 callers
Terminal output of a qsv load run with 20 callers, showing turn gap and barge-in stop latency percentiles.
Twenty callers at once, also against the mock agent.

Works with

Related

telephony-wer-bench is a sibling project. It measures how well streaming speech-to-text hears phone audio: word error rate, time to first partial and final latency under G.711, packet loss and noise. This toolkit tests how the agent behaves in a call. That one tests how well the speech-to-text stage hears it.

Status

Early. It works and has tests, but it has only been run against the bundled mock agent and a local LiveKit server. Open source, MIT license. Issues and pull requests are welcome.