Tuning Turn Detection and VAD: False Interruptions vs Missed Interruptions

The one axis behind both failures

Barge-in is the moment a user starts speaking while the agent is still talking. Getting it right means the agent yields to the user quickly when they genuinely take the turn, and keeps talking when the user is only pausing, thinking, breathing, or dropping in a filler word. Those two goals pull in opposite directions, and that tension is the whole problem.

It helps to name the two errors precisely, because you tune against them by name:

• False barge-in (a false interruption). The agent yields when it should not have. The user paused mid-sentence, said um, dropped a backchannel like mhm or yeah, took a breath, or a TV was playing in the background, and the agent stopped talking or cut itself off. To the user, the agent seems twitchy and keeps abandoning its own sentences.
• Missed barge-in (a missed interruption, also called talk-over). The agent fails to yield when it should have. The user clearly started a new turn, and the agent kept talking over them for a second or more. To the user, the agent seems deaf and rude.

If you have tuned detectors before, this is precision and recall. Every knob in the stack sits somewhere on a single sensitivity axis. Push toward more sensitive and you cut missed barge-ins while adding false barge-ins. Push toward less sensitive and you cut false barge-ins while adding missed barge-ins and slowing the agent down. You cannot drive both errors to zero by turning one knob. You either pick a point on the curve that fits your product, or you move to a smarter detector that shifts the whole curve.

The levers, and which way each one pushes

Here are the controls that actually move barge-in behavior in a modern voice stack, whether you run Vapi, Retell, Bland, Pipecat, or self-hosted LiveKit. The parameter names differ between stacks, but the levers are the same underneath. The table shows what happens when you turn each one up.

Directions are the reliable part. Exact defaults depend on your stack and version, so confirm them in your own config.

VAD threshold and the raw detector

A voice activity detector answers one narrow question: is there speech in this frame. Silero VAD, which many stacks use, returns a speech probability between 0 and 1 for each short window and compares it to a threshold. Lower the threshold and quieter or noisier audio starts counting as speech, which catches soft interruptions but also fires on background talk and breaths. Raise it and you get the opposite.

The important mental model: a VAD reports speech presence, not turn intent. It does not know whether the user is interrupting, agreeing, or thinking aloud. That is why a VAD alone is a bad barge-in detector, and why the other levers exist on top of it. If silero vad is not detecting an interruption, the usual causes are a threshold set too high, a minimum-silence or minimum-speech window that filters the interruption out, or an audio format mismatch. Silero expects 16 kHz mono in a fixed window size (512 samples at 16 kHz), and feeding it the wrong rate quietly degrades the probabilities.

# Silero VAD, illustrative defaults. Lower threshold and the
# shorter windows = more sensitive = fewer missed, more false.
get_speech_timestamps(
    audio, model,
    threshold=0.5,               # speech-probability cutoff
    min_speech_duration_ms=250,  # ignore blips shorter than this
    min_silence_duration_ms=100, # silence before ending a segment
    speech_pad_ms=30,
)

Parameter names from Silero VAD. Values shown are its documented defaults, not a recommendation. Confirm against your version.

Minimum interruption duration and minimum interruption words

These are the two cheapest, highest-leverage knobs for the "turn detection too sensitive, keeps interrupting the user" problem. A minimum interruption duration says the user must keep speaking for, say, a few hundred milliseconds before the agent yields, which throws away coughs, clicks, and stray syllables. A minimum interruption words count says the transcriber must produce at least N words before the agent yields, which is how you get interruption detection to ignore filler words and one-word backchannels.

The trade is direct and worth stating plainly. Both add latency to real interruptions, because the agent now waits to be sure. A duration gate adds a fixed delay. A words gate adds however long transcription takes to emit those words, and if you set it to two or three words you will miss short but genuine interruptions like stop, no, or wait. Many teams land on a small duration gate plus a words gate of zero or one, then let a semantic detector handle the harder filler cases.

Endpointing and end-of-utterance timeouts

Endpointing is the other side of the same coin, and it is usually the real reason a "voice assistant starts speaking before the user finishes." Endpointing decides when the user's turn is over. If it is silence-based and the timeout is short, a normal mid-sentence pause (for example "I want to book a flight to, uh, Denver") gets scored as end-of-turn, the agent takes the floor, and now you have both a false end-of-turn and, a beat later, the agent talking over the user as they finish. Lengthen the minimum silence or the minimum endpointing delay and the agent stops cutting people off, at the cost of a slower reply after every genuine turn. Frameworks such as LiveKit Agents expose this directly as a minimum and maximum endpointing delay so you can bound both the responsiveness and the worst-case wait.

Semantic turn detection: Smart Turn v3 and realtime semantic VAD

Everything above trades one error for the other along a fixed curve. Semantic turn detection is how you move the curve itself. Instead of asking only "is there silence," a semantic model asks "does this sound or read like a finished thought." Because it uses content and prosody rather than a raw gap, it can hold the turn through a pause or a filler word without you having to lengthen a blunt silence timer that also slows down real turn-ends.

Two concrete forms you are likely to reach for:

• Audio end-of-utterance models such as Smart Turn v3. These classify from the audio whether an utterance is complete or the speaker is merely pausing, and feed that into endpointing. Smart Turn v3 is multilingual and small enough to run quickly on CPU, so the end-of-utterance and EOU latency it adds is modest while it meaningfully cuts premature endpointing. It is available to run in Pipecat pipelines and elsewhere.
• Realtime semantic VAD. The OpenAI Realtime API lets you choose the turn-detection mode. server_vad ends the turn on silence and exposes threshold, prefix_padding_ms, and silence_duration_ms. semantic_vad instead uses a model to judge whether the user is done, so a natural pause is less likely to be read as end-of-turn, which reduces false interruptions. Its eagerness setting (low, medium, high, or auto) trades how quickly it commits against how long it waits to be sure.

// OpenAI Realtime: silence-based turn taking
"turn_detection": {
  "type": "server_vad",
  "threshold": 0.5,
  "prefix_padding_ms": 300,
  "silence_duration_ms": 200
}

// OpenAI Realtime: model-judged turn taking,
// less likely to end a turn on a natural pause
"turn_detection": {
  "type": "semantic_vad",
  "eagerness": "auto"
}

Field names from the OpenAI Realtime turn-detection settings. Shown to illustrate the two modes, not tuned values.

How to measure both directions from recorded calls

Here is the part most teams skip, and it is the part that turns tuning from guesswork into engineering. You cannot improve what you only feel. You have to count both errors, on the same recordings, before and after every change. Tuning by listening to a demo call or two is how you ship an agent that felt fine in the office and talks over customers in production.

The measurement is a small labeling and counting job. Do it on real recorded calls, not scripted ones, because real calls carry the accents, phone-line codec artifacts, background noise, and hesitation patterns that actually trip the detector.

1. Get clean audio, ideally dual-channel.

   If your stack can record the user and the agent on separate channels, overlap becomes trivial to see: any span where both channels are hot is a talk-over candidate. If you only have a single mixed channel, segment it into who-is-speaking-when before you label. This is turn segmentation for scoring, not identifying any individual.

2. Mark every candidate interruption.

   Find every point where the user starts speaking while the agent is talking. Each one is a candidate. This is the denominator for both error rates.

3. Label the user's intent at each candidate.

   For each candidate, decide: was this a real turn-take, or was it a backchannel, filler, or noise. This is the human judgment the detector is trying to approximate, so it has to come from a person for the ground truth.

4. Score the agent's response.

   For real turn-takes: did the agent yield, and how many seconds did it take to go silent after the user's onset. For backchannels and filler: did the agent wrongly stop or hesitate. Now every candidate lands in one of the boxes below.

5. Roll it up into two rates and one distribution.

   Missed barge-in rate = real turn-takes the agent talked over, divided by all real turn-takes. False barge-in rate = non-turns the agent yielded to, divided by all non-turns (or per minute of agent speech). Time to yield = the distribution of seconds-to-silence on the real ones. Report the median and the tail, because the tail is what users remember.

The confusion matrix you are scoring toward. Both off-diagonal cells matter, and one knob trades one for the other.

Two things make this rigorous rather than anecdotal. First, if your stack lets you run the VAD or turn detector offline, replay the exact same audio through it at several threshold settings and count the boxes at each one. That gives you the real curve for your calls, not a vendor's benchmark. Second, always report both rates together. A single number like "yields 95 percent of the time" hides which error you are making, and the two are traded against each other. A team can cut talk-over to near zero and not notice they doubled their false barge-ins, because they were only watching one column.

A tuning loop that converges

Putting it together, the loop that reliably gets a team to a good operating point:

1. Instrument first.

   Log every VAD trigger, interruption event, and turn-end with timestamps, so a recording can be scored without re-listening to all of it.

2. Measure the baseline both ways.

   Run the scoring above on a real batch and write down your current false barge-in rate, missed barge-in rate, and time-to-yield distribution.

3. Move the cheapest knob toward your dominant error.

   If false barge-in dominates, raise the minimum interruption duration first, then consider a words gate or a semantic detector. If missed barge-in dominates, lower the VAD threshold or the duration gate, and confirm interruptions are even enabled.

4. Re-measure on the same batch.

   Confirm the error you targeted went down and watch how far the other one went up. Keep the change only if the trade is worth it for your product.

5. Add a semantic detector when the blunt knobs run out.

   If no single threshold gives you an acceptable rate of both, that is the signal to move the curve with semantic turn detection rather than keep trading one error for the other.

Whether a snappier median time to yield is worth a few more false barge-ins is a product call, not a universal answer. A high-stakes support line and a casual assistant will pick different points on the curve. The engineering job is to make the trade visible and measured so the product call is an informed one.

Frequently asked