Why Your Voice Agent Keeps Talking Over the Caller (Pipecat and LiveKit Debugging Guide)

You ship a voice agent. In the demo it feels sharp. Then real callers start interrupting it, and it plows straight through them, finishing a sentence nobody is listening to anymore. The caller talks louder, the agent keeps going, and the call falls apart.

The frustrating part is that "the agent won't stop talking" is not a single failure. In a cascaded stack, the moment a caller barges in has to travel through voice activity detection, a turn or interruption decision, and then all the way back down to whatever is still holding audio: the TTS service and the output transport. A stall or a dropped signal at any one of those stages produces the same audible symptom. So the fix depends entirely on which stage is stalling, and you cannot tell which one by listening alone.

This guide walks the pipeline, shows you how to instrument it so you stop guessing, then maps seven specific symptoms (including the ones people report most: StartInterruptionFrame firing but TTS continuing, and min_interruption_duration looking like it is ignored) onto the exact layer that causes each one.

The pipeline where barge-in actually happens

A cascaded voice agent is a loop. Caller audio comes in, gets turned into text, a turn decision decides whether the caller is really taking the floor, the LLM produces a reply, TTS turns that reply into audio, and the output transport streams it back. Barge-in is what happens when new caller audio arrives while the bottom half of that loop is still emitting the previous reply.

Two clocks decide whether barge-in feels good, and it helps to separate them because they are fixed in different places:

• Detection latency: caller starts talking, and how long until your stack decides that is an interruption. This lives in the VAD window and the turn or interruption logic.
• Cancellation latency: the decision is made, and how long until the last caller-audible sample of the bot actually stops. This lives in the TTS buffer and the output transport, plus any telephony gateway between you and the caller.

"Time to yield" is the sum of both. When people say interrupts are slower, how do I cancel bot speech faster, they usually tune one clock and ignore the other. You have to measure both.

Instrument it before you guess

Every root cause below is confirmed the same way: line up three timelines and see where they disagree. Caller speech onset, the agent's actual audio envelope, and your framework's interruption events. If you only ever listen to the mixed call, you will keep tuning the wrong clock.

1. Get a recording you can measure

Pull a stereo or dual-channel recording if you possibly can, with the caller on one channel and the agent on the other. A mixed mono file still shows you that talk-over happened, but it makes "when did the agent go quiet" much harder to measure because both voices sit in the same waveform. Most stacks can hand you separate tracks: telephony providers expose dual-channel recording, and the agent frameworks can record the input and output legs.

2. Measure time to yield from the envelope

You do not need anything fancy to get a first number. Compute a short-window energy envelope on each channel, find where the caller channel comes alive while the agent channel is still hot, and then find where the agent channel drops and stays down. The gap is your time to yield for that event.

# first-pass time-to-yield from a stereo call recording
# channel 0 = caller, channel 1 = agent (swap if yours is reversed)
import numpy as np, soundfile as sf

audio, sr = sf.read("call_01.wav")      # shape (n, 2)
caller, agent = audio[:, 0], audio[:, 1]

def envelope(x, sr, win_ms=20):
    w = int(sr * win_ms / 1000)
    frames = len(x) // w
    e = np.array([np.sqrt(np.mean(x[i*w:(i+1)*w]**2)) for i in range(frames)])
    t = np.arange(frames) * win_ms / 1000.0
    return t, e

tc, ec = envelope(caller, sr)
ta, ea = envelope(agent, sr)

thr = 0.02                             # tune to your noise floor
caller_on = ec > thr
agent_on  = ea > thr

for i in range(1, len(tc)):
    if caller_on[i] and agent_on[i]:       # overlap starts: caller cut in
        onset = tc[i]
        j = i
        while j < len(ea) and ea[j] > thr:  # wait for agent to fall silent
            j += 1
        yield_t = ta[j] if j < len(ta) else ta[-1]
        print(f"caller in {onset:.2f}s, agent silent {yield_t:.2f}s, "
              f"time to yield {yield_t-onset:.2f}s")
        break

This is a starting point, not a scorer. A fixed energy threshold trips on breaths and line noise, so for anything you report on, run a real VAD (for example Silero or WebRTC VAD) per channel and hold the "silent" decision across a few frames so a pause inside a word does not read as a yield. But even the crude version tells you the one thing you need next: is the delay before the agent stops, or after the decision to stop.

3. Turn on the framework's interruption logs

Both stacks emit events at exactly the moments you care about. Log them with timestamps and overlay them on the audio.

• Pipecat emits frames as the turn changes: UserStartedSpeakingFrame when the VAD calls a turn start, StartInterruptionFrame when it decides to interrupt, and BotStoppedSpeakingFrame when the bot audio is done. The distance from UserStartedSpeakingFrame to StartInterruptionFrame is detection latency; from there to the audio actually stopping is cancellation latency.
• LiveKit exposes the agent state and the current speech handle. Watch when agent_state moves off speaking and when the speech handle for the turn is interrupted, and compare those to the recording.

Symptom to root cause map

Find your symptom, jump to the layer, confirm it against the recording. The detail for each row is below the table.

The knobs, side by side

Once you know which clock is slow, these are the levers. Names and defaults drift between releases, so treat this as a map of what to look for and confirm the exact values against the version you have installed.

A test you can run this afternoon

Guessing is expensive; a repeatable measurement is cheap. Here is a small protocol that turns "it feels like it interrupts badly" into numbers you can move.

1. Record with barge-in on purpose. Run a handful of calls where you interrupt the agent at known points: early in a long sentence, mid-word, and right as it starts a new sentence. Keep the recordings dual-channel.
2. Score three fixed things per interruption. Did the agent yield at all, how many seconds it took to yield, and did it keep talking over the caller after the cut-in. Those three are objective and they map straight onto the two clocks above.
3. Set pass thresholds up front. For example: yields on every event, time to yield under a target you pick for your use case, and no sustained talk-over past the buffered tail. Write the thresholds down before you look at the results so you are not grading on a curve.
4. Change one knob, re-measure. Move min_interruption_words, or the VAD window, or the TTS chunk size, one at a time, and watch the two clocks separately. If a change helps detection but hurts cancellation, you will only see it because you split the number.

The trap is that a handful of hand-run calls is not your real traffic. Real callers interrupt in messier ways than you do on purpose, so a stack that passes your five scripted barge-ins can still fail on the long tail. The honest version of this test runs across a real batch of your own recorded calls and scores every barge-in event the same way, so the number reflects production, not a demo. For a fuller walk-through of building this measurement, including what counts as an event and how to set thresholds, see how to test voice agent barge-in: 3 objective signals.

FAQ