Your Voice Agent's Dead Air Is an Architecture Problem

You have shipped a voice agent. It answers correctly, the transcripts look great in review, and the demo with scripted questions went fine. Then real callers start using it, and the complaint that comes back is not that it is wrong. It is that it feels dead. The caller finishes a sentence, and there is a beat of silence, then another, and by the second and a half of nothing they have either started repeating themselves or decided the thing is broken and asked for a human.

That silence is not a model quality problem and you cannot prompt your way out of it. It is the sound of a single loop doing everything in order: waiting to be sure the caller stopped, running retrieval, calling the model, then synthesizing speech, all before the first sound comes back. Each step is defensible on its own. Added up, they blow past the window in which a human expects a response, and the whole system reads as slow no matter how good the eventual answer is.

Why a faster model does not save you

The instinct is to reach for a faster model, and it is the wrong instinct because the model is only one term in the sum. A conversational turn feels natural when the reply starts within roughly 300 milliseconds of the caller going quiet. That is not a target you picked, it is roughly the gap length at which people in a real conversation start to assume something went wrong. Now count what has to happen inside that window in a naive agent: endpoint detection to confirm the caller actually stopped, a retrieval call that a typical vector database answers in 50 to 300 milliseconds, the model generating an answer, and text to speech producing the first audio frame.

You do not need precise numbers to see the problem. Retrieval alone can eat most of the budget, and generation has not even started. Swapping in a model that is twice as fast moves one term and leaves the structure intact: the caller still waits for the slowest sequential path through the whole pipeline. The fix is not to make the expensive work instant. It is to stop making the caller wait for it.

Two loops, two jobs, two budgets

The pattern that works borrows straight from dual-process thinking. You run two loops at different speeds.

The fast loop owns the conversation surface. It listens, detects turns, handles interruptions, decides when to make a small holding sound, and drives the spoken channel. Its entire job is to react inside the human patience window, so it is never allowed to block on anything expensive. It does not do retrieval. It does not call the big model and wait.

The slow loop owns the work. Retrieval, tool calls, business logic, and the actual answer synthesis all live here, and this loop is allowed to take a second or more because the fast loop is covering for it. It reads and writes shared conversation state, and it hands finished audio back to the fast loop to speak.

The contract between them is the whole design. The fast loop dispatches a turn to the slow loop and immediately goes back to watching the microphone. If the slow loop produces an answer quickly, great, speak it. If it does not, the fast loop fills the gap with something natural so the caller never hears silence. Here is the core of the fast loop.

# fast_loop.py -- owns the mic, turn-taking, and the spoken channel.
# It must react inside a human's patience window, so it never blocks
# on retrieval or the big model. It hands real work to the slow loop
# and keeps the conversation alive while that work runs.
import asyncio

FILLER_AFTER_MS = 400  # no audio from the slow loop by now -> say something

async def handle_turn(turn, slow_loop, tts, state):
    state.begin_turn(turn)
    answer = asyncio.create_task(slow_loop.respond(turn, state))

    # Race the real answer against a short timer. If reasoning is slow,
    # emit a natural holding phrase so the caller never hears dead air.
    done, _ = await asyncio.wait({answer}, timeout=FILLER_AFTER_MS / 1000)
    if not done:
        await tts.say(state.pick_filler())   # e.g. "let me pull that up"

    reply = await answer
    await tts.stream(reply)

The holding phrase is doing real work, not papering over a bug. A short "let me check that for you" resets the caller's clock and buys the slow loop another second without the interaction feeling broken. The rule is that the filler must be honest and content-free. It can say the agent is looking something up, because it is. It must never assert anything about the answer, because the slow loop has not produced one yet, and a filler that guesses at content is how you end up contradicting yourself out loud.

This is the same perceived-latency principle behind streaming partial progress in text agents, applied to a channel where you cannot stream tokens as they arrive because speech is linear and committed the moment it leaves the speaker.

Barge-in is where naive agents fall apart

The moment you have an agent that talks for more than a second, callers will interrupt it, and handling that badly is worse than the dead air you started with. When the caller starts speaking over the agent, three things have to happen at once, and in the right order.

Stop the mouth immediately. Abandon the reasoning that was in flight for the turn the caller just cut off, because they have moved on and its answer is now stale. And critically, roll the stored transcript back to only what the caller actually heard, not what the agent had planned to say.

That last one is the subtle bug. If your text to speech had queued three sentences and the caller interrupted after the first, the next turn must reason as if only that first sentence was ever spoken, because from the caller's point of view it was. Keep the unsaid text in the transcript and the agent starts referring to things it never actually told the caller.

# barge_in.py -- the caller talks while the agent is speaking.
# Stop the audio, abandon the in-flight turn, and make the transcript
# match what the caller actually heard, not what we planned to say.
async def on_user_speech(state, tts, answer_task):
    tts.cancel()                       # kill the active speech stream this frame

    if answer_task and not answer_task.done():
        answer_task.cancel()           # the slow loop may be mid-retrieval
        try:
            await answer_task
        except asyncio.CancelledError:
            pass

    # Truncate the transcript to what was actually flushed to the speaker,
    # or the next turn reasons over words that were never in the room.
    state.truncate_spoken_to(tts.flushed_chars())

tts.flushed_chars() is the number of characters actually converted to audio and played, which your speech layer can report from its playback cursor. If you cannot get that, track it yourself by counting what the player confirms rather than what you sent it. This is the difference between a barge-in that feels like talking to a person who stops to listen, and one that talks over itself and loses the thread.

Warm the expensive work before the caller finishes

The fast loop hides latency. Speculative retrieval removes it. The idea is that you do not have to wait for the caller to finish before you start guessing what they want, because partial transcripts are available while they are still talking.

A small, cheap classifier watches the partial transcript, predicts the one or two most likely things the caller is asking about, and fires those retrievals in the background. By the time the turn actually completes, the slow loop reads warm results out of a cache instead of paying for a cold vector search on the critical path.

# slow_thinker.py -- runs one turn ahead of the caller. While the user
# is still talking, guess likely intent and warm the cache so the real
# retrieval is a hit, not a cold query on the critical path.
async def prefetch_loop(stream, retriever, cache):
    async for partial in stream.partial_transcripts():
        if len(partial.split()) < 4:
            continue                             # too little signal to guess yet

        intent = await cheap_classifier(partial)  # small, fast, non-reasoning
        for query in intent.likely_queries[:2]:   # cap the speculation
            if query not in cache:
                asyncio.create_task(warm(retriever, cache, query))

async def warm(retriever, cache, query):
    cache[query] = await retriever.search(query)  # answer path reads this later

The economics are the same as any speculative execution. You do wasted work on the guesses that turn out wrong, so you cap the number of queries per turn and you measure the hit rate. If the classifier is guessing right often enough that the saved retrieval latency outweighs the cost of the misses, keep it. If not, tune the classifier or narrow when it fires. This composes cleanly with a real semantic cache in front of retrieval, where a warm speculative result and a genuine cache hit look identical to the answer path.

Measure each loop against its own budget

The failure mode with two loops is that you report one blended response time and lose the ability to see which loop is the problem. A slow retrieval regression can hide behind a snappy fast loop until callers start complaining, and then you are debugging a vague "it feels slow" with no signal pointing anywhere. Give each loop its own budget and trace against it.

# budgets.py -- each loop gets its own budget and is measured separately.
# A blended "response time" hides which loop actually regressed.
BUDGETS_MS = {
    "vad_to_endpoint":  250,   # fast loop: confirm the caller stopped talking
    "endpoint_to_audio": 400,  # fast loop: first sound back to the caller
    "retrieval":        300,   # slow loop: vector search
    "synthesis":       1200,   # slow loop: the big model's full answer
}

def check(span_name, elapsed_ms, budgets=BUDGETS_MS):
    budget = budgets[span_name]
    if elapsed_ms > budget:
        record_violation(span_name, elapsed_ms, budget)  # per-loop alert
    return elapsed_ms <= budget

Wire record_violation into whatever you already use for agent tracing, as a span attribute you can alert on per loop. Now a regression is a specific named span breaching a specific budget, not a customer-support ticket that says the bot is laggy.

Where this bites you

A few things go wrong even after you split the loops, and they are worth knowing before they happen to you.

Fillers that lie are the most common self-inflicted wound. If the holding phrase ever implies anything about the answer and the slow loop then produces something different, the agent has contradicted itself in front of the caller. Keep fillers strictly about the act of looking, never about what was found.

Over-eager barge-in makes the agent impossible to talk to. If a cough or a background voice trips voice activity detection, the agent stops mid-sentence for no reason. Use learned turn detection rather than a raw silence threshold, and require a short burst of real speech before you treat it as an interruption.

Shared state between two loops running concurrently is a race waiting to happen. The slow loop is reading and writing the same conversation state the fast loop mutates on barge-in. Decide on a single owner for each field, and make the slow loop's writes conditional on the turn it started still being the active one, so a cancelled turn cannot commit its answer after the caller has already moved on. This is the same discipline as running tool calls in parallel without corrupting shared context.

And do not let speculative retrieval run unbounded. Two guesses per turn is a design choice, not a default. Left uncapped, a chatty caller and an eager classifier will fan out into dozens of background searches, and you have traded a latency problem for a cost and load problem.

The takeaway

A voice agent lives or dies on whether it feels present, and presence is an architecture property, not a model property. One sequential loop makes the caller wait for the slowest thing you do, and no model is fast enough to hide that. Split the fast loop that owns the caller's attention from the slow loop that does the work, fill the gap with honest holding phrases, cancel cleanly on barge-in so the transcript matches reality, and warm the expensive retrieval before the caller has finished the sentence. The answer quality was never the issue. The waiting was.

If you have a voice agent that is technically correct but callers keep dropping off, the fix is usually in how the loops are split rather than which model you picked. Book a consultation call and we can find where your turns are actually spending their time.