Building Realtime Voice Agents: Sub-800ms Latency Budget and Barge-In

What you need to know

A voice agent lives or dies on a single number: how long the caller waits between finishing their sentence and hearing your agent begin its reply. Hold that under roughly 800 milliseconds and the conversation feels natural; drift past it and every exchange picks up a small, corrosive pause that makes the agent feel slow and eventually not worth talking to. This guide is about architecting a cascaded voice agent — speech-to-text, then a language model, then text-to-speech — that holds a sub-800ms round trip in the real world, on a Mumbai mobile line or a London landline, without pretending latency is someone else's problem.

The good news is that the budget is achievable with today's tooling if you are disciplined about two things: streaming every stage so the pipeline overlaps instead of running in series, and handling interruptions so the caller can talk over the agent the way they would talk over a person. Get those right and you have an agent that feels alive. Get them wrong and no amount of prompt engineering will save the experience. Here is the shape of the whole thing before we go deep:

• The target is a sub-800ms round-trip turn. Past about 800ms callers notice awkward pauses; past about 1,500ms the conversation feels broken.
• A naive sequential pipeline costs roughly 1,400ms. Streaming every stage — STT partials, LLM tokens, sentence-boundary TTS — drops that to 500-700ms.
• Know the budget. STT 100-300ms, LLM inference 350-1,000ms, TTS 90-200ms, plus 50-200ms of network hops between vendors.
• Barge-in is non-negotiable. When the caller interrupts, stop speaking within about 150ms, flush the TTS buffer, and reroute the new audio to STT.
• Cascaded versus speech-to-speech is a real fork. Cascaded is predictable and debuggable at roughly $0.0095-0.17 per minute; speech-to-speech can be cheaper or far dearer, and is harder to trace.

Why 800ms is the line: the perceptual physics of conversation

The 800ms figure is not a vendor marketing number; it comes from how humans actually take turns. In natural conversation the gap between one person stopping and the next starting averages a couple of hundred milliseconds — often less, sometimes with a slight overlap. We are exquisitely tuned to that rhythm, and we read a longer silence as hesitation, confusion or a bad line. When a machine leaves a gap of a second or more before every reply, the listener does not consciously time it; they simply feel that the thing on the other end is slow, or stupid, or not really listening.

That gives us a practical ladder. A round-trip turn under roughly 800ms reads as responsive — the caller barely registers the delay. Somewhere past 800ms the pauses become noticeable and the agent starts to feel awkward, the way a video call with a satellite delay does. Past about 1,500ms the conversation feels broken: callers begin to repeat themselves, talk over the agent, or assume the line has dropped. Those two thresholds — 800ms for "notice" and 1,500ms for "broken" — are the goalposts every architectural decision has to serve.

Telephony sharpens the stakes. A caller who has dialled a support number on a phone has decades of muscle memory for how a phone call sounds, and they are far less forgiving of dead air than someone typing into a chat widget. This is true whether the number rings in Mumbai or Manchester: an Indian customer waiting on a mobile connection and a British customer on a landline both expect the other party to come back promptly, and both will start talking into the gap if you leave one. Designing for the phone means designing for the least patient version of your user, so the 800ms budget is a floor to defend, not an aspiration to approach.

The latency budget, stage by stage

Because a cascaded agent is a chain, the round-trip time is the sum of what each link costs — unless you overlap them, which is the whole game and which we come to next. First, know the raw budget. The ranges below are the ones to plan against as of mid-2026; your exact numbers depend on model choice, audio chunk size, region and how many vendor boundaries the audio crosses.

The single most useful thing this table shows is that the language model is the expensive link, at 350-1,000ms, and it is also the one you have least control over once you have picked a model. That is why streaming the LLM matters more than any other optimisation: you cannot make the model finish faster, but you can start speaking before it has finished. The STT and TTS stages are comparatively cheap, and for TTS the only number that lands on the critical path is time-to-first-audio — how long until the first slice of speech comes back — not how long the full sentence takes to synthesise.

TTFA is where your text-to-speech vendor choice shows up in the budget. Gradium's 2026 benchmark measured median time-to-first-audio across the main engines, and the spread is wide enough to matter when you are fighting for a sub-800ms total:

Two lessons hide in that table. First, the gap between the quickest and slowest median is well over 150ms — an entire barge-in budget — so the engine you pick genuinely moves your round trip. Second, median is not the whole story: ElevenLabs trades a slower median for a much tighter spread, roughly a 28ms interquartile range, which means fewer ugly outlier turns where one reply arrives late and the caller notices. On a phone line, where every stray pause is felt, predictable-but-slightly-slower can beat fast-but-jittery. Measure P50 and P95 for your own traffic before you commit, because the benchmark ranking may not survive contact with your region, your accents and your audio codec.

The STT side has moved fast too. As of mid-2026, ElevenLabs Scribe v2 Realtime reports sub-150ms transcription latency over a WebSocket connection while holding 93.5% accuracy on the FLEURS benchmark across 30 languages — which matters for the multilingual reality of Indian and UK call centres alike. On turn detection, Deepgram Flux posts a median end-of-turn detection under 300ms, and that number feeds directly into how quickly you can hand off to the language model.

Streaming everything: partials, token-to-TTS handoff, sentence-boundary flushing

Here is the mental model that makes the budget work: a well-built voice turn is not a relay race where each runner waits for the baton, it is three runners going at once with the baton passed the instant a fragment is ready. Concretely, three overlaps do almost all the work.

Streaming STT partials. A streaming recogniser emits interim transcripts as the caller is still speaking, and a final transcript a beat after they stop. If you wait for the full, settled utterance before doing anything, you pay the whole recognition cost on the critical path. If instead you act on the final partial the moment endpointing fires, you save 200-400ms — the recogniser has effectively been working while the caller talked.

Token-to-TTS handoff at the sentence boundary. This is the big one. Language models stream their output token by token, and English prose reaches its first natural break — a full stop, question mark or exclamation — long before the reply is complete. The trick is to buffer tokens only until that first sentence boundary appears, then flush that complete sentence straight to text-to-speech while the model keeps generating the rest. The caller starts hearing audio while the model is still thinking, which hides the bulk of the 350-1,000ms LLM cost behind speech. This single overlap saves 200-500ms, and it is the difference between an agent that pauses to think and one that starts answering and thinks aloud.

Put those overlaps together and the arithmetic changes completely: a naive sequential turn that would have cost around 1,400ms collapses to 500-700ms — comfortably inside the 800ms line. The sketch below shows the shape of a streamed turn: consume STT partials, stream LLM tokens, and flush each sentence to TTS as soon as a boundary appears. It is deliberately model-agnostic; swap in whichever providers you use behind stt_stream, llm_stream and tts_out. If you already stream your text LLM responses over server-sent events, the same discipline applies here — our guide on streaming LLM responses at scale with SSE and backpressure covers the plumbing.

import asyncio, re

# Streaming voice turn: partial STT -> LLM token stream ->
# flush a sentence to TTS the moment a boundary appears.
# Run STT, LLM and TTS as overlapping stages; never block one
# on the previous stage finishing.

SENTENCE_END = re.compile(r'[.!?](\s|$)')

async def handle_turn(audio_in, tts_out, cancel):
    # 1) Streaming STT: consume partials, act on the final.
    user_text = ""
    async for partial in stt_stream(audio_in):
        if cancel.is_set():
            return
        if partial.is_final:
            user_text = partial.text
            break

    # 2) Stream LLM tokens; buffer until a sentence boundary,
    #    then flush that sentence to TTS immediately.
    buffer = ""
    async for token in llm_stream(user_text):
        if cancel.is_set():              # barge-in: abandon this turn
            return
        buffer += token
        if SENTENCE_END.search(buffer):
            sentence, buffer = split_first_sentence(buffer)
            await tts_out.speak(sentence)   # first audio in ~90-200ms

    if buffer.strip():                   # flush the trailing fragment
        await tts_out.speak(buffer)

Turn-taking: VAD vs semantic endpointing vs model-based turn detection

Streaming answers "how fast can I reply?" Turn-taking answers the equally important "when should I reply?" — and getting it wrong is what makes an agent either cut the caller off mid-sentence or sit in silence after they have clearly finished. There are three broad approaches, and mature agents blend them.

Voice-activity detection (VAD)

The classic approach detects the presence or absence of speech energy and declares the turn over after a fixed window of silence — say 500-700ms of quiet. It is cheap, fast and language-agnostic, and it is a fine default. Its weakness is that it is deaf to meaning: it cannot tell the difference between someone who has finished and someone who is simply pausing to think, so a short silence timeout cuts off slow talkers and a long one adds dead air to everyone.

Semantic endpointing

Semantic endpointing layers linguistic understanding on top of VAD. Instead of ending the turn on silence alone, it asks whether the utterance sounds complete — has the caller reached a grammatical or intonational stopping point? "My account number is…" followed by a pause is clearly unfinished; "…and that is the problem" is clearly done. This lets the agent wait patiently through mid-thought pauses and respond promptly once the caller has genuinely finished, which is the single biggest lever on perceived naturalness.

Model-based turn detection

The newest generation folds turn-taking into a dedicated model trained on conversational audio. Deepgram Flux, for instance, posts a median end-of-turn detection under 300ms — fast enough that the detection itself is not a bottleneck. Model-based detection generalises across accents and speaking styles better than a hand-tuned silence timeout, which matters enormously across the range of English spoken by callers in India and the UK. Whichever you choose, evaluate it as rigorously as you evaluate the rest of the agent; our guide on evaluating multi-turn conversational agents in production covers how to score turn-taking and interruption behaviour rather than just final answers.

Barge-in done right: the 150ms stop-flush-reroute loop

Barge-in is the ability of the caller to talk over the agent and have the agent shut up and listen — exactly as you would if someone interrupted you mid-sentence. It is the feature that most separates an agent that feels human from one that feels like an answering machine, and it is unforgiving of sloppy engineering because the requirement is a hard, low number.

The rule is this: on a genuine interruption, the agent must stop speaking within about 150ms, flush the text-to-speech buffer so no already-generated audio keeps playing, and route the new utterance back to speech-to-text as a fresh turn. All three parts matter. Stopping late feels rude; the caller has started talking and the agent is still droning on. Failing to flush the buffer is worse — the agent goes quiet for a moment and then, maddeningly, resumes an old sentence because buffered audio was still queued. And failing to reroute means the interruption is heard but not acted on. Doing this well needs two capabilities working together: semantic VAD to decide that the incoming sound is a real utterance rather than a cough, a laugh or a backchannel "mm-hmm", and instant TTS cancellation so the in-flight audio is killed rather than drained.

The handler below shows the loop. It fires when speech is detected during agent playback, checks that the sound is a real utterance, cancels synthesis and flushes the player in the same breath, aborts the current turn, and starts a fresh STT stream on the interrupting audio.

# Barge-in: when the caller starts talking while the agent is
# speaking, stop within ~150ms, flush the TTS buffer, and route
# the new audio back to STT. Semantic VAD decides "real speech"
# versus a cough or backchannel ("mm-hmm").

async def on_user_speech_detected(event, session):
    if not session.agent_is_speaking:
        return
    if not is_real_utterance(event):     # semantic VAD gate
        return                           # ignore coughs / backchannels

    # 1) Stop playback + cancel synthesis NOW (target < 150ms).
    await session.tts.cancel()           # kill in-flight synthesis
    session.player.flush()               # drop buffered PCM frames

    # 2) Abort the current LLM/TTS turn cleanly.
    session.cancel.set()

    # 3) Reroute the interrupting audio into a fresh STT stream.
    session.cancel = asyncio.Event()
    await session.start_turn(event.audio)

Notice how this ties back to the streaming turn from earlier: both handle_turn and the LLM loop check the shared cancel event, so setting it here makes the abandoned turn unwind on its own. That shared-cancellation pattern is what keeps barge-in clean — you never leave an orphaned LLM stream writing to a TTS engine that the caller has already talked over.

Cascaded pipeline vs speech-to-speech models

Everything so far assumes a cascaded pipeline — three separate stages you wire together. The 2026 alternative is a speech-to-speech (S2S) model that takes audio in and emits audio out in a single network, handling recognition, reasoning and synthesis internally. S2S can have a lower latency floor because there are no vendor boundaries to cross, and it can capture tone and interruption more naturally. But the choice is a genuine fork with real trade-offs, and for most production teams the answer in 2026 still leans cascaded.

The cost picture is the headline. Cascaded pipelines are cheap and, more importantly, predictable: roughly $0.0095-0.17 per minute, and you know which stage every fraction of a rupee or penny goes to. Speech-to-speech is a lottery by comparison — the spread across the field is about 182x, running from around $0.00165 per minute for Gemini 2.0 Flash Live up to about $0.30 per minute for OpenAI Realtime GPT-4o, and crucially the cost grows as the conversation lengthens because the audio context keeps expanding. A five-minute call that starts cheap can end expensive.

The reliability caveat is the one that keeps cascaded ahead for production. On the Full-Duplex-Bench-v3 benchmark, Gemini Live 3.1 had the fastest latency of the field at 4.25 seconds — but it produced no speech at all in 22% of examples, and in 86% of those silent cases it still fired tool calls. Think about what that means for a live agent: roughly one turn in five, the caller hears nothing while the system quietly triggers an action in the background. In a cascaded pipeline that failure is visible — you have a transcript showing the model returned empty, and you can catch it, retry it or fall back. In an opaque S2S model it is a silent, hard-to-reproduce gap. When something goes wrong at 2am on a support line, the ability to read exactly what the STT heard, what the LLM decided and what the TTS spoke is worth a great deal.

Telephony vs web, and cost-per-minute planning

Where the audio comes from changes both your latency and your bill, and this is where the dual reality of an Indian and a UK deployment shows up most sharply. A web caller connects over WebRTC, which gives you wideband Opus audio, low jitter and a short path to your servers. A telephone caller comes in over the public switched telephone network, which typically hands you narrowband, 8kHz audio through a carrier — and that has two consequences. First, the telephony leg adds a network hop, pushing you towards the upper end of that 50-200ms hop budget. Second, narrowband phone audio is simply harder to transcribe accurately than clean wideband, so your STT accuracy — and therefore your whole agent's reliability — is lower on the phone than in a browser demo. Test on real phone lines early; an agent that shines on a laptop microphone can stumble badly on a call from a moving train.

The telephony provider layer differs by market. In India you will typically front your agent with a cloud-telephony provider — the likes of Exotel, Ozonetel, Knowlarity or Plivo — or a global carrier, and you will care about local number provisioning and the regulatory rules around outbound calling. In the UK the same role is played by providers such as Twilio, Vonage or a BT-backed carrier, with Ofcom's rules on calling line identity and nuisance-call regulation shaping what you can do. The engineering pattern is the same in both places, but the compliance and the per-minute carrier economics are not, so plan them per market rather than assuming your Indian setup ports cleanly to Britain or vice versa.

Region placement is the lever you control. Every time the audio crosses a continent you add real milliseconds, so co-locate your STT, LLM and TTS endpoints in the region nearest the caller — a Mumbai (ap-south) region for Indian traffic, a London (eu-west) region for UK traffic — and keep the vendor boundaries in that same region wherever you can. A pipeline whose three stages all live in London will comfortably beat one that recognises in London, reasons in a US data centre and synthesises back in Europe, purely on hop count. If you are serving both markets, run two regional deployments rather than one central one; the latency you save is the latency the caller feels.

On cost, the model is usually the cheaper half once you are at scale, and the numbers are worth committing to memory. As of pricing in July 2026, OpenAI's Realtime model (gpt-realtime) prices audio at roughly $32 per million input tokens and $64 per million output tokens, which works out to a real cost of about $0.18-0.46 per minute uncached, dropping to roughly $0.05-0.10 per minute with prompt caching, and around $0.30 per minute all-in for a typical input/output split. At mid-2026 exchange rates — roughly ₹85 and £0.79 to the US dollar — that headline $0.30 per minute is about ₹25 or £0.24 per minute, while a lean cascaded turn at the bottom of its $0.0095 range is under ₹1 or a penny a minute. Prompt caching is the biggest single lever: caching the system prompt and stable context can more than halve the per-minute cost, which is why it belongs in your architecture from day one rather than as a later optimisation.

Two further levers are worth naming. If your volume is high and your privacy or unit-economics constraints are tight, self-hosting the LLM stage can undercut per-minute API pricing substantially — the trade-off is that you now own serving, batching and quantisation, which our guide on self-hosted LLM serving cost, quantisation and batching unpacks. And if your agent calls tools — checking an order, booking a slot, taking a payment — every tool call adds to the LLM stage's latency, so design those tools to be fast and to fail gracefully; our guide on designing tools for AI agents covers schemas, errors and retries that keep a tool call from blowing your turn budget.

The bottom line

A realtime voice agent is a latency-engineering problem wearing a conversational costume. The perceptual budget is fixed by human biology — sub-800ms feels natural, past 1,500ms feels broken — and no clever prompt buys it back. You hit the budget by streaming every stage so the pipeline overlaps instead of running in series, turning a 1,400ms sequential turn into a 500-700ms streamed one, and by treating barge-in as a hard 150ms stop-flush-reroute loop rather than an afterthought. Choose cascaded over speech-to-speech unless conversational feel genuinely outweighs traceability, because the per-stage transcript is what lets you debug the call that goes wrong at 2am and because the cost is predictable rather than a 182x lottery. Place your STT, LLM and TTS in the region nearest the caller — Mumbai for India, London for the UK — cache your prompts, and test on real phone lines, not laptop microphones. Do all that and you have an agent that a caller in Mumbai or Manchester talks to the way they would talk to a person: quickly, and without ever feeling they are waiting for a machine to catch up.