How to Reduce Voice AI Latency
2 min read · Bolcho
The number-one reason a voice agent feels robotic isn't the voice — it's the pause before it speaks. Most stacks chain three separate HTTP calls (speech-to-text, then the LLM, then text-to-speech), and each hop stacks round-trips until the caller waits 3–5 seconds and starts talking over the bot. This guide breaks down exactly where that time goes and how a streaming pipeline collapses it to under 800ms — the threshold where a phone conversation starts to feel human.
Where the seconds go
In a naïve request-chained stack, every turn pays four separate taxes before the caller hears anything. The fix isn't a faster model — it's removing the waiting between stages.
The five moves that cut latency
Ordered by impact — the first two matter most.
- 1
Stream, don't chain
Use WebSockets/WebRTC so partial transcripts feed the LLM and partial tokens feed TTS. Stages overlap instead of finishing one before the next starts.
- 2
Start TTS on the first tokens
Begin synthesising audio as soon as the first few words of the reply exist, rather than waiting for the full sentence.
- 3
Use endpointing / VAD well
Detect end-of-speech quickly so you don't add a second of dead air waiting to be sure the caller finished.
- 4
Cache repeated phrases
Greetings, disclaimers and confirmations can be pre-synthesised and served from a TTS cache instead of re-generated every call.
- 5
Put compute near the caller
Regional nodes cut the network round-trips that a globally-hosted, US-routed stack pays on every turn for Indian calls.
How Bolcho gets under 800ms
Bolcho was built streaming-first: a single WebSockets/WebRTC pipeline where STT, LLM and TTS overlap, phrase caching for repeated audio, and India-side routing. The result is a time-to-first-word under 800ms without you tuning any of it.
Sub-800ms
Time-to-first-word, out of the box
1 pipeline
Unified stream, not 3 chained HTTP calls
<200ms
Mid-call language switch, no added lag
You don't tune this
Streaming, endpointing and caching are the default on Bolcho — you write the prompt, we keep the conversation under 800ms.
Build it on Bolcho
Everything in this guide ships out of the box. Spin up an agent in minutes, free.
Frequently asked questions
What is a good latency for voice AI?
Aim for under 800ms time-to-first-word — the point at which a phone conversation feels natural. Above ~1.2s callers start to notice the pause; above 2s they talk over the agent or hang up.
Why is my voice agent so slow?
Almost always because it chains three separate HTTP calls — STT, then LLM, then TTS — so round-trips stack up to 3–5 seconds per turn. Streaming the stages so they overlap is the fix.
Does a faster LLM fix latency?
Only partly. The biggest wins come from streaming (overlapping stages), starting TTS on the first tokens, good endpointing, and phrase caching — not just a quicker model.
How does Bolcho stay under 800ms?
A unified WebSockets/WebRTC streaming pipeline where STT, LLM and TTS overlap, TTS phrase caching for repeated audio, and India-side routing — all on by default.