How to Build a Voice AI Agent (Step by Step)

What a voice agent is really made of

Every voice agent is the same real-time loop: it hears the caller (speech-to-text), decides what to say (an LLM guided by your prompt and knowledge), speaks back (text-to-speech), and takes actions (tools) - all streamed over a phone line (telephony). The whole loop has to complete in well under a second, every single turn, or the conversation feels broken.

Build vs. buy: the honest tradeoff

You can assemble the pipeline yourself with open frameworks (LiveKit, Pipecat) or buy a managed platform. From-scratch gives you total control and the most work - you own latency tuning, failover, telephony compliance and every upgrade. A platform trades some control for weeks of your life back. Most teams that start fully DIY end up on a platform for everything except their core differentiator.

The right call depends on one question: is the voice pipeline your product, or a feature of it? If it's a feature, buy.

Build it in five steps

On a managed platform you configure the agent rather than assemble the stack. The five steps below are the whole job - the rest of this guide drills into the ones that decide whether the agent actually works.

The system prompt that does the work

The prompt is the agent. A vague prompt gives a vague agent; a tight one with a single goal, a clear call flow and hard guardrails gives an agent that behaves on every call. Here is a real, production-shaped prompt for a lead-qualification agent - copy it and fill in the braces.

Notice the structure: Role, Context, How to talk, Call flow, Hard rules, Tools. Keep replies short (it's a phone call), match the caller's language, say numbers as words, and never let it invent a price or go off-script. The 'hard rules' block is what keeps it safe in the real world.

# Role
You are Riya, a scheduling assistant for {Business}. You make and take
phone calls. Your one job: understand the caller's need and book a
qualified demo. You are warm, quick, and never robotic.

# Context
- {Business} sells {product}. Typical buyers: {who}.
- Working hours: Mon-Sat, 10am-7pm IST. Only ever book inside these.
- You cannot discuss anything outside {product}. If asked, say so and offer
  to connect a human.

# How to talk
- Keep replies to one or two short sentences. This is a phone call, not an essay.
- Match the caller's language. If they speak Hindi, speak Hindi. If they
  mix Hindi and English, mix only after they do. Never start in Hinglish.
- Say numbers and dates as words ("four thousand five hundred", "next Tuesday").
- One question at a time. Acknowledge ("Got it, thank you") and move on.

# The call flow
1. Greet, confirm you are speaking to the right person, ask if it's a good time.
2. Discovery - ask, one at a time: company size, current process, main problem,
   timeline, budget range, decision-maker.
3. If they qualify (>= {threshold}), offer two demo slots inside working hours.
4. Confirm the slot, say you'll send details on WhatsApp, thank them, end.

# Hard rules (guardrails)
- Never invent prices, features, or availability. If unsure, say you'll check.
- Never take payment or card details on the call.
- If the caller is upset, is the wrong person, or asks for a human - stop
  qualifying, apologise briefly, and hand off.
- If voicemail: leave a 10-second message with a callback number, then end.

# Tools
- book_demo(date, time, name, phone) - only after they agree to a slot.
- end_call() - after a clear goodbye, never mid-question.

Function calling, done right

Tools are how the agent does things instead of just talking. Each tool is a function with a name, a description the model reads to decide when to call it, and a typed schema for its arguments. The two rules that separate a demo from production: confirm before any irreversible action, and never block the conversation while a tool runs.

A real booking tool looks like this:

{
  "name": "book_demo",
  "description": "Book a demo slot. Call ONLY after the caller has agreed to a specific date and time inside working hours.",
  "parameters": {
    "type": "object",
    "properties": {
      "name":  { "type": "string", "description": "Caller's full name" },
      "phone": { "type": "string", "description": "E.164, e.g. +9198XXXXXXXX" },
      "date":  { "type": "string", "description": "YYYY-MM-DD, Mon-Sat only" },
      "time":  { "type": "string", "description": "HH:MM 24h IST, 10:00-19:00" }
    },
    "required": ["name", "phone", "date", "time"]
  }
}
// The platform calls your webhook with these args, waits for your JSON
// response, and the agent speaks a "let me just confirm that..." filler
// while it waits - so a 400ms booking call never sounds like dead air.

What a voice agent actually costs

This is the number every other guide dodges. A voice minute is the sum of four live services plus infra. Here is a realistic breakdown for a domestic Indian call on an efficient stack (Deepgram STT, a small fast LLM, Azure neural TTS, Plivo telephony). Prices move, so treat these as indicative - but the shape is right: telephony and TTS dominate, the LLM is the cheapest part.

A typical 2-minute call therefore costs roughly 5-6 rupees in raw provider cost. Caching repeated phrases (greetings, confirmations) and a 140-series origination can cut the telephony and TTS lines meaningfully at volume.

Latency: the sub-800ms budget

The single biggest reason agents sound robotic is lag. A human expects a reply within about 500-800ms of finishing their sentence. Miss that and callers talk over the agent or hang up. Latency isn't one number - it's a budget you spend across the loop, and streaming every stage is what keeps it in range.

The goal is to start speaking the reply before the LLM has even finished generating it - stream the first sentence to TTS while the rest is still being written.

Making it sound human, not robotic

Low latency gets you in the room; conversation dynamics make it believable. The models are good enough now - what trips callers' 'this is a bot' alarm is the turn-taking.

Indian languages and Hinglish

This is where generic, US-built stacks fall over - and where most of the India market actually is. Real callers greet in English, switch to Hindi to explain the problem, and drop in English product names. Your STT has to transcribe code-mixed speech, your LLM has to answer in the same register, and your TTS has to pronounce both scripts naturally.

The rule that makes it feel native: match the caller's language, and only start mixing Hindi and English after they do - never lead with Hinglish.

Test it before you dial out

Your first agent will have bugs you can't see from the config screen - they only show up on real calls. Run the hard scenarios yourself, then listen to the first 72 hours of real calls before you scale. Define success numerically before launch, not after.

Stay compliant (India)

Voice agents make real calls to real people, and India has rules. Getting this wrong is a bigger risk than any model bug.

Launch on Bolcho

Bolcho runs the whole loop for you - managed STT-LLM-TTS, telephony, Indian-language and Hinglish handling, caching and failover - so you configure the agent and go live, usually in minutes. You write the prompt; it handles the 80% nobody warns you about.

Paste the prompt above, add your knowledge and tools, attach a number, and make your first call today.