A voice AI agent is judged in the first two seconds. If the reply arrives quickly and sounds natural, callers keep talking. If there is a pause, they say hello twice and ask for a person.
That makes voice unlike every other support channel. Text forgives a slow answer, whereas a phone line does not. This guide covers the latency budget, the design rules, the honest cost per minute and the cases where voice genuinely beats chat.
What a voice AI agent actually does on a call
The pipeline is short. Audio streams in, the system decides the caller has finished a thought, speech becomes text, retrieval finds the relevant passage, the model drafts a reply, and synthesis speaks it back.
Each stage seems fast on its own. Together, however, they easily exceed two seconds, and that is where most deployments feel robotic. Therefore the engineering work is mostly about overlapping stages rather than making any single one quicker.
The latency budget every voice AI agent has to respect
Aim for under 900 milliseconds from the end of the caller’s sentence to the first sound of the reply. Above 1.2 seconds, callers start filling the silence themselves.
| Stage | Typical budget | How to protect it |
|---|---|---|
| Endpointing | 200 to 400 ms | Tune the silence threshold per language |
| Speech to text | 150 to 300 ms | Stream partial results, do not wait for final |
| Retrieval | 80 to 200 ms | Warm the index, cap the candidate set |
| Model first token | 300 to 700 ms | Short prompts, streaming output |
| Speech synthesis | 150 to 300 ms | Start speaking on the first clause |
Barge-in is the feature callers notice first
People interrupt. They interrupt machines more than they interrupt humans, especially when the reply starts with a preamble. So the agent must stop speaking the moment the caller talks, and it must keep the partial answer in context.
Without barge-in, callers talk over the system and the transcript becomes noise. With it, conversations feel ordinary. This one behaviour separates a usable voice AI agent from a demo.
Accents, names and numbers
Speech recognition handles conversational Indian English reasonably well now, although accuracy still drops on three things: proper nouns, alphanumeric identifiers and code-mixed sentences. Order numbers are the classic failure.
The fix is design, not model choice. Read identifiers back for confirmation. Constrain the expected format when you know it. Offer a keypad fallback for anything longer than eight characters. Our piece on the multilingual AI chatbot covers the code-mixing problem in more depth.
Grounding matters more on voice, not less
On a web page, a customer can see the source link under an answer. On a call they cannot, so they simply believe what they hear. That raises the cost of a wrong answer considerably.
For that reason the retrieval layer should be strict. If nothing relevant is found, the agent says so and offers a transfer. Citations still matter internally, because reviewers need to audit calls afterwards. We covered that discipline in AI chatbot citations.
When a voice AI agent beats chat, and when it does not
Voice wins when hands are busy, when typing is friction, when the phone is the only channel a customer has, or when urgency is high. Chat wins when the answer needs a link, a list or a written record.
Most teams need both, which argues for one knowledge base behind every channel. Otherwise the phone line and the website drift apart, and customers notice. Our guide to the WhatsApp AI chatbot makes the same argument for messaging.
What a voice AI agent costs to run
| Cost line | Driver | Notes |
|---|---|---|
| Telephony minutes | Inbound and outbound legs | Cheap in India, still per minute |
| Speech to text | Audio minutes processed | Streaming costs more than batch |
| Model usage | Turns per call | Short prompts keep this small |
| Speech synthesis | Characters spoken | Premium voices cost noticeably more |
| Human fallback | Transfer rate | Often the largest line early on |
Because the per-minute cost is real, containment targets matter here. A call that transfers after ninety seconds costs more than one answered by a person immediately, so a poorly tuned deployment can raise cost rather than lower it. The measurement discipline in our guide to AI chatbot ROI applies directly.
Handover rules for a voice AI agent
Transfers should be warm and rare, in that order. Warm means the agent receives a short summary, the caller identity and any action already taken. Rare means the abstain rule is tuned so the system does not give up too early.
- Transfer immediately on any request for a human, without a retention loop.
- Transfer when retrieval confidence falls below the threshold twice in a row.
- Transfer on distress signals, billing disputes and anything safety related.
- Never transfer silently, since callers assume the line dropped.
Consent, recording and DPDP
Call recordings are personal data, and voice itself can identify a person. Consequently you need a clear notice at the start of the call, a retention period you actually enforce and restricted internal access to transcripts.
Redaction helps too. Strip card numbers and identifiers before storage rather than afterwards. Our overview of enterprise AI chatbot security and compliance covers the residency and retention questions in detail.
How to evaluate a voice AI agent before launch
Vendor demos use clean audio and cooperative callers. Your evaluation should not. Record real calls, replay them through the system, and grade the outcomes.
- Test with background noise, poor signal and speakerphone audio.
- Include callers with regional accents and code-mixed speech.
- Interrupt deliberately to check barge-in behaviour.
- Measure ninety-fifth percentile response time, not the average.
- Grade fifty transcripts by hand before you trust any dashboard.
A 60-day rollout that does not embarrass anyone
Start with one call type, ideally a high-volume informational one such as order status or branch timings. Keep the scope narrow until the latency and accuracy numbers hold.
- Weeks one and two: choose the call type, prepare the content, set the abstain rule.
- Weeks three and four: run internal calls only, then fix what recordings reveal.
- Weeks five and six: route ten percent of live traffic, with a fast path to agents.
- Weeks seven and eight: widen scope only after resolution and transfer quality hold steady.
Common voice AI agent mistakes
- Long scripted greetings, which burn the caller’s patience before anything useful happens.
- No barge-in, so the conversation turns into two monologues.
- Reading long lists aloud instead of offering to send a message.
- Hiding the transfer option, which pushes callers to hang up and redial.
- Optimising average latency while the slowest calls stay unbearable.
Where Intellowork fits
Intellowork treats voice as one channel over a shared knowledge base rather than a separate product. The same grounded content answers on the web widget, on WhatsApp, in Slack and Microsoft Teams, and on a call, so a fix in one place applies everywhere.
Retrieval is permission-aware, answers carry their source for internal review, and unresolved calls transfer with the full context an agent needs. You can see the platform at Intellowork or work through our enterprise chatbot evaluation checklist first.
Frequently asked questions
How fast does a voice AI agent need to reply?
Under 900 milliseconds feels natural, and beyond 1.2 seconds callers start interrupting. Measure at the ninety-fifth percentile, because the slow tail is what people remember.
Can it handle Hindi and English on the same call?
Yes, provided the speech model supports code-mixed input and retrieval is multilingual. Accuracy still drops on product names, so keep a glossary and confirm identifiers back to the caller.
Do we need to replace our contact centre platform?
Usually not. Most deployments sit in front of the existing telephony stack and transfer into it, which keeps routing, recording and reporting where they already are.
Is voice more expensive than chat?
Per contact, yes. Telephony, speech recognition and synthesis all bill by usage. The case rests on resolution and on reaching callers who will never use another channel.
What should the agent do when it is unsure?
Say so plainly and offer a transfer. Guessing on a call is worse than guessing in chat, since the caller has no visible source to check.
How do we test accents properly?
Replay your own recorded calls rather than reading scripts. Real audio carries the noise, overlap and regional pronunciation that scripted tests never reproduce.
The takeaway
A voice AI agent succeeds on timing and honesty. Keep the response under a second, let callers interrupt, ground every answer in real content, and transfer cleanly when the system is unsure. Get those four right and the channel works. Miss any one of them and callers will simply ask for a person.
Related reading
- AI chatbot for Slack and Teams
- Benefits of AI chatbots in customer service
- Build versus buy for an enterprise AI chatbot
- AI copilots versus AI agents
Written by the Exuverse team, led by Tarun Gupta, who builds enterprise retrieval and assistant systems for Indian and global teams.