Most voice agents fail for one reason: someone designed a chatbot and bolted a speaker onto it. A text agent can think for four seconds and nobody notices. On a phone call, four seconds of silence means the caller says “hello, are you there?” and the turn collapses.
Voice is a real-time systems problem wearing an AI costume. The model matters far less than the plumbing around it. Below is the stack I build, the latency budget I hold it to, and the parts that break in production once real customers start calling.
What is an AI voice agent for customer support?
An AI voice agent is a pipeline that answers or places a phone call, transcribes the caller in real time, decides what to do, speaks back, and can hand the call to a human. It is not a single model. It is six or seven services chained together, each of which spends part of a shared latency budget.
That distinction matters commercially. When a vendor demos “one API for voice agents,” they are hiding the chain, not removing it. If you want to understand the general shape of agent systems first, my post on what AI agents actually are covers the definitions I use across every build.
What does the voice agent stack actually look like?
Seven layers, each with a distinct failure mode. If you cannot name which layer is causing a bad call, you cannot fix it.
The seven layers of a production voice agent, and how each one fails:
| Layer | Job | Typical choice | How it fails |
|---|---|---|---|
| Telephony | Carry PSTN audio to your socket | Twilio Programmable Voice with Media Streams | Dropped frames, jitter, one-way audio |
| ASR | Streaming speech to text with partials | Deepgram, OpenAI, ElevenLabs built-in | Accent and noise errors on 8kHz phone audio |
| Turn detection | Decide when the caller has finished | Semantic endpointing plus a VAD silence window | Cuts people off mid-sentence or waits too long |
| Reasoning | Pick an intent, call a tool, form a reply | OpenAI or Anthropic Claude with tight tool schemas | Slow first token, hallucinated policy |
| TTS | Stream audio back with low first-byte time | ElevenLabs streaming voices | Long time-to-first-byte, robotic prosody |
| Barge-in | Stop playback the instant the caller speaks | Server-side clear of the outbound audio buffer | Agent talks over an angry customer |
| Escalation | Warm transfer with context to a human | SIP or Twilio dial verb plus a CRM note | Cold transfer, caller repeats everything |
I keep this table on the wall of every voice project. When a call sounds wrong, the first question is never “which prompt”, it is “which of the seven”. That habit came out of the same debugging discipline I wrote up in lessons from running AI agents in production.
What latency budget does a voice agent need?
Budget the whole turn, not individual services. The number that matters is perceived gap: the silence between the caller finishing a sentence and hearing the first syllable back. Below roughly 800ms the call feels like a conversation. Past about 1.2 seconds callers start repeating themselves.
The per-turn latency budget I plan against on phone builds. These are engineering targets, not vendor guarantees; measure your own stack:
| Stage | Target | Notes |
|---|---|---|
| Telephony ingress | 20-80ms | Carrier plus WebSocket hop to your server |
| Streaming ASR finalisation | 100-200ms | After the endpoint is declared, not after speech |
| Turn detection window | 200-500ms | The single biggest lever you control |
| LLM time to first token | 200-600ms | Depends on model, prompt size, tool call depth |
| Tool call round trip | 0-400ms | Only if the turn needs live data |
| TTS time to first audio byte | 75-300ms | Use streaming synthesis, never wait for full audio |
| Telephony egress plus jitter buffer | 20-80ms | Playback smoothing on the carrier side |
| Perceived gap (target) | under 800ms | Sum of the path actually on the critical path |
Where the budget usually blows up
In my experience four things eat the budget, in this order:
- A fixed silence threshold set too high. Anything above 700ms of required silence makes the agent feel slow no matter how fast the model is.
- Waiting for a complete LLM response before starting TTS instead of streaming the first sentence as soon as it exists.
- Synchronous tool calls that hit a slow CRM in the middle of the turn rather than pre-fetching on call connect.
- Prompts that grew to several thousand tokens of policy text, which pushes time to first token up on every single turn.
"Nobody on a support call has ever complained that the AI was insufficiently intelligent. They complain that it interrupted them."
Why do turn detection and barge-in decide if the call feels human?
Because humans do not take turns on silence alone. We read intonation, grammar and context. A voice activity detector that fires after a fixed pause will cut off anyone who says “my account number is… hang on, let me find it.”
The fix is layered endpointing: a fast VAD for raw speech presence, plus a semantic check on the partial transcript to decide whether the sentence is actually complete. Trailing conjunctions, digits mid-sequence and filler words all argue for waiting longer. A finished question argues for firing immediately.
Barge-in is the other half. When the caller speaks over the agent, you must stop outbound audio within a couple of hundred milliseconds and, critically, truncate the conversation history to what was actually heard. Skip that truncation and the model believes it said three sentences the caller never received, which produces confidently wrong follow-ups.
Barge-in checklist I run before any voice agent goes live:
- Outbound audio buffer is cleared server-side, not just muted on the client.
- The transcript is trimmed to the audio actually played, using playback position rather than generation length.
- Short acknowledgements such as “mhm” and “yeah” do not trigger a full interruption.
- Background TV or a second voice in the room does not cancel the agent mid-sentence.
What I learned wiring ElevenLabs and Twilio for outbound calls
The integration itself is straightforward; the operational details are what cost time. ElevenLabs documents a native Twilio path where you import a Twilio number with its label, Account SID and auth token, and ElevenLabs then configures that number with the correct settings. It documents a separate custom-LLM path for running your own model, RAG pipeline or function-call routing on your server while the agent stays on the Twilio number.
The order I now build outbound voice agents in:
- 1Get a single call connecting end to end with a hardcoded greeting before adding any AI. If the audio path is wrong, nothing downstream is diagnosable.
- 2Log raw call recordings and full transcripts from day one, timestamped per stage, so you can attribute every awkward pause to a layer.
- 3Add streaming ASR and tune the endpointing window against real recorded calls, not against your own clean-microphone test calls.
- 4Put the LLM in with a deliberately small prompt and one or two tools. Add policy text only when a real call proves you need it.
- 5Wire barge-in and transcript truncation before you scale volume, because the failure only shows up with interrupting humans.
- 6Add escalation last, and test it on a live human queue during business hours before any dialler goes out.
Two Twilio-side details matter more than people expect. Phone audio is narrowband, so ASR accuracy on names, postcodes and reference numbers is meaningfully worse than on a laptop microphone; design confirmations that spell things back. And outbound calling carries compliance obligations around consent, caller ID registration and calling windows that vary by country. I keep those rules in the dialler layer, not the prompt.
I have built the call-flow side of this twice over: AgentFlow as a visual builder for AI agent workflows, and a GoHighLevel AI calling and SMS suite where the voice agent had to hand off cleanly into an existing CRM pipeline.
When should an AI voice agent escalate to a human?
Escalate early and cheaply. A transfer that happens in twenty seconds costs you almost nothing; a caller who fights the bot for four minutes and then transfers costs you the relationship.
My default escalation triggers, all evaluated per turn:
- Two consecutive turns where intent classification fails or confidence drops below threshold.
- The caller asks for a human, in any phrasing, at any point. No retention loop.
- Detected frustration or raised voice, which is cheap to approximate from ASR text plus interruption frequency.
- Anything touching payments, refunds above a set value, cancellations, or a legally binding commitment.
- Regulated or safety-critical domains where a wrong answer has real consequences, which is why the NHS-facing clinical work I did on Synthicare was built decision-support first, never autonomous.
Warm transfer beats cold transfer every time. Pass the transcript summary, the verified caller identity and the attempted actions into the agent screen-pop. The decision of how much autonomy to grant before a human steps in is the same judgement call I unpack in how to deploy AI agents in production safely.
How much do AI voice agents for customer support cost to run?
Cost scales with connected minutes, not with how many agents you configure. Deploying ten voice agents that each take five calls a day is cheaper than one agent taking a thousand. Check each vendor's official pricing page for current rates, because voice pricing changes frequently and per-minute bundles get repackaged often.
The cost structure has five components that all multiply by minutes:
- Telephony: per-minute inbound and outbound carrier charges, plus number rental, which varies sharply by country.
- ASR: usually per minute of audio processed, and you pay for silence too unless you gate on VAD.
- LLM: tokens per turn multiplied by turns per call multiplied by call volume. Long system prompts are a recurring per-turn tax.
- TTS: usually per character or per second of generated audio, so verbose agents cost more than concise ones.
- Engineering and monitoring: call recording storage, transcript review, and the human queue that receives escalations.
The honest comparison is not agent versus human on cost per minute. It is deflection rate at acceptable quality. If the agent fully resolves a third of routine calls and warm-transfers the rest without annoying anyone, the economics work even at unflattering per-minute rates. If it resolves nothing and irritates callers, it is a cost with a bad review attached. Deflection rate, not per-minute price, is the number to put in the business case.
What actually works for AI voice agents in customer support
Narrow scope, fast turns, honest escalation. Every voice deployment I have seen succeed did those three things and nothing clever.
What works, based on builds that survived contact with real callers:
- One job per agent. Order status, appointment booking, qualification, or triage. Not all four.
- Confirm slot values back to the caller before acting, especially anything numeric captured over narrowband audio.
- Pre-fetch customer context on call connect so the first turn is not blocked on a CRM lookup.
- Keep the system prompt short and push policy into retrieved snippets or tool responses.
- Review a sample of real recordings weekly. Transcript metrics hide prosody problems that recordings expose in ten seconds.
What to skip:
- Long scripted greetings. Callers barge in over them anyway, which immediately tests your interruption handling.
- Making the agent pretend to be human. It backfires the moment it fails, and disclosure rules exist in several markets.
- Chaining three models in series on the critical path because each one adds first-token latency to every turn.
- Launching an outbound dialler before inbound calls have run clean for a couple of weeks.
If you are scoping a voice deployment, the fastest path is a single high-volume intent, a two-week build, and a hard latency target measured from real calls. That is how I scope AI engineering engagements, and it is the same delivery pattern behind the voice and automation work I do for SaaS support teams, usually with OpenAI or Anthropic Claude on the reasoning layer.
Key takeaways
- A voice agent is a real-time pipeline of seven layers, not a single model, and each layer spends part of one shared latency budget.
- Perceived response gap under about 800ms feels conversational; past 1.2 seconds callers start repeating themselves.
- Turn detection and barge-in handling cause more bad-sounding calls than model quality ever does.
- Truncate conversation history to the audio actually played, or the model will reference sentences the caller never heard.
- Voice agent cost scales with connected minutes across telephony, ASR, LLM tokens and TTS characters, so measure deflection rate rather than per-minute price.
- Escalate to a human on repeated intent failure, explicit request, detected frustration, or anything financial or legally binding.
Frequently asked questions
- How much do AI voice agents for customer support cost?
- Cost is per connected minute, stacked across five layers: telephony, ASR, LLM tokens, TTS characters, and engineering. Vendors repackage voice pricing often, so check official pricing pages for current rates. The useful metric is cost per resolved call, which depends on your deflection rate far more than on any single vendor's per-minute price.
- What latency is acceptable for an AI voice agent on a phone call?
- Target under 800ms of perceived silence between the caller finishing and hearing the first syllable back. Above roughly 1.2 seconds, callers assume the line dropped and start talking again. Measure the perceived gap from real recordings, not the sum of vendor benchmark numbers, which exclude telephony and turn detection.
- Is ElevenLabs with Twilio a good stack for voice agents?
- Yes, for most support use cases. ElevenLabs documents a native Twilio integration where you import a Twilio number with its Account SID and auth token, and a separate custom-LLM path when you need your own model or RAG pipeline. Twilio Media Streams handles the raw call audio over WebSocket if you build the pipeline yourself.
- Can an AI voice agent handle angry customers?
- It should detect them and transfer, not handle them. Rising interruption frequency plus negative sentiment in the transcript is a cheap and reliable escalation trigger. Design the agent to hand off within two turns of detecting frustration, with a warm transfer that passes the transcript summary so the customer never repeats themselves.
- What is the difference between an IVR and an AI voice agent?
- An IVR routes on fixed menu options and keypresses. An AI voice agent understands free-form speech, calls tools mid-conversation, and can complete a task rather than just routing it. The tradeoff is determinism: IVR behaviour is fully predictable, while a voice agent needs guardrails, confirmations and escalation rules to stay safe.
- Do I need to tell callers they are talking to an AI?
- Disclose it. Several jurisdictions have rules on automated call disclosure and outbound consent, and the requirements differ between markets. Beyond the legal position, undisclosed AI backfires the moment the agent fails. A one-line disclosure in the greeting costs nothing and removes the worst failure mode.
- Can AI voice agents make outbound calls, not just answer them?
- Yes, and the technical path is nearly identical, but the compliance surface is much larger. Outbound calling involves consent records, caller ID registration, calling-window restrictions and do-not-call handling that vary by country. Keep those rules in the dialler layer where they are testable, never in the prompt.
- Is it worth building a voice agent instead of buying a platform?
- Buy first if a platform covers your intents. Build when you need your own model on the critical path, custom tool routing, or data that cannot leave your infrastructure. Most teams should start on a managed platform, measure deflection rate for a month, then decide whether custom engineering earns its cost.
Sources
Written by Syed Husnain Haider Bukhari
AI engineer, data scientist, and founder of Revolutionary Technologies LLC. Ships production AI agents, automations, and data platforms for teams in the US, UK, and UAE — including AgentFlow, AI Walay, and ProLeads.
Get in touch →Related pages
Want this built instead of researched?
Book a 30-minute scoping call. You get a one-page plan and a fixed-scope quote within 48 hours.
Start a conversation