Self-Hosting an AI Voice Agent: Twilio, Media Streams, and the Latency Budget

The demo is easy. A voice agent that answers a call, transcribes, thinks and speaks back is a weekend project. What is hard is making it feel like a phone call instead of a walkie-talkie, and keeping it that way at three in the morning when a speech provider has a bad region. This post is about how to self host an AI voice agent properly: the real architecture, the latency budget you are fighting, and the operational tax nobody quotes you up front.

I have built this both ways, on managed platforms and from scratch on Twilio Media Streams with my own STT, LLM and TTS chain. The engineering is genuinely interesting and genuinely unforgiving. Miss the budget by 300ms and callers start talking over the agent; get turn detection wrong and it interrupts itself. Let me walk through where the milliseconds go and how to spend them.

When should you self-host instead of using Retell or Vapi?

Most teams should not self-host on day one. Platforms like Retell, Vapi and ElevenLabs' Conversational AI already solved turn-taking, barge-in and reconnection, and they charge a per-minute markup for it. If you just need an AI voice receptionist that books appointments, a managed platform gets you live in a week. I compared the three in detail in Retell vs Vapi vs ElevenLabs, and for the majority of use cases one of them is the right answer.

You self-host when a managed platform actively blocks you. That is a shorter list than vendors of custom builds would like you to believe.

"Self-hosting a voice agent is not a cost saving until you are at the scale where the per-minute markup outweighs an engineer's on-call pager."

What is the real architecture of a self-hosted voice agent?

The whole system is a loop that turns inbound audio into outbound audio fast enough to feel conversational. Twilio answers the PSTN call and, via a Media Streams instruction in your TwiML, opens a WebSocket to your server and starts forwarding the caller's audio as base64-encoded frames. From there you own everything.

The critical design decision is that this is all streaming, all the time. The naive version records the caller, waits for silence, sends the whole clip to STT, waits, sends the whole transcript to the LLM, waits, synthesises the whole reply, then plays it. That serial pipeline stacks every stage's latency end to end and feels like a satellite delay. The streaming version overlaps them: TTS for the first sentence starts while the LLM is still generating the third.

The latency budget: where the milliseconds go

Human turn-taking in conversation has gaps of around 200ms. You will not hit that over the phone, but you should target sub-800ms from the moment the caller stops speaking to the moment they hear the first audio of the reply. Anything past about 1.2 seconds and callers assume the line dropped and start talking again. Here is a realistic budget for a well-built self-hosted stack.

Notice the total is less than the naive sum. That is the entire point of streaming: turn detection, STT finalisation and the first LLM token overlap, and TTS for sentence one starts before the LLM finishes sentence two. The single biggest lever is LLM time-to-first-token, which is why voice agents almost always use a faster, smaller model than you would pick for a chat product. A 200ms improvement in TTFT is worth more than any prompt cleverness.

Handling barge-in and interruptions

Barge-in is the feature that separates a real voice agent from a demo. When a caller interrupts, the agent must stop talking immediately, discard the rest of its planned reply, and listen. If it keeps speaking over the caller for even half a second, the illusion collapses and people hang up.

Mechanically, you keep the STT stream open the entire time, including while the agent is speaking. When STT reports the caller has started a new utterance during agent playback, you do three things at once: send Twilio a `clear` message to flush any audio it has buffered but not yet played, cancel the in-flight LLM and TTS generations so they stop consuming tokens, and truncate the conversation history to reflect only what the caller actually heard. That last step matters more than people expect. If the agent planned a three-sentence answer but was cut off after one, the model's memory must record only the sentence that was actually spoken, or it will reference things the caller never heard.

Turn detection and VAD: the unglamorous hard part

Voice activity detection tells you whether someone is speaking. Endpointing tells you whether they are finished. These are different problems and conflating them is why home-grown agents interrupt people mid-sentence. A fixed timer of, say, 500ms of silence equals end-of-turn will cut off anyone who pauses to think and will feel sluggish for people who speak in short bursts.

The pragmatic approach is to lean on your STT provider's built-in endpointing, which uses acoustic and linguistic cues rather than a raw silence timer, and then layer a small amount of your own logic on top: a longer grace period after a filler word like "um", and a shorter one after a clearly complete sentence. Getting this right is largely a tuning exercise against recordings of real calls, not something you solve once in code. Every domain speaks differently, and a receptionist for a law firm needs different endpointing from an outbound survey bot.

A minimal Media Streams WebSocket handler

Here is the skeleton of the Twilio side of the loop in Python with FastAPI. It shows the message shapes Twilio actually sends and how you push audio back and clear the buffer on barge-in. I build these backends the way I describe in FastAPI for AI agents, because the async model maps cleanly onto juggling three concurrent WebSockets.

import base64
import json
from fastapi import FastAPI, WebSocket, WebSocketDisconnect

app = FastAPI()

@app.websocket("/media")
async def media_stream(ws: WebSocket):
    await ws.accept()
    stream_sid: str | None = None

    async for raw in ws.iter_text():
        msg = json.loads(raw)
        event = msg["event"]

        if event == "start":
            # Twilio opened the stream; grab the SID we need to talk back.
            stream_sid = msg["start"]["streamSid"]

        elif event == "media":
            # 20ms of 8kHz mu-law audio, base64-encoded. Forward to STT.
            chunk = base64.b64decode(msg["media"]["payload"])
            await stt.send(chunk)

        elif event == "stop":
            break

    # --- helpers used by the agent loop -------------------------------

    async def speak(mulaw_8k: bytes) -> None:
        """Stream a chunk of TTS audio back to the caller."""
        await ws.send_text(json.dumps({
            "event": "media",
            "streamSid": stream_sid,
            "media": {"payload": base64.b64encode(mulaw_8k).decode()},
        }))

    async def barge_in() -> None:
        """Flush Twilio's playback buffer the instant the caller interrupts."""
        await ws.send_text(json.dumps({
            "event": "clear",
            "streamSid": stream_sid,
        }))

The `clear` event is the load-bearing detail most tutorials omit. Without it, Twilio keeps playing whatever it has already buffered even after you stop generating, so the caller hears the agent talk over them for a second after they interrupt. In production this handler is wrapped in reconnection logic, a heartbeat, and a supervisor that ties together the STT, LLM and TTS streams as concurrent tasks.

Function calling: booking and lookups mid-call

A voice agent that only talks is a novelty. The value is in doing things: checking availability, booking an appointment, looking up an order. That is function calling, and voice adds a latency constraint chat does not have. A calendar lookup that takes two seconds is invisible in a chat window and excruciating on a phone call, because the caller is sitting in silence.

The technique that saves you is a filler utterance. When the LLM decides to call a tool, you immediately synthesise something like "let me check that for you" and play it while the tool runs in the background. It buys 1.5 to 2 seconds of wall-clock time without dead air, which is usually enough for a calendar or CRM lookup. Keep the tools themselves fast and idempotent; a booking call that the caller triggers twice because they repeated themselves must not create two appointments. The prompt discipline for this is its own subject, which I cover in prompt engineering for voice AI agents.

Scaling and cost

Each concurrent call holds three open WebSockets and a slice of an event loop. This is I/O-bound, not CPU-bound, so a single modern instance handles a surprising number of concurrent calls, but you are gated by the connection limits and rate limits of your STT, LLM and TTS providers long before you run out of CPU. Plan for provider quotas first.

The honest accounting is that raw provider cost per minute for a self-hosted stack often lands below a managed platform's price, but the gap is smaller than it looks once you load in the engineering and on-call time. Managed platforms are not charging you for compute; they are charging you for the pager. Whether that trade favours self-hosting is a scale question, and the answer flips somewhere in the tens of thousands of minutes per month.

The operational burden you are signing up for

This is the part that gets glossed over. When you self-host, every failure mode in the chain becomes yours. STT providers have bad regions. LLM APIs rate-limit under load. TTS latency spikes at peak hours. Twilio's WebSocket occasionally drops mid-call and you have to reconnect without the caller noticing. Your instance cold-starts and the first call of the hour takes two seconds to answer.

None of this is exotic, but it is real engineering with a pager attached. That is exactly the kind of production system I take on as full-stack Python development: the Media Streams glue, the streaming pipeline, the failover, and the observability that tells you when the latency budget slips. If you have a genuine reason to self-host and want it built to survive real traffic rather than a demo, that is the work. If you are not sure you have that reason yet, start on a managed platform and revisit when the per-minute maths forces your hand.