ElevenLabs Conversational AI for Outbound Calls: Build Notes

Outbound voice agents rarely fail on voice quality. That is the one part ElevenLabs has genuinely solved. They fail because the agent talks over the customer, because the trunk garbles DTMF, because the call outcome never reaches the CRM, or because nobody checked whether you were allowed to dial that number in the first place.

These are build notes from shipping ElevenLabs conversational AI outbound calling against real phone numbers. I am covering the load-bearing parts: telephony, agent config, latency, barge-in, CRM writeback, compliance and cost. Vendor dashboards get redesigned every quarter. The architecture underneath does not.

What is ElevenLabs conversational AI outbound calling?

ElevenLabs conversational AI outbound calling is a hosted voice agent that dials a number, holds a real-time spoken conversation, and hands you a transcript plus structured results at the end. ElevenLabs orchestrates speech-to-text, the language model and text-to-speech behind one WebSocket session so you are not stitching three vendors together yourself.

Think of the system as three layers. The agent holds the prompt, voice, tools and knowledge base. The phone number binds that agent to the PSTN, either through the native Twilio integration or a SIP trunk you own. The caller is your own code, a batch calling job, or an automation platform triggering a dial.

"Nobody has ever cancelled a voice AI project because the voice sounded slightly synthetic. They cancel because the agent interrupted a customer mid-sentence and the CRM said the call went fine."

Native Twilio integration or your own SIP trunk?

Use the native Twilio integration for pilots and low volume, and bring your own SIP trunk once you need carrier control, existing numbers, or a specific route. ElevenLabs supports both, and batch calling works either way, so this decision is reversible.

The ElevenLabs SIP trunking docs are specific about the boundary conditions. Their SIP deployment sends and receives audio as G711 at 8kHz or G722 at 16kHz, so your trunk must support that natively or resample. Signalling authenticates either by digest credentials or by IP allowlist, with digest recommended. Media encryption can be disabled, allowed or required, and required plus TLS transport is the configuration to aim for.

Whichever path you take, test DTMF early. Outbound calls hit IVR menus constantly, and a trunk that mangles tones ruins every call that reaches a phone tree.

What actually goes into the agent config?

Scope one agent to one job. The single biggest quality win in outbound voice is refusing to build a general-purpose agent that qualifies leads, books appointments, answers billing questions and handles complaints in one prompt.

Dynamic variables are the piece people underuse. Passing the CRM record into the call at creation time removes an entire class of hallucination, because the agent never has to guess the account status, the last order, or the appointment slot it is confirming.

If you want a visual way to lay out branches and handoffs before writing code, that is the problem I built AgentFlow for. Call flows are easier to argue about on a canvas than in a prompt file.

What latency budget does an outbound voice agent need?

Measure one number: time from the moment the caller stops speaking to the first audio byte coming back. In the builds I have shipped, once that gap stretches past roughly a second, people start repeating themselves or saying hello, and the conversation degrades from there.

Chase the tail, not the average. A healthy median hides the turns where a slow CRM lookup left the caller listening to dead air.

The single cheapest latency fix is prompt length. Every turn re-sends your system prompt, so a bloated prompt taxes every single response for the entire call. Cut it, then measure again before touching anything else. The same discipline applies to model selection generally, which I go into on the OpenAI integration page.

How do you stop the agent talking over people?

Interruption handling is a threshold problem, not a model problem. The agent must stop speaking the instant the caller genuinely starts, while ignoring the noises that are not real turns.

There is also a design answer, not just a config answer. Short agent turns are interrupted less, because the caller gets a chance to speak naturally. If your agent monologues for fifteen seconds, no amount of barge-in tuning saves the call. The same asymmetry shows up in text channels, which I unpacked in AI chatbot vs live chat.

How do call outcomes get into the CRM?

Through a post-call webhook that writes a structured outcome, not a transcript dump. Define the outcome taxonomy before you build the agent, because the taxonomy is what your sales team will actually filter on.

A minimal FastAPI handler for that webhook. Verify, normalise, suppress, then write.

from fastapi import FastAPI, Request, HTTPException

app = FastAPI()

OUTCOMES = {
    "qualified", "not_interested", "callback",
    "voicemail", "no_answer", "wrong_number", "do_not_call",
}

@app.post("/webhooks/call-completed")
async def call_completed(request: Request):
    raw = await request.body()
    if not valid_signature(request.headers, raw):
        raise HTTPException(status_code=401, detail="bad signature")

    event = await request.json()
    conversation_id = event["conversation_id"]
    if already_processed(conversation_id):
        return {"status": "duplicate"}

    fields = event.get("analysis", {}).get("data_collection", {})
    outcome = fields.get("outcome", "no_answer")
    if outcome not in OUTCOMES:
        outcome = "no_answer"

    if outcome == "do_not_call":
        suppress(event["to_number"])

    upsert_crm(
        phone=event["to_number"],
        outcome=outcome,
        transcript_url=event.get("transcript_url"),
        duration_seconds=event.get("call_duration_secs", 0),
    )
    mark_processed(conversation_id)
    return {"status": "ok"}

If your CRM is GoHighLevel, the writeback is a pipeline stage move plus a tagged note, and the callback outcome should create the follow-up task automatically. I have built that loop before on the GoHighLevel AI calling platform, and the mechanics of the API side are on the GoHighLevel integration page.

Is AI outbound calling legal? TCPA, consent and disclosure

In the United States, an AI-generated voice on an outbound call is treated as an artificial voice under the TCPA. The FCC adopted a Declaratory Ruling on 2 February 2024, released on 8 February, confirming this, which means calls using AI voices require prior express consent from the called party, and telemarketing calls require that consent in writing. This is not legal advice, and you should have counsel review your specific programme.

Treat compliance as code, not as a policy document. Every control below is something the system enforces at dial time.

Rules vary by jurisdiction and keep moving. UK and UAE programmes carry their own direct marketing constraints, and several US states layer requirements on top of the federal baseline. Make the controls configurable per market.

How much does ElevenLabs outbound calling cost per minute?

Cost is a stack of meters, not one number. You pay for agent conversation minutes, possibly for model tokens, for carrier minutes on the telephony leg, and for number rental. Check the official ElevenLabs pricing page and your carrier's rate card before you budget, because both move.

The number that matters is cost per completed outcome, not cost per minute. Multiply cost per connected minute by average handle time, then divide by your connect rate and your outcome rate. A campaign with an eleven percent connect rate is paying for roughly nine dialling attempts for every conversation it gets, and the attempts that never connect still cost carrier fees.

This is also where duration caps pay for themselves. Cutting average handle time by twenty seconds across ten thousand calls removes over fifty hours of billed audio, which is a bigger saving than any vendor negotiation you are likely to win.

What I would tell you before you start

Run the pilot on a list you already have consent for, with one narrow objective, and a human listening to every call for the first two days. Everything you learn in those two days is worth more than a month of prompt iteration.

Outbound voice is one of the few AI builds where the model is the easy part. Scoping and shipping this is what my AI engineering services cover, and the same architecture drives the sales automation I build for SaaS teams. Still deciding whether voice is the right channel at all? Start with how to choose an AI chatbot for a small business.