Is AI Calling Agent open source and free to use?

Yes. The repository is public on GitHub at revolutionarybukhari/ai-calling-agent and the README declares the MIT License, though the license file it links to was never committed, so GitHub reports no license. You can clone it, install the requirements and run it today. It has 47 stars and 18 forks as of July 2026.

How does it handle interruptions when the caller talks over the bot?

It does not, and that is a property of this architecture. Replies are synthesised to an MP3 and played by Twilio, so the microphone reopens only after playback finishes. True barge-in requires a bidirectional media stream instead of the play-then-Gather loop this repo uses. That is the main limitation to understand before adopting it.

What does it cost to run?

The repo has no pricing of its own. Your costs are the underlying vendors: Twilio per-minute voice and phone number rental, OpenAI per token, ElevenLabs per character of synthesised speech, and MongoDB hosting. The token caps in the code exist partly to keep the per-turn model and speech spend bounded.

Can it make outbound calls or only answer inbound ones?

Both. The make_call endpoint takes a phone number as a query parameter and creates a Twilio call with inline TwiML that plays an opening clip and then hands the caller to the speech handler. Inbound calls use a near-identical twin route, so the persona and the model call behave the same in either direction.

Can I use my own voice provider or language model?

Yes. Text-to-speech lives in one module behind a single function, and the repo already contains a commented OpenAI TTS implementation next to the live ElevenLabs one. The model call is similarly isolated in the chatbot module, which ships both a direct chat-completions path and a LangChain agent path you can switch between.

How does it remember a caller between calls?

Identity is the phone number. The inbound webhook looks it up in MongoDB to decide whether the caller is known, and the tools layer maps that number to a user id whose chat history is a document with a messages array. The LangChain agent path reads and writes that array; the direct completion path does not.

Is this production-ready for my business?

It is a working reference implementation, not a managed product. Expect to move credentials out of config.py into real environment variables, add authentication, call recording consent and retry handling, and replace the ngrok tunnel with a proper deployment target. If you want that hardened version built for your stack, that is the kind of engagement I take on.

AI Calling Agent — Open-Source Twilio + OpenAI Voice Bot — Case Study

Verify it yourself

GitHub repository — Public source. 47 stars and 18 forks as of July 2026.
README and setup instructions — Documents the four HTTP endpoints and the install steps referenced on this page.

Introduction

I built AI Calling Agent because I wanted to know exactly where the seconds go in a phone conversation with a language model. It is a Flask application that takes a Twilio call, turns speech into text, asks OpenAI for a reply, synthesises that reply with ElevenLabs, and plays it back before opening the microphone again. I published it in April 2024 and it has picked up 47 stars and 18 forks as of July 2026, which means other engineers have been reading the same code you are about to read about.

The Challenge

A chat interface forgives latency. A phone call does not. If a caller finishes a sentence and hears nothing, they assume the line dropped and start talking over the reply. The constraint I hit is that every turn stacks four network round trips in sequence: Twilio's speech recognition, the OpenAI completion, the text-to-speech request, and then Twilio fetching the generated audio file back off my server. None of them can be skipped, and each one has its own failure mode. On top of that, a phone number is the only identity you get, so any notion of a returning caller has to be built from that alone.

The Solution

The whole loop lives in one Flask app with a small module per external service. Twilio posts to a voice webhook that checks whether the caller is known, plays an opening audio file, and opens a Gather with speech input. The transcript comes back to a speech handler, which passes it to the chatbot module and gets a short reply from the OpenAI API. That text goes to ElevenLabs, gets written to a uniquely named MP3 under the static directory, and Twilio is instructed to play that file and immediately open another Gather. Users, chat history, session logs and appointments are modelled in MongoDB, reached through the caller's number.

Technical Deep Dive

Turn-taking is Twilio Gather, not a media stream. The inbound handler and the speech handler both append a Gather with input set to speech, a ten second timeout and speechTimeout set to auto, so Twilio decides when the caller has stopped talking instead of my code guessing at silence thresholds. The trade-off is honest: you get endpointing for free, and you give up the ability to barge in mid-sentence.

Outbound calls are created with inline TwiML rather than a webhook fetch. The make_call endpoint takes a phone number as a query parameter and calls the Twilio REST client with a TwiML string that plays the opening audio and then hands control to the speech handler. Inbound calls point at a near-identical twin route, the only substantive difference being which Twilio field the handler reads the caller's number from.

Response length is capped in two places at once. The chat-completions path requests gpt-3.5-turbo with max_tokens set to 70 and temperature 0.5, and the system prompt separately instructs the persona to stay under eighty words. Capping tokens bounds both the completion time and the text-to-speech time, which are the two costs that scale with output length.

Speech synthesis is a POST to the ElevenLabs text-to-speech endpoint with the eleven_turbo_v2 model and stability and similarity_boost both at 0.5, with the response body written to disk in 1024-byte chunks. An OpenAI tts-1 implementation using the alloy voice is still in the file, commented out, because swapping voice providers is the easiest way to move the latency figure without touching the dialogue code.

Every generated clip gets a unique filename built from a timestamp. Twilio fetches audio by URL, and a fixed filename means a cached or half-written file from the previous turn can be served into the current one. Unique names per turn remove that class of bug entirely.

State is a MongoDB document per caller. The Mongo module defines separate collections for users, chat history, session logs, therapy progress and appointments, and the agent tools layer maps a phone number to a user id before touching any of them. History is written with an upsert that pushes each new turn onto a messages array.

A second, richer response path ships alongside the simple one. It builds a LangChain OpenAIFunctionsAgent on gpt-4-1106-preview with streaming enabled, two retries and a 150-token ceiling, replays stored history into a MessagesPlaceholder as alternating human and AI messages, and writes the result back. It is heavier per turn than the direct completion, and the routes as shipped call the direct one.

The requirements file is wider than the running code. pinecone-client and faiss-cpu are both listed, and the chatbot module imports FAISS and OpenAIEmbeddings, but no vector store is ever constructed and no Pinecone client is instantiated anywhere in the tree. There is no retrieval path in this repo, only the dependency footprint of one that was planned.

Configuration is the weakest part of the repo and worth stating plainly. config.py calls load_dotenv and then assigns its values as inline literals rather than reading them back out of the environment, so the committed .env.example beside it has no effect until that module is changed. An ngrok URL is threaded through the modules from the same file, because local development means Twilio has to reach a tunnel on your machine.

Key Features

Inbound and outbound in one service

A POST voice webhook answers incoming calls; a GET make_call endpoint dials out to any number passed as a query parameter. Each direction has its own speech handler, and the two are near-identical, so the persona and the model call behave the same either way.

Caller recognition from the phone number

Before answering, the inbound handler asks MongoDB whether the number has interacted before. Unrecognised numbers hear a pre-recorded alert clip and the response ends there; recognised numbers get the opening clip followed by a Gather that starts the conversation.

Conversation memory modelled in MongoDB

The LangChain agent path reads stored turns back as alternating human and AI messages and upserts new ones onto a per-user messages array. The direct chat-completions path that the routes call today sends only the current utterance, so enabling memory means switching which function the handler calls.

Swappable voice layer

Text-to-speech is isolated in a single module with one function signature. The live path calls ElevenLabs turbo; a commented OpenAI TTS implementation sits above it, so changing voice vendor is a one-file edit.

Persona defined entirely in the system prompt

The shipped persona is a mental-health consultant with explicit instructions on brevity, tone and ending each turn with a question. Repointing the agent at sales, support or intake means editing that prompt, not the call plumbing.

Graceful call termination

The speech handler checks each transcript for a goodbye and appends a closing line and a hangup, so the call ends on the caller's cue rather than waiting out a Gather timeout.

Results & Impact

✓The repository is public and has 47 stars and 18 forks as of July 2026, which is the only audience number on this page and the only one I can verify.
✓It runs as a complete loop once you supply what is not in the tree: credentials in config.py, a tunnel URL, and the static MP3 clips the call flow plays, none of which are committed.
✓The README documents four endpoints, and all four exist in app.py: a status route, the inbound voice webhook, the outbound make_call route and the speech handler.
✓Every external service is isolated behind its own module, so the Twilio, OpenAI, ElevenLabs and MongoDB layers can each be replaced without touching the others.
✓The README declares an MIT license and links to a LICENSE.md that was never committed, so GitHub detects no license on the repository at all. That is a real gap I would rather state than paper over.

Lessons Learned

"Latency in a voice agent is a chain, not a number: recognition, then completion, then synthesis, then Twilio fetching the audio back off your server. Output length is the one variable that sits inside two of those links at once, which is why the code caps completion tokens and the prompt caps the spoken reply at eighty words."

"Interruption is where the play-then-listen architecture stops being enough. Because a reply is a fully rendered audio file, a caller who starts talking over it has to wait for the file to finish before Twilio reopens the microphone. Real barge-in needs a bidirectional media stream, and that is a rewrite of the transport, not a patch on it."

"State belongs outside the call. Modelling history as a MongoDB document reached through the phone number makes a dropped call an interruption rather than an amnesia event. The honest caveat is that this repo stops one step short: the handler that ships enabled calls the stateless completion path, so the stored history only engages if you point it at the agent function."

"Public repositories punish shortcuts in configuration. config.py loads dotenv and then assigns inline literals anyway, which means the checked-in example file does nothing until that module is rewritten to read the environment. It is the first thing I would fix here and the first thing I look for in anyone else's repository now."

Conclusion

AI Calling Agent is the smallest honest version of a production voice bot: real telephony, real speech recognition, a real language model, real persistence, and no abstraction hiding where the time goes. If you are evaluating whether to build a phone agent, reading these eight Python files will tell you more about the shape of the problem than any vendor demo.

Frequently asked questions

Is AI Calling Agent open source and free to use?: Yes. The repository is public on GitHub at revolutionarybukhari/ai-calling-agent and the README declares the MIT License, though the license file it links to was never committed, so GitHub reports no license. You can clone it, install the requirements and run it today. It has 47 stars and 18 forks as of July 2026.
How does it handle interruptions when the caller talks over the bot?: It does not, and that is a property of this architecture. Replies are synthesised to an MP3 and played by Twilio, so the microphone reopens only after playback finishes. True barge-in requires a bidirectional media stream instead of the play-then-Gather loop this repo uses. That is the main limitation to understand before adopting it.
What does it cost to run?: The repo has no pricing of its own. Your costs are the underlying vendors: Twilio per-minute voice and phone number rental, OpenAI per token, ElevenLabs per character of synthesised speech, and MongoDB hosting. The token caps in the code exist partly to keep the per-turn model and speech spend bounded.
Can it make outbound calls or only answer inbound ones?: Both. The make_call endpoint takes a phone number as a query parameter and creates a Twilio call with inline TwiML that plays an opening clip and then hands the caller to the speech handler. Inbound calls use a near-identical twin route, so the persona and the model call behave the same in either direction.
Can I use my own voice provider or language model?: Yes. Text-to-speech lives in one module behind a single function, and the repo already contains a commented OpenAI TTS implementation next to the live ElevenLabs one. The model call is similarly isolated in the chatbot module, which ships both a direct chat-completions path and a LangChain agent path you can switch between.
How does it remember a caller between calls?: Identity is the phone number. The inbound webhook looks it up in MongoDB to decide whether the caller is known, and the tools layer maps that number to a user id whose chat history is a document with a messages array. The LangChain agent path reads and writes that array; the direct completion path does not.
Is this production-ready for my business?: It is a working reference implementation, not a managed product. Expect to move credentials out of config.py into real environment variables, add authentication, call recording consent and retry handling, and replace the ngrok tunnel with a proper deployment target. If you want that hardened version built for your stack, that is the kind of engagement I take on.

AI Calling Agent — Open-Source Twilio + OpenAI Voice Bot