Wiring speech is not the same as placing a call.

Search "how to build an ai voice agent" and you get a pile of 30-minute Python tutorials. Most of them are honest about the code. They are quieter about the product. The demo talks through your laptop mic. Nobody's cell phone rings. That is still a voice agent. It is not a phone agent. If you wanted the second thing, you are about to spend a week solving the first.

This is the map, not another copy of the STT-LLM-TTS hello world. Four ways to build. One of them is "do not build a voice agent at all." Checked 17 September 2026 against vendor docs named below, plus DialMCP's own voice AI API map and safety model.

What you are actually assembling

Under the hood, almost every production voice agent is the same loop. LiveKit's February 2026 Python tutorial still uses this as the default, and they list the aliases you will see in other posts: cascaded pipeline, ASR-LLM-TTS, listen-think-speak. Same machine.

1. Speech-to-text turns the caller's audio into text as they talk.
2. A language model reads the transcript, the prompt, and any tools, then decides what to say (or what to call).
3. Text-to-speech turns that reply back into audio.
4. Turn-taking decides when they finished, when you may speak, and how to stop the instant they interrupt.

AssemblyAI's August 2026 "three ways" post is right that step 4 is the piece people underestimate. A model that sounds great in a recording still feels broken if it talks over the caller or waits a beat too long after they stop.

There is a newer shape: speech-to-speech. OpenAI's Realtime API and GPT-Live, and Google's Gemini Live, take audio in and emit audio out without a text hop in the middle. LiveKit documents the trade: lower latency and better prosody, worse observability, fewer providers, usually higher cost per minute. Twilio's 10 September 2026 Agent Connect tutorial is the PSTN version of that idea. GPT-Live is full-duplex, so barge-in is the model's job, and the audio format can match Twilio Media Streams (8 kHz mu-law) instead of a 24 kHz playground.

Both shapes still need a fifth layer that the browser tutorials skip:

• Transport. WebRTC or a WebSocket in an app. SIP or a carrier Media Stream if a real number has to ring.
• A number. Purchased DID, imported Twilio/Telnyx/Vonage number, verified caller ID, or (in DialMCP's case) the user's SMS-verified mobile.
• Rules. Recording notice, AI disclosure, calling hours, rate limits, opt-out. If those live only in the system prompt, they are not rules.

If your "voice agent" never leaves the browser, you can stop at STT, LLM, TTS, and a VAD. The rest of this page is for people who wanted a phone to ring.

Four ways to build

AssemblyAI's taxonomy is useful if you add a fourth column. Their three paths are "how much of the speech stack do you own." The fourth path is "you already have a reasoning agent; you only needed the phone."

Those last two cells are the split this site cares about. Vapi, Bland, Retell, and ElevenLabs Agents are volume platforms: campaigns, imported numbers, contact-center jobs. See the API map for who can actually dial E.164. DialMCP is not in that row. It is the fourth path.

Custom: you write the glue

Pipecat (Daily, open source, Python) is the honest version of "build it yourself" in 2026. The happy path is still transport -> STT -> LLM -> TTS -> transport. Deepgram's Pipecat guide wires Deepgram for speech, OpenAI for the model, Daily for WebRTC. You can swap Cartesia, ElevenLabs, Twilio, whatever the plugin list allows. You also own every buffer.

Do this when you have a reason: data residency, a model that is not in anyone's dropdown, a multi-agent bus, a lab. Do not do this because a blog said "full control." Full control of a media pipeline is a second product.

Orchestrator: you pick the parts, they move the audio

LiveKit Agents is the open-ish media layer. Their Python tutorial (21 February 2026) gets a talking agent up with livekit-agents, Silero VAD, and a turn-detection model. LiveKit Inference can sit in front of STT/LLM/TTS so you are not juggling three API keys on day one. PSTN is extra: attach a SIP trunk, then create a SIP participant. LiveKit is infrastructure. It will not pretend to be a dialer.

Vapi is the declarative cousin. You point STT, LLM, and TTS at vendors, and Vapi owns call lifecycle and turn-taking. Outbound PSTN is a first-class API (/call with an assistant and a destination). Caller ID is an imported Twilio, Vonage, or Telnyx number, or a Vapi number. That is the right shape for a support line. It is a heavy shape for "Claude, call my dentist."

Twilio Agent Connect (Python SDK, tac[server,gpt-live] as of the 10 September 2026 post) sits on Programmable Voice Media Streams. The inbound tutorial answers a Twilio number with GPT-Live. A companion Node tutorial the same day originates an outbound call, then bridges it. You need a Twilio number, a public WSS endpoint (ngrok in dev), and an OpenAI project with GPT-Live access. Twilio is also blunt in that post: outbound calling has rules, and in the US that includes the TCPA.

OpenAI's own Realtime SIP docs are inbound. You point a SIP trunk at sip:[email protected];transport=tls, handle realtime.call.incoming, then accept or reject. That is "the phone called the model." Outbound still needs a carrier in front. Do not copy a Realtime playground into a cron job and expect a cell phone to ring.

All-in-one: one WebSocket

AssemblyAI's Voice Agent API (generally available 14 April 2026, per their August post) collapses STT, LLM, TTS, turn detection, and tool calls into one WebSocket. They publish a flat $4.50/hour and "roughly 1 second" end-to-end on their own stack. Treat those as vendor numbers, not a benchmark of the category. The architectural claim is the useful one: you are no longer timing three streams against each other.

This path is great for a headset or an in-app agent. It does not, by itself, answer "whose caller ID shows up on a mobile." If you need PSTN, you still add a carrier, or you pick a platform that already originates calls.

The fourth path: the agent already exists

A lot of "I want to build an AI voice agent" tickets are actually "I already live in Claude / Cursor / Codex / ChatGPT / Gamut, and I want that session to place one phone call." Building STT and TTS for that job is wasted work. The client already has a model. What it lacks is a tool that can originate a PSTN call, wait on hold, and bring back a transcript.

That is an MCP server, not a voice pipeline. Remote MCP is the transport explainer. DialMCP is the hosted phone. Four tools: place_call, get_call, end_call, list_calls. The call is async on purpose so a six-minute hold does not trip the client timeout. Details: tool reference.

You do not get a custom voice, a campaign dashboard, or a pool of DIDs. You get the user's SMS-verified mobile as caller ID, US and Canada only, server-enforced disclosure and hours. If that is a bad fit, it is a bad fit. Use Vapi. Use Twilio. Do not dress DialMCP up as a Bland alternative.

The parts that make the demo die on a real line

Tutorials fail in the same five places. Budget time here, not in picking a "better" TTS.

Hold music and IVR. Cascaded STT will happily transcribe a menu as if a human were talking. Speech-to-speech will try to "converse" with a recording. A phone agent needs a policy: listen for a tree, enter DTMF, or hang up. DialMCP's voice agent is built to navigate trees and hold; a laptop demo is not.

8 kHz mu-law. PSTN audio is narrowband. A model that sounded warm on a 24 kHz WebRTC mic will sound thin, and names will drop. Twilio's GPT-Live note about matching Media Streams' wire format exists because this mismatch is the default.

Caller ID. Callees ignore unknown numbers, or they never pick up a rotating DID. Volume platforms give you a provisioned business number and a spam-label problem. DialMCP presents the user's own verified mobile so a callback reaches a person. ElevenLabs can present a Twilio-verified caller ID for outbound-only use. Those are different products. Pick on purpose.

Inbound vs outbound. OpenAI Realtime SIP, ChatGPT Voice, and 1-800-ChatGPT are "you call the model." Twilio inbound TAC is the same direction with your number. Outbound is the other product: you originate, you disclose first, you take TCPA-shaped hours seriously. DialMCP only starts calls between 8:00 and 21:00 in the destination's local time, aligned with US telemarketing windows, and unknown time zones are treated conservatively.

Prompt-only safety. "Always say you are an AI" in the system prompt is not a control. Models skip it. DialMCP discloses before the ask, records, rate-limits (1 concurrent, 3/hour, 10/day, 2/day to the same number), blocks 911 and premium-rate ranges, and honors opt-out platform-wide. If you are building the stack, those have to be server-side or they are theater. Read Safety before you copy a weather-tool sample onto the public telephone network.

A decision tree that does not flatter us

1. You wanted to talk to a model. Stop. ChatGPT Voice, or an in-app Realtime session. No voice API will place an outbound call for that job.
2. You wanted an inbound receptionist, a clinic line, or a campaign. Vapi, Bland, Retell, or ElevenLabs Agents, with a number you import. That is their ICP.
3. You wanted to own the media layer. LiveKit or Pipecat, then a SIP trunk when you are ready to leave the browser. Twilio Voice if the phone network is the starting point and the model is a plugin.
4. You wanted speech-to-speech on a Twilio number this week. Agent Connect + GPT-Live, inbound or outbound, with a real WSS URL and a compliance read.
5. You already have an MCP client and you wanted one personal outbound call from a number the other person can call back. Connect DialMCP. Walkthrough: how to make a phone call with an AI assistant. Shape: app vs agent.

If you are on step 5, you are not "building an AI voice agent." You are attaching a phone to an agent you already run. That is the whole point of saying it out loud. The SERP is full of people teaching you to rebuild the mouth. Check whether you only needed the number.

If you still want to build the speech stack

Keep the first slice embarrassing small. One inbound Twilio number, or one LiveKit room, with a single tool (look up an order, read a calendar). Log the transcript on every turn. You will catch the real bugs there: barge-in that fires on a cough, a name the STT always mangles, a prompt that invents a confirmation number.

Do not start with outbound. Outbound is where you inherit TCPA-shaped hours, consent, caller ID reputation, and a human who did not ask to be woken up. Twilio's own outbound GPT-Live writeup tells you to read the Voice terms before you copy the makeCall helper. Believe them.

Ship a kill switch before you ship a personality. A global hang-up you can hit without redeploying is more important than a warmer voice. DialMCP's platform kill switch exists because this class of product fails loudly when it fails. Yours should too, even if the "platform" is one FastAPI process and a Twilio number.

Further reading

• Which voice AI APIs can call a cell phone
• DialMCP tool reference
• How to make a phone call with an AI assistant
• AI phone call app vs agent
• Safety and anti-spam
• Remote MCP servers explained
• AssemblyAI: three ways to build (2026-08-04)
• LiveKit: first Python voice agent (2026-02-21)
• Twilio Agent Connect + GPT-Live (2026-09-10)

FAQ

What is an AI voice agent?

A system that listens, decides, and speaks in a tight loop, over a mic or a phone. The usual implementation is streaming speech-to-text, a language model, text-to-speech, and turn-taking. Speech-to-speech models skip the text hop.

How long does it take to build one?

An in-app demo: an afternoon on LiveKit, Pipecat, or an all-in-one WebSocket. A production PSTN agent: days to weeks once you add a number, outbound originate, hold/IVR, and server-side limits. Connecting a hosted phone MCP to a client you already use: minutes.

Do I need Python?

Only if you are writing the pipeline. Orchestrators and hosted MCP servers are config plus a client. Twilio Agent Connect happens to be a Python (and Node) SDK. DialMCP's remote URL is Streamable HTTP. No Python required on your side.

Can OpenAI Realtime call a cell phone by itself?

No. The SIP docs cover incoming calls to the API. Outbound PSTN needs a telephony provider such as Twilio in front. Same answer as of 17 September 2026.

Where does DialMCP fit?

It is not a voice-agent framework. It is a hosted remote MCP server that lets an existing agent place an outbound US/Canada call from the user's SMS-verified number. Four tools, OAuth, no API key, free during launch. It will not run your contact center.

Should I build or buy?

Build the speech stack if the voice is the product (latency, on-prem, a custom mouth). Buy or host the phone if the product is already an agent and you only needed it to originate a call. Mixing those two jobs is how a weekend tutorial becomes a quarter.