AI Voice Agents Explained: How Vapi and Retell Actually Work

AI voice agents are software systems that answer or place phone calls, understand what a caller says, decide what to do, and speak back — in real time, without a human on the line. The reason AI voice agents feel almost human is that four hard problems — hearing, understanding, deciding, and speaking — have to complete in under a second, on every conversational turn, over a phone line that adds its own delay. Platforms like Vapi and Retell exist to orchestrate that loop so you don't have to wire it together from scratch.

This post explains what actually happens inside a voice agent, where the latency comes from, how Vapi and Retell differ, and what a production build needs beyond a demo that works once in a quiet room.

How AI Voice Agents Work, Turn by Turn

Every AI voice agent runs the same fundamental pipeline on each turn of a conversation:

1. Speech-to-text (STT/ASR) — the caller's audio is streamed to a transcription model (Deepgram, AssemblyAI, or similar) that emits words as they're spoken.
2. Turn detection / endpointing — the system decides when the caller has finished talking. This is the most underrated component. End the turn too early and you interrupt people; end it too late and the agent feels sluggish.
3. LLM inference — the transcript, plus the system prompt and conversation history, goes to an LLM (GPT, Claude, or a smaller/faster model) which produces the next thing to say — and, when configured, calls tools.
4. Text-to-speech (TTS) — the LLM's text is synthesized into audio by a voice model (ElevenLabs, Cartesia, PlayHT, etc.) and streamed back to the caller.

The trick that makes this feel human is streaming everything. The STT emits partial words, the LLM streams tokens as it generates, and the TTS starts speaking the first sentence before the LLM has finished the paragraph. Nothing waits for the previous stage to fully complete. A voice agent that batches each stage sounds like a walkie-talkie; one that streams and pipelines sounds like a person.

Where the latency actually lives

Latency is the whole game. The target that separates "natural" from "robotic" is roughly the pause length of an ordinary human conversation — a short, sub-second window of perceived response time. That budget gets eaten by:

• Network + telephony transport — the call rides over SIP/PSTN via a carrier like Twilio or Telnyx, which adds real round-trip time before your code ever sees the audio.
• Endpointing delay — the silence window the system waits through to be sure the caller stopped.
• Time-to-first-token from the LLM — not total generation time, because you stream, but the first token gates when speech can start.
• TTS time-to-first-byte — how fast the voice model returns the opening audio chunk.

You optimize the perceived latency, not the total, by overlapping these stages and by picking fast models for the parts that block speech. This is also why "just use the biggest model" is usually wrong for voice: a slightly less capable model that returns its first token faster often produces a better-feeling call than a smarter one that makes the caller wait.

Interruptions, Tools, and State

Two more things turn a text chatbot into a real voice agent.

Barge-in (interruption handling). Humans interrupt each other. When the caller starts talking while the agent is mid-sentence, the agent has to stop speaking immediately, discard the queued audio, and re-listen. Getting barge-in right — without cutting off on background noise or the caller's own "mm-hm" — is a large share of what makes a deployment feel finished.

Function/tool calling. A voice agent that can only talk is a voicemail greeting. The value comes when the LLM can call functions mid-call: check availability, look up an order, create a CRM record, book a slot. In our autonomous revenue infrastructure builds, those tool calls hit an n8n webhook that runs the actual business logic — query Supabase, hit a scheduling API, write the lead — and returns a compact result the agent reads back. The agent stays a thin conversational layer; the durable state and logic live in the workflow and the data ledger behind it.

Vapi vs Retell: Suppliers, Not Rivals

Vapi and Retell solve the same core problem — orchestrating the STT → LLM → TTS loop over telephony with interruption handling — and both are legitimate production choices. They are infrastructure suppliers you build on, not competitors you pick a side in. The differences are about control surface and how you prefer to model a conversation.

Vapi leans toward a flexible, developer-first assistant model. You configure the transcriber, model, and voice provider independently, wire tools via server URLs, and get fine-grained control over the pipeline and the telephony layer. It suits builds where you want to swap components — a different TTS voice, a faster model — and manage call behavior programmatically. This is the model behind our Inbound Voice Architecture (Vapi) package.

Retell leans toward structured conversation design, with strong support for flow-style logic where the path through a call is more explicitly defined. Teams that want the conversation's structure to be legible — states, transitions, guardrails on what the agent can say — often find that model faster to reason about for well-scoped call types like inbound qualification or appointment setting. We frame that approach in the Inbound Voice Architecture (Retell) package.

In practice the decision is driven by the call type, not brand loyalty:

• Open-ended, tool-heavy, "figure it out" conversations → the flexible assistant model fits well.
• Tightly scoped, compliance-sensitive, repeatable call flows → the structured model is easier to keep on rails.

Because both expose the same underlying primitives — bring-your-own model, bring-your-own voice, webhook tools, telephony numbers — you are rarely locked out of anything by the choice.

What a Production Voice Agent Actually Needs

A demo agent needs a prompt and a phone number. A production agent — one you'd put in front of real customers and real revenue — needs the parts nobody shows in the demo:

• A real telephony plan. Provisioned numbers, caller ID, concurrency limits, and graceful failover when a call drops mid-conversation. Voice traffic is stateful; a reconnect strategy matters.
• Tool endpoints that are fast and idempotent. Every function call sits inside the latency budget, so a slow lookup makes the agent stall audibly. And because networks retry, a "book the meeting" tool must not double-book on a repeated call.
• A source of truth behind the agent. The LLM is not your database. Bookings, transcripts, call outcomes, and lead data belong in a durable store (we use Supabase as the ledger) so the call is captured even if the model hallucinates a summary.
• Guardrails and fallbacks. What the agent may and may not say, when to hand off to a human, and what happens on silence, a wrong number, or an out-of-scope request.
• Post-call automation. The call ending is the start of the workflow: transcribe, summarize, score the lead, sync to the CRM, trigger follow-up. That's the difference between a phone toy and business infrastructure.
• Observability. Per-call transcripts, latency metrics, and interruption logs, so you can actually diagnose why one call felt off instead of guessing.

The conversation is the easy, visible part. The tool wiring, state management, telephony resilience, and post-call automation are the hard, invisible majority that determine whether the thing survives contact with real callers — and that part has to be engineered, not prompted.

Build a Voice Agent That Holds Up in Production

If you're evaluating AI voice agents for inbound qualification, appointment setting, or outbound campaigns, the platform choice matters far less than the system you build around it. We engineer the full stack — Vapi or Retell for the conversation, n8n for the logic, a real data ledger for state — so calls turn into captured, actioned outcomes instead of transcripts nobody reads.

Schedule a Build and we'll scope the exact call flow, telephony, and automation your deployment needs.