Voice AI is software that listens and talks. Speech to text turns your voice into words, a language model decides what to say, and text to speech turns the reply back into a voice. The newest real-time speech models do all three in one step, fast enough to hold a natural conversation, interruptions included.
This guide explains the building blocks, why speed decides whether a conversation feels human, what voice agents can do for a business today, and where accuracy and privacy still bite.
- Voice AI has three parts: speech to text (listening), a language model (thinking) and text to speech (talking).
- Real-time speech models take audio in and give audio out directly, which cuts delay and lets you interrupt.
- Speed decides how natural it feels. In conversation, people usually start answering within about 200 milliseconds.
- Voice agents can answer calls, take bookings and translate live, but they need a narrow job and a human fallback.
- Your voice is personal data, and some laws treat a voiceprint as biometric data. US calls with an AI voice generally need consent.
How voice AI works: three building blocks
The middle step is a language model, the same kind of model behind text chatbots. It reads the words, decides what to do and writes a reply. Our guide to how large language models work explains that part.
The last step, text to speech, turns the reply into a voice. Modern voices can whisper, laugh, pause and change pace, which makes the best AI narration sound remarkably human.
OpenAI, Google and ElevenLabs each sell all of these pieces, and OpenAI and Google also sell models that skip the text in the middle. As of September 2026, these are the current options on their developer pages:
| Job | OpenAI | ElevenLabs | |
|---|---|---|---|
| Speech to text | GPT-Transcribe, GPT-Live-Transcribe | Gemini 3.5 Transcribe | Scribe v2, Scribe v2 Realtime |
| Text to speech | GPT-4o Mini TTS | Gemini 3.8 Flash TTS | Eleven v3, Flash v2.5 |
| Real-time conversation | GPT-Live 1, GPT-Realtime-2.1 | Gemini 3.8 Live | ElevenAgents, which chains the parts above |
Specialists matter too. Deepgram, for example, offers Flux, a speech recognition model built for voice agents, with turn detection included.
Two designs: a chain of models or one real-time model
There are two ways to build a talking system. The classic way chains three models together. The newer way uses one model that hears audio and speaks audio.
| Chain: speech to text, model, text to speech | Real-time speech model | |
|---|---|---|
| How it works | Three models pass text along | One model, audio in and audio out |
| Delay | Each step adds its own wait | Built for low delay |
| Interruptions | Need extra turn-detection work | Built in, often called barge-in |
| Tone of voice | Mostly lost when speech becomes text | Can hear tone and answer in kind |
| Control | Swap any part, and read the text in the middle | Fewer parts to tune or inspect |
| Best for | Strict scripts, any language model, full logs | Natural conversation and live translation |
OpenAI describes its Realtime API as built for speech-to-speech voice agents, with interruptions, fast first audio, natural turn-taking and tool use during the call. Google’s Live API lists barge-in and “affective dialog,” which adapts its replies to how the user sounds.
The chain is not obsolete. ElevenAgents, ElevenLabs’ agent platform, combines its own speech recognition, a language model of your choice, its voices and a separate turn-taking model. That flexibility is useful when you need a particular model or a written record of every step.
Why speed decides whether it feels natural
Human conversation is fast. In a study of ten languages, from major world languages to small indigenous communities, answers most often started within 200 milliseconds of the question ending. Against that rhythm, a voice agent that pauses for a second or two sounds hesitant or broken.
The individual pieces are fast. The total is what callers feel. A chained agent adds up the listening, the thinking, the first syllable of speech and the network trip both ways.
The hardest part is often knowing when you have finished talking. Wait too little and the agent cuts you off mid-sentence. Wait too long and every answer arrives late. That is why vendors now ship dedicated turn-detection models, and why real-time models advertise natural turn-taking as a feature.
What voice AI can do for a business now
Voice is useful wherever typing is slow, impossible or unwelcome. The jobs that work well today:
Answering calls. Opening hours, bookings, order status and routing, at any hour. The same rules apply as for chat: automate the routine, keep a person for the rest. Our guide to AI customer support covers what to automate.
Live translation. OpenAI offers GPT-Realtime-Translate and Google offers Gemini 3.5 Live Translate, in preview, for streaming speech-to-speech translation. Google says its model supports more than 70 languages.
Transcription and notes. Meetings, interviews and voice memos become searchable text within minutes.
Voiceovers and narration. Product videos, courses and audio versions of articles. Our AI voiceover guide covers tools, scripts and consent.
Hands-free work. Talking to an assistant while your hands are busy. Cyborb, our desktop agent, has one called Anima: a hands-free voice presence that speaks replies and listens, on the Pro and Max plans.
Two rules matter before a voice agent talks to customers. In the US, the FCC ruled in February 2024 that AI-generated voices count as “artificial” voices under the Telephone Consumer Protection Act. Calls that use them need the called person’s prior express consent, unless an emergency or exemption applies. In the EU, since August 2, 2026, people must be told when they are talking to an AI system, unless it is obvious.
Accuracy: where voice AI still slips
Speech recognition is very good with one clear speaker in a quiet room. It gets worse with accents it heard less often in training, background noise, people talking over each other, and rare names or product terms.
It can also invent. Researchers who studied OpenAI’s Whisper found that about 1% of transcriptions contained whole phrases or sentences nobody said, and people with aphasia, a condition that affects speech, were hit harder. Newer models may do better, but the habit of checking should stay.
For voice agents, the fix is simple: confirm before acting. “That is Tuesday the 14th at 3 p.m. for two people, is that right?” catches a mishearing before it becomes a mistake. Then send the details by text or email as well, so the customer has a written record.
Privacy: your voice is personal data
A recording of your voice is personal data, and it can reveal more than your words: who you are and, often, how you feel. Before you use a voice tool, check how long it keeps recordings and whether they train its models.
Some laws go further. Illinois’ Biometric Information Privacy Act lists a voiceprint as a biometric identifier. A business must tell people in writing and get a written release before collecting one. Courts can award $1,000 per negligent violation and $5,000 per intentional or reckless one. Collecting the same person’s voiceprint again by the same method counts as one violation, not many.
Voice cloning raises the stakes. A cloned voice can read your ads, and it can also impersonate you on the phone. Only clone voices with clear, written consent, and agree on a family or team code word for urgent calls. Our guide to AI scams shows how voice-clone fraud works.
FAQ
What is the difference between voice AI and a voice assistant?
Older voice assistants matched spoken commands to a fixed list of actions. Today’s voice AI runs on large language models, so it can hold an open conversation, follow up on what you said and use tools during a call.
How fast does a voice agent need to be?
Close to human pace. People usually start answering within about 200 milliseconds, so every part of the system has to be quick, and the agent must detect the end of your turn without cutting you off.
Is it legal to use an AI voice to call customers?
In the US, AI-generated voices fall under the rules for artificial voices, so you need the person’s prior express consent, with limited exceptions. In the EU, people must be told they are talking to an AI unless it is obvious. Check the rules wherever your customers live.
Can AI clone my voice?
Yes. Many tools can make a convincing copy from recordings of a person. That is useful for your own voiceovers and dangerous in a scammer’s hands, which is why a code word for urgent calls is worth setting up.
How accurate is speech recognition now?
Very accurate with clear audio and common words, and weaker with heavy accents, noise, cross-talk and rare names. It can occasionally invent text, so check anything that matters, especially names, numbers and dates.
- Voice AI chains listening, thinking and talking, or does all three in one real-time model.
- Speed is the difference between a conversation and a phone tree.
- Give voice agents a narrow job, a read-back habit and a human fallback.
- Get consent before calling US customers with an AI voice, and disclose AI in the EU.
- Treat recordings and voiceprints as sensitive personal data.
Next, see how transcripts become action items in AI meeting notes, or learn how to spot a deepfake, including a cloned voice.
- Models, OpenAI API docs, accessed September 2026
- Getting started with the Realtime API, OpenAI API docs, accessed September 2026
- Models, Gemini API docs, Google, updated September 2026
- Gemini Live API overview, Gemini API docs, Google, updated September 2026
- Models, ElevenLabs documentation, accessed September 2026
- ElevenAgents overview, ElevenLabs documentation, accessed September 2026
- Models and languages overview, Deepgram docs, accessed September 2026
- Universals and cultural variation in turn-taking in conversation, Stivers and others, PNAS, 2009
- Declaratory Ruling on AI-generated voices in robocalls, FCC 24-17, Federal Communications Commission, February 2024
- Article 50: transparency obligations for providers and deployers of certain AI systems, European Commission AI Act Service Desk, accessed September 2026
- Careless Whisper: speech-to-text hallucination harms, Koenecke and others, FAccT 2024
- Biometric Information Privacy Act, 740 ILCS 14, Illinois General Assembly
