AI Phone Agents and Voice Bots

The phone is still where many customers turn when something is urgent or complicated. A missed call can mean a lost order, a lost booking or a patient who goes elsewhere. AI phone agents promise to answer every call, day and night. They can handle a good share of those calls, within limits that you set before the first call.

This is part 2 of Level 2 in our AI Work learning path. In Level 1, you set up an AI chatbot for customer service. A phone agent follows the same principles, but it adds three challenges: speed, speech recognition and legal rules for calls. You will learn how a phone agent works, which calls it should take, how to design a call flow with a handover, what you must tell callers and how to measure the results.

How an AI phone agent works

An AI phone agent, also called a voice bot or voice agent, answers or makes phone calls and talks with the caller in natural language. Unlike the old phone menus (“press 1 for sales”), it understands free speech. Most systems chain three technologies:

1. Speech recognition, also called speech-to-text, turns the caller’s words into text.
2. A large language model reads the text, decides what to do and writes the answer. It can look things up in a knowledge base and use connected tools, such as your calendar or CRM.
3. Speech synthesis, also called text-to-speech, turns the answer into a voice.

Some newer systems skip the text step and process audio directly. The principle stays the same: listen, decide, speak.

Two problems weigh much more on the phone than in a chat. The first is latency: the delay between the end of the caller’s sentence and the start of the answer. Every step in the chain adds time, and people are very sensitive to pauses. A study of conversations in ten languages found that speakers in all of them avoid talking at the same time and keep the silence between turns short (Source: Stivers et al., PNAS, 2009). If a bot pauses much longer than a person would, callers repeat themselves, talk over it or think the line is dead. A good system also handles barge-in: when the caller interrupts, the agent stops talking and listens.

The second problem is speech recognition itself. Background noise, poor lines, accents, dialects and unusual names cause errors, and an error here travels through the whole chain. If the system hears the wrong date, the model books the wrong date. A 2020 Stanford study tested the speech recognition of five large technology companies. On average, the systems made almost twice as many errors for Black speakers as for white speakers (Source: Koenecke et al., PNAS, 2020). The systems have improved since then, but the lesson remains: test with the real voices of your callers, not only with your own team.

The voices themselves often come from text-to-speech providers. Our AI Productivity comparison lists them in the area Text-to-Voice.

Which calls an AI agent should take

Inbound calls come in from customers. Outbound calls go out from your company. The table shows typical uses and what to watch in each case:

Start with inbound calls that come often and carry little risk. Outbound calls with an AI voice need more care, because the person on the other end did not choose to call you. The figure sorts common calls into three groups:

Design the call flow and the handover

A call flow is the planned path of a call, from the greeting to the goodbye. Design it on paper before you configure any tool. Most good flows have four steps: greet, understand the request, handle it and confirm.

The most important part is the exit. A handover passes the call to a person. Good phone agents hand over early and with context. In a warm transfer, the person who takes over receives a short summary, so the caller does not have to repeat everything. In a cold transfer, the call is simply forwarded, and the caller starts again from the beginning.

Five rules keep callers from getting stuck:

• Always offer a way to a person, and say how: “Say ’team’ at any time.”
• Limit the attempts. If the agent has not understood the request after two tries, it hands over.
• Read back critical details: names, dates, times, amounts and phone numbers.
• Pass on the context with every transfer or callback ticket.
• Log every outcome: resolved, transferred, callback or hung up.

Level 3 goes deeper into quality control and escalation paths for all kinds of AI work.

Tell callers it’s an AI and ask before you record

Being open about AI is not only polite. In many places, it is the law.

• AI disclosure. Since August 2, 2026, the EU AI Act requires that AI systems which talk with people directly are designed so that people learn they are dealing with an AI, unless this is obvious (Article 50).
• Outbound calls. In the US, the Federal Communications Commission ruled in February 2024 that AI-generated voices count as “artificial” voices under the Telephone Consumer Protection Act. Calls with them fall under its rules on consent, identification and opt-out (Source: FCC, Declaratory Ruling, 2024, fcc.gov). In Germany, advertising calls to consumers require their prior express consent under the Act against Unfair Competition (UWG).
• Recording. In Germany, recording someone’s private conversation without consent can be a criminal offense (Section 201 of the Criminal Code). In the US, federal law requires the consent of one party to the call, but some states, such as California, require the consent of everyone on the call.
• Transcripts are personal data. Under the GDPR, recordings and transcripts need a purpose, a legal basis, a retention period and clear information for callers.

This overview is not legal advice. Have your legal team or data protection officer check your setup before the first real call.

Sensitive industries: the example of healthcare

Medical, dental and veterinary practices get many calls about appointments, so phone agents are common there. Our comparison lists tools for this in the area Customer Service & Telephony. According to the vendors, Weave is a communications platform for practices with phones, texting and scheduling, and Confido Health offers voice AI agents that answer calls, schedule visits and update the practice software. Vida builds AI agents for calls, texts, email and chat in general, while Calilio and KrispCall are cloud phone systems with AI features:

VidaCustomer Service & Telephony · AI at the coreAI phone agents that answer calls, SMS, email and web chat around the clock and schedule appointments. CalilioCustomer Service & Telephony · Software with AI featuresCloud telephony and call center software with AI features and virtual numbers from more than 100 countries. KrispCallCustomer Service & Telephony · Software with AI featuresCloud phone system with calls, SMS, virtual numbers from more than 100 countries and an AI assistant. WeaveCustomer Service & Telephony · Software with AI featuresCommunications platform for medical, dental and veterinary practices: phone, texting, scheduling and payments. Confido HealthCustomer Service & Telephony · AI at the coreVoice AI agents for healthcare providers: answer calls, schedule visits and update the practice software.

Short descriptions from our tool comparison, based on the vendors' information — not tested by ABXK.

Healthcare shows why sensitive industries need extra rules:

• Health data is specially protected. Under the GDPR, it belongs to the special categories of personal data (Article 9). In the US, a practice that lets a vendor handle protected health information needs a business associate agreement under HIPAA.
• No medical advice. The agent books, moves and reminds. Questions about symptoms, medication or test results go to qualified staff.
• Emergencies come first. If a caller says words like “chest pain” or “can’t breathe”, the agent points them to the emergency number and connects them to staff at once.
• Collect only what the task needs. A booking needs a name, a way to identify the patient and a time, not the full medical history.

The same care applies to banks, insurers and law firms: identity checks, no binding advice and a quick way to a person.

Measure what callers experience

Phone agents produce plenty of numbers, and some of them flatter. The most common one is the containment rate: the share of calls handled without a person. It counts every call that never reached your team, including callers who gave up and callers who had to call again. The example shows the gap:

In the example, 70 percent of calls are “contained”, but only 52 percent were really resolved. Look at these metrics together: