How to Make a Great AI Voice Agent: The Build Rules That Actually Matter

Start with the three parts: ear, brain and mouth

Every voice agent that runs as a pipeline has three parts. Speech to text is the ear: it turns the caller's audio into text. The language model is the brain: it decides what to say next. Text to speech is the mouth: it turns that reply back into audio. One full exchange, caller speaks then agent speaks, is called a turn, and a normal call is dozens of turns.

I teach this first because clients ask technical questions on calls. They want to know why the agent is slow, why it mishears names, why it keeps talking after they interrupt. If you understand the three parts you can diagnose the problem on the call, and a client who sees you diagnose it live trusts you more. If the transcript is wrong, fix the ear. If the transcript is right but the reply is poor, fix the brain. If the reply is right but sounds off, fix the mouth or the phone audio.

In the dashboard I showed in the course, speech recognition took roughly 390 milliseconds, the model roughly 710 milliseconds and the voice roughly 480 milliseconds. Those figures were what the screen displayed on that day for that configuration, not a benchmark. The point is the shape: add them up and you are already past a second before the phone network adds anything.

Latency: aim for 500 to 1,000 milliseconds, and treat 800 as the sweet spot

Latency is the silence between the caller finishing a sentence and the agent starting to answer. It is measured in milliseconds. My rule of thumb from the course: anything between 500 and 1,000 milliseconds is workable, and around 800 feels about right.

Too slow is the problem everyone expects. A one second gap is a little awkward but usable. Beyond that, a human would be a better choice for that call. Too fast is the problem nobody expects. If the agent replies the instant you stop breathing, it talks over the caller and feels pushy, because real people leave a short pause.

Telephony makes this harder. A browser test, which uses web audio, usually has lower latency than a phone call, because the voice also has to travel over the carrier network. In my estimate that adds on the order of 600 milliseconds, and it depends heavily on the provider. I have tried several telephony providers and they are not equal. This is my rough estimate and not a controlled measurement, so measure your own line.

The model is the slowest part, so do not use a thinking model

In a single turn, speech recognition and speech generation are usually quick. The language model is the slowest piece, for the same reason a person takes longest when they have to think about a hard question. Thinking models are built to reason before they answer. That is exactly what you do not want when the caller said 'hey, what's up'. A reasoning model can push a turn to three, five or even ten seconds, and it costs more.

My rule: never use a thinking model for a conversational voice turn. Use the fast tier from your provider, the 'instant', 'flash' or 'mini' class. In the course my picks were Claude Haiku, Gemini Flash and a GPT mini class model. Model names and versions change every few months, so remember the principle and not the names: the quick, cheap tier answers fast enough to hold a conversation, and the large reasoning tier does not.

If you really need heavy reasoning for something, such as analysing a call after it ends, do it after the call, not between the caller's sentences.

Live models: one model that listens, thinks and speaks

A live (speech-to-speech) model has all three parts baked into a single model. It listens, thinks and speaks without a separate transcription step and a separate voice step. Gemini's live model is the example I use in the course. In my experience, live models give better latency and noticeably better prompt adherence, meaning they follow the instructions in the prompt more faithfully.

The trade is control. In a pipeline you can swap the ear, brain and mouth independently and tune each. With a live model there are fewer knobs, which is good for simplicity and worse if you need to change one layer. Cost also behaves differently, mostly through how large the prompt is. I cover that in the cost guide linked at the bottom.

Start with whichever you can get to sound good on a phone line. Do not choose a live model because it is new, and do not choose a pipeline because it is familiar.

Write the prompt like a person, then trim it

My process is simple. I ask an AI assistant to write the first prompt for the specific business, for example a receptionist for a personal injury law firm that takes basic details, books a consultation and is empathetic. I give it my own voice agent playbook so it follows the rules I trust. Then I read the output, delete what the agent does not need, and test it by talking to it.

A few things made the biggest difference in the course:

Give it a personality in words. A voice agent has no personality of its own, so it sounds flat. Tell it to be warm, professional, calm or bubbly, whatever the business needs. Tell it to use filler sounds such as 'um' and 'ah' sparingly. That makes the voice feel human. It does not fix a wrong answer, so use it for tone only.

Keep it short. I aim for roughly 3,000 to 5,000 tokens. A prompt gets read again on every turn, so a bloated prompt costs money and adds delay.

Delete the extras the AI adds. When I asked for a better prompt, it also added example conversations, a JSON block and CRM logging instructions. I removed all of that, because the agent does not log to the CRM by itself. A tool does that, and pasting it into the prompt only makes the prompt longer.

Turn interruption on. If you do not allow interruption, the agent keeps talking over the caller and they cannot cut in.

Use a global prompt for rules that apply to every step, such as tone, and keep step-specific instructions in the step itself. Tools cannot be attached to the global prompt, only to the nodes that act.

Keep big knowledge files out of the live call

It is tempting to upload every brochure and policy so the agent knows everything. Resist that for a narrow job. Searching a large document in the middle of a conversation adds delay, and delay is the thing callers feel most. Imagine reading a hundred page book while someone waits on the phone. Chunking helps but does not remove the wait.

For a personal injury intake agent I attached nothing. The few facts it needed lived in the prompt. If an agent genuinely needs a large knowledge base, treat it as a separate latency test and measure the slowest turn, not the average.

Tools: how the agent actually does things

A prompt can make an agent sound right. Only a tool lets it do something. The booking pattern I build in the course has four parts: the agent decides to call a tool, the tool sends a request to a webhook, an n8n workflow runs, and the workflow books the slot on cal.com and returns a result.

Here are the mistakes I hit live, in the order they hurt:

Test the trigger before you build on it. I click 'test tool' first to confirm the workflow fires at all. If the trigger is broken, everything after it fails and you waste an hour looking in the wrong place.

Match the method. If the webhook is set to POST, the tool must send POST. If it is set to GET, send GET. Mixing them gives a 'not registered' error.

Publish the workflow. Saved is not live. When I forgot, the tool returned 'not found' even though everything looked correct.

Set the webhook to wait for the Respond to Webhook node. Otherwise the agent fires the tool and moves on without waiting for the answer, and an unused response node throws an error.

Describe every parameter. I define client name, phone number, email and call time as parameters, each with a plain description, and the agent will not fire the tool without them. If you hardcode a value, every caller is booked under that name.

Send the time as ISO 8601 with an offset. Calendar APIs expect a precise format. A loose phrase like 'tomorrow at 11' will not work.

Add the country code. The booking failed until the phone number included +91. I now tell the agent to always add the country code if the caller does not say it.

Strip spaces from email addresses. Speech transcription loves to add spaces. I tell the agent to remove them. When a variable came through as undefined, converting it to a string fixed it.

Give the agent the current date and time. At first it booked a call in 2024, because a language model does not know today's date. Dograh has current time variables. I pass one into the prompt and the agent stops booking in the past.

Check availability once, not on every turn

The common way to check a calendar is to make the agent call a tool every time the caller mentions a time. That produces the same awkward 'give me a moment, let me check' on every turn. There is a better design.

Fetch the booked times for the next seven days. Convert them to the local timezone, because APIs usually return UTC. Strip out everything the agent does not need, such as attendee names and emails. Aggregate the many items into one list. Pass the list to a small AI step that writes a single sentence: these times are booked, everything else is free. Return that sentence to the agent through the Respond to Webhook node.

Now the agent holds the booked times in its context and can answer a caller about any time without another tool call. It also saves tokens and removes dead air. I limit the booking window to seven days and write that rule into the prompt, so the agent politely refuses anything further out.

Test on a real phone number, not just in the browser

A browser test proves the prompt. It does not prove the carrier path, the delay, the number routing or the booking. Before I call an agent ready, I attach a phone number, publish the agent to that number and call it from my mobile.

Publish matters here too. If the agent is not published to the number, a generic agent picks up and you spend ten minutes wondering what went wrong. Then complete the real task: book the appointment, change your mind, correct a wrong email, interrupt mid-sentence, stay silent. Read the transcript and check the calendar.

My first version of the law firm agent, I scored about a six or seven out of ten. It was good enough to demo and not good enough to sell. Scoring yourself honestly is what pushes you to the next iteration.

Settings most builders skip

• Turn detection: transcription-based was the setting I recommended in the course. Test it with people who pause mid-sentence.
• Ambient noise: an optional background sound can make a call feel like it came from an office. Use it only if it helps.
• Voicemail detection: for outbound, I disconnect when voicemail is detected. Talking to a voicemail wastes tokens and minutes.
• Context compaction: on a very long call the context grows and every reply gets slower. Compaction summarises the history so replies stay quick.
• Template variables: pass values such as the caller's name into the prompt, so the agent can say it correctly.
• Call schedule and circuit breaker for campaigns: restrict calling hours and stop a campaign automatically if too many calls fail.
• Concurrency: set it to what your phone provider actually allows. The default may be higher than your plan.

Outbound is different: do not make the AI cold sell

Outbound calling is easy to build. You create the agent, add a booking tool and a calendar check, upload a CSV and start the campaign. The hard part is deciding who it calls.

In my experience an AI voice agent should not cold sell to strangers. People can tell it is an AI within a minute or two, and cold lists are already burned. I use outbound voice on warm leads: people who filled in a website form or replied to an ad, where a fast call actually helps. The goal of the call is not to close. It is to qualify the person and convince them to speak to a human. Even that saves a lot of staff time.

Two practical CSV notes from the course. The phone column must be named exactly as the platform expects, lowercase, with no spaces in the numbers. And in a spreadsheet, type an apostrophe before a number that starts with a plus sign, otherwise the sheet turns the plus into an equals sign.

What this guide does and does not prove

• The course is a recorded walkthrough with visible mistakes and fixes, not a controlled benchmark. Latency figures are examples from specific screens and days.
• Model names, platform features and prices change quickly. Rely on the principles and re-check any specific vendor claim before you use it.
• I describe what worked for me. I do not claim results for your business, and no deployment count or revenue figure from the videos is used here as evidence.
• Recording, consent and calling rules differ by country and industry. Get legal advice for your own deployment.

Keep going

If you want the full build from zero, read the beginner guide. If you are worried about the bill, read the cost guide. If you would rather have it built than build it, book a call and we will scope it with you.

• How to make an AI voice agent as a beginner
• How to reduce the cost of your AI voice agent
• AI voice agents for enterprise clients
• Natural-sounding AI voice agents
• Have RapidXAI build your voice agent
• Talk to the team