AI Voice Agents: Architecture and Testing

An AI voice agent receives spoken input, decides what to say or do, and returns a spoken response. The architecture matters: a speech-to-speech system handles audio directly, while a chained system turns speech into text, runs a text workflow, and converts the answer back to audio. The right choice depends on whether your team needs conversational responsiveness or detailed control over each stage. OpenAI’s voice-agent guide describes both approaches.

What happens during a call?

A voice interface receives audio and identifies the caller’s turn. The application supplies instructions and, if needed, calls a tool such as an appointment lookup. It then produces an answer. Production designs also need interruption handling, a way to recover from unclear speech, and a clear handoff when the request is outside the system’s scope.

A voice model is not a substitute for authorization. A tool should check what the caller is allowed to do before reading or changing an account. A model’s statement that a caller is verified is not itself evidence of verification.

Choose a path

Question	Speech-to-speech	Chained speech and text
What travels through the conversation model?	Audio	Text produced by a transcription step
What can the team inspect between stages?	Depends on the implementation and available logs	The intermediate text can be reviewed or filtered before the next stage
Main trade-off	Natural audio interaction, with fewer independently replaceable stages	More explicit control over stages, with additional integration work

Neither choice guarantees a particular response time or transcription accuracy. Measure those outcomes with your own languages, microphones, background noise and call flows.

Worked example: appointment enquiries

Imagine a clinic wants a system that answers opening-hours questions and checks appointment availability. This is a hypothetical design, not a deployed case study.

Start with read-only opening hours from an approved source. For an appointment request, ask for the minimum information needed to find an available time.
Let a tool return available slots; the agent can explain them. Do not book or cancel anything until the application has checked the caller’s identity and the user confirms the selected action.
If the lookup fails, the caller interrupts repeatedly, or the question concerns medical advice, offer a transfer or a callback route rather than guessing.
Test a small set of calls: clear speech, noisy speech, an interruption, an invalid date, an unavailable tool and a request outside the allowed scope. Record what happened and fix the failure before expanding the use case.

Privacy, security and evaluation

Decide which audio or transcripts are retained, who can access them, and when they are deleted before deploying. Rules for recording and consent depend on the relevant jurisdiction and should be checked for the actual use case. Protect tool credentials on the server and limit what each tool can do. OWASP describes the risks of excessive agency.

A useful evaluation checks more than whether the agent sounds natural: did it understand the task, follow the rules, call only allowed tools, recover from errors and hand off appropriately? OpenAI’s realtime evaluation guide explains why multi-turn and tool-failure tests matter.

A voice agent is a poor fit when every decision requires specialist judgement or when the organisation cannot provide a reliable human fallback. Begin with a narrow, measurable task and expand only after the evaluation supports it.