Deep dive · AI & automation

Building a real-time AI voice interviewer solo: architecture, latency budget and the parts that bite

How to build an AI voice agent for live interviews: a Pipecat + Daily + Groq architecture, Pipecat vs LiveKit, and a voice agent latency budget with sources.

By Akash Bansal, founding engineerUpdated 4 November 20268 min read

I built NextMantra on my own. It's an AI recruiter that runs the first round of hiring: live voice interviews with candidates, résumé-to-role scoring and a ranked shortlist for the hiring team. It is in production. This post is the engineering write-up: the architecture, where the latency goes, which choices matter and which don't, and what changes when the conversation is a job interview rather than a support call.

What is the architecture of a real-time AI voice agent?

A real-time voice agent is a streaming pipeline, not a request-response API. Audio frames flow in continuously, and every stage passes partial results to the next before the previous one has finished. NextMantra's split looks like this:

• Web app: Next.js for candidates and recruiters, NestJS for the API, PostgreSQL for jobs, candidates, transcripts and scores.
• Interview agent: a Python service on FastAPI running a Pipecat pipeline.
• Real-time audio: Daily WebRTC rooms. The candidate joins from the browser and the bot joins the same room as a participant.
• In-call inference: Groq, chosen for fast token generation.
• After the call: multi-LLM scoring of the transcript against the role's rubric.
• Infrastructure: AWS, defined as code, shipped through an AES-encrypted, tamper-evident deploy pipeline.

The most useful design decision in the whole system is that line between the gold box and everything else. Inside it, every millisecond counts and the model's only job is to run a good conversation. Outside it, nothing is real-time, so I can use slower, stronger models and several of them. I cover that half in the multi-LLM scoring post.

Pipecat vs LiveKit: which framework should you build on?

Both are good, both are open source, and the choice matters less than people think. The real decision is which real-time transport you want to depend on.

I built NextMantra on Pipecat with Daily as the transport. What I'd tell anyone choosing today: write the same three-turn conversation in both, run each from a phone on a weak network, and measure. A day spent on that prototype is cheaper than a framework migration later. Docs: Pipecat, LiveKit Agents.

What is a realistic voice agent latency budget?

Aim for about 800 ms at the median, from the moment the candidate stops speaking to the first audio byte the agent plays. Daily's engineers, who also maintain Pipecat, set that as their target in their write-up on a fast voice bot. They note that around 500 ms is typical between human speakers and that pauses longer than 800 ms feel unnatural. Research on turn-taking across ten languages found the same pattern everywhere: speakers avoid overlap and keep silence between turns short (Stivers et al., PNAS 2009).

Here is how that budget breaks down using public numbers. The right-hand column is a sample allocation of my own, there to show the arithmetic. It is not a measurement of NextMantra.

Two things stand out. The LLM line has the widest range, from 80 ms to 1.5 seconds, so the choice of model and provider decides whether you hit the budget at all. And turn detection is a cost people forget to count, because it's time spent waiting on silence rather than computing.

Where does voice agent latency actually hide?

The stages in the table are well documented. The surprises usually come from the things around them.

Prompt length raises time to first token

An interview agent's prompt is long: the role, the rubric, the candidate's résumé, the question plan and the conversation so far. First-token time grows with input length on every provider. Independent benchmarks such as Artificial Analysis are useful for comparing providers, but read their method: their headline latency is measured with long inputs, so test with your own prompt at your own length. Keep what the model needs for this turn in the prompt and move the rest into state you manage in code.

Waiting for the whole answer before speaking

Stream everything. Send LLM tokens to TTS as soon as you have a sentence, and send TTS audio to the room as soon as you have the first chunk. A pipeline framework does this by default. Hand-rolled glue code often doesn't.

Distance between services

Every hop between your bot, the STT provider, the LLM and the TTS provider is a network round trip. Put the bot in the same region as the vendors it calls, and pick vendors with endpoints near your users.

Cold starts

If a bot process has to boot, load models and open connections when the candidate clicks "join", the first turn is slow even if the rest are fast. Keep warm capacity for the interviews you expect, and measure the first turn separately from the steady state.

What's different when the voice agent is an interviewer?

Most voice-agent guides are written for support and booking bots. If a booking or answering bot is all you need, buying an AI receptionist usually beats building one. An interview breaks some of their assumptions.

1. Candidates pause to think. "I'd approach that by... hmm..." is not the end of a turn. A plain silence timer cuts people off mid-thought, which is both rude and unfair. This is where semantic turn detection (a model that hears whether the utterance sounds finished, such as Smart Turn) earns its place, and where you trade a little latency for not interrupting.
2. The agent must follow a plan. A support bot reacts. An interviewer has to cover specific competencies in a set time. The robust pattern is to keep the interview plan as explicit state in code (which question, which follow-ups are allowed, time left) and give the LLM only the current step. A model given the whole plan at once tends to wander or skip ahead.
3. Barge-in cuts both ways. If the candidate starts talking, the agent should stop. If the agent is cut off by background noise, it should recover without losing its place. Test with real rooms: fans, traffic, a second person speaking.
4. The transcript is evidence. Every word is later used to score a person for a job. Store the full transcript with timestamps, keep audio according to your retention policy, and never let the live model write the score.

Why separate the live model from the scoring models?

Because they have opposite requirements. The live model has to be fast, and it's fine for it to be a smaller model on fast inference, because its output is one conversational turn. The scoring step is a high-stakes judgement about a person, and it has no latency budget at all. It can take a minute, use several models from different families, and flag disagreements for a human. Mixing the two jobs gives you a slow conversation and a rushed judgement.

The research on LLM judges also says single-model scoring has known biases, which is why NextMantra scores with multiple LLMs against the role's rubric. The details are in Multi-LLM scoring: why one model shouldn't grade a human.

How much does a voice AI interview cost to run?

Price it per minute, as a sum of four meters, then add the post-call scoring:

• Transport: WebRTC participant minutes (candidate plus bot).
• Speech-to-text: billed per audio minute by most providers.
• LLM: input and output tokens per turn. The long prompt is re-sent every turn, so input tokens dominate. Prompt caching, where your provider supports it, cuts this.
• Text-to-speech: billed per character or per minute of generated audio.
• Scoring: one pass per model over the full transcript, once per interview.

Pull each vendor's current price page, multiply by your expected interview length, and add compute for the bot. The LLM input-token line is the one that most often surprises people, because it grows with both prompt size and the number of turns.

Can one engineer really build and run this?

Yes, if you buy the hard infrastructure instead of building it. I didn't write a WebRTC stack, a speech model or a pipeline scheduler. Daily handles real-time media, Pipecat handles the streaming pipeline and the vendor integrations, and the model providers handle inference. The work left over is the part that makes it an interviewer and a product: the interview flow, turn-taking that suits people who think out loud, the transcript and scoring pipeline, the recruiter-facing app, and infrastructure as code on AWS so the environment can be rebuilt from the repo.

What I would not do solo is run my own GPUs for speech and LLM inference from day one. Self-hosting can win on latency and cost at scale, as Daily's write-up shows, but it adds a heavy operations burden. Start on providers and move a stage in-house when its numbers justify it.

A checklist before you build a voice agent

1. Write the latency budget first, stage by stage, and log every stage's timing per turn from the first prototype.
2. Prototype the same flow on Pipecat and on LiveKit Agents for a day before committing.
3. Choose turn detection for your conversation type. Interviews need patience. Order-taking needs speed.
4. Keep conversation state in code, and give the model only the current step.
5. Stream every stage, colocate services, and keep capacity warm.
6. Keep high-stakes judgement out of the real-time loop.
7. Price it per minute across all four meters before you price it for customers.

If you're a founder or CTO planning a voice agent and want one engineer to own it end to end, this is the kind of build I do: here's how I work, and there's more technical depth on the engineering page. For the founder's-eye view of the same product, see the AI voice interviewer build story.