How do I build an AI voice agent that answers phone calls?

By Naol Ketema, founder · 2026-10-21 · Markdown

The parts of a voice agent

An AI voice agent that answers calls has four jobs: pick up the call, turn the caller's speech into text, decide what to say, and turn that text back into speech. Underneath sits telephony: a number, a carrier, audio streaming, and handling people who talk over the agent.

Most teams only want to own the middle step, the decision. Agent Identity handles the number, the call, speech to text and text to speech, and lets you choose who does the deciding.

Option 1: Aid Voice, no server

Aid Voice is the built-in voice agent. New numbers answer with it by default (incoming_call_action hosted_agent). You give it standing instructions (up to 8,000 characters) and an optional greeting, and it answers every call with them.

import { AgentClient } from "@agentidentity/sdk";

const org = new AgentClient({ baseUrl: "https://api.agent-identity.dev", apiKey: process.env.AID_ORG_KEY! });
const agentId = process.env.AID_AGENT!;

await org.phone.setHostedAgentConfig(agentId, {
  instructions: "You answer for Blue Harbor Dental. Be warm and brief. Never quote prices; offer to email the price list.",
  greeting: "Thanks for calling Blue Harbor Dental, how can I help?",
  voicemailMode: "leave_message",
  voicemailGreeting: "Sorry we missed you. Please leave a message after the tone.",
});

At the start of every call Aid Voice gets a briefing: the agent's name, handle, email and number; the caller's number, plus their name, company and notes if the number belongs to one of your contacts; and the time in UTC. Unknown callers are greeted neutrally.

• It can press keypad keys (0-9, * and #) to get through phone menus.
• It ends the call once the conversation is over.
• It records follow-ups: work for your agent to do after the call, each with an action and details.
• It cannot run your own tools during the call. That is the main limit.

Option 2: your own agent over a WebSocket

Set the incoming call action to auto_accept with a wss:// address. When a call is answered, Agent Identity opens a WebSocket to your server and sends JSON frames. By default this is text mode: you get transcript frames with what the caller said, and you send text frames that are spoken back. Your code never touches audio.

await org.phone.setIncomingCallAction(agentId, {
  incomingCallAction: "auto_accept",
  clientWebsocketUrl: "wss://agent.example.com/stream",
});

import { WebSocketServer } from "ws";
import type { PhoneStreamToAgent, PhoneStreamFromAgent } from "@agentidentity/sdk";

new WebSocketServer({ port: 8080 }).on("connection", (socket) => {
  const send = (frame: PhoneStreamFromAgent) => socket.send(JSON.stringify(frame));

  socket.on("message", async (data) => {
    const frame: PhoneStreamToAgent = JSON.parse(data.toString());

    if (frame.event === "transcript" && frame.is_final) {
      const reply = await yourModel(frame.text); // your LLM, your tools
      send({ event: "text", delta: reply, done: true, turn_id: frame.turn_id });
    }
    if (frame.event === "barge_in") {
      // The caller talked over the agent. Speech has already stopped; stop generating.
    }
  });
});

• transcript frames carry text, is_final and turn_id. Partial results arrive while the caller is still talking; the final one ends the turn.
• Send text in pieces with done: false and a last piece with done: true. It is spoken a sentence at a time as soon as each sentence is complete.
• barge_in means the caller interrupted. Text that still arrives for the interrupted turn_id is dropped.
• Send dtmf to press keys, clear to stop speaking, and stop (optionally with after_speech: true) to end the call.
• The connection carries an X-Call-Context header with the call id, the agent's number, the direction and contacts, signed with the agent's signing secret. Verify it before trusting it.

If you want your own speech recognition or a speech-to-speech model, reply to the handshake with X-Use-AID-Speech-To-Text: false, X-Use-AID-Text-To-Speech: false, or both, and you exchange raw PCMU audio at 8 kHz instead.

The voice is part of the incoming call setting and applies to Aid Voice and to text your agent sends. listVoices returns the catalog with each voice's id, name, description and language; save the id as voice and its language as language.

Other ways to handle an incoming call

• Your server (webhook): each call is POSTed to your HTTPS endpoint, which has 5 seconds to reply answer or reject. A slow or non-2xx reply declines the call. Useful for routing per caller.
• Forward: send the call to a human's phone number or a SIP address. The destination sees the agent's number as caller ID.
• Decline (auto_reject): refuse the call, or take a voicemail if voicemail_mode is leave_message.

Test it before you write a model

Point the agent's client_websocket_url at wss://api.agent-identity.dev/v1/phone/echo-text and call the number from your phone. It greets you and says back what you said, which proves the number, the call and text mode work end to end. wss://api.agent-identity.dev/v1/phone/echo does the same with raw audio.

Which one to choose

• Choose Aid Voice for reception, booking requests, message taking and FAQ-style calls, where instructions plus after-call follow-ups are enough.
• Choose your own agent when the agent must look something up or act during the call, or when you already have an agent and want it on the phone.
• If your server cannot be reached when a call comes in, the caller is hung up on rather than left on a silent line, or sent to voicemail if voicemail_mode is leave_message. Set voicemail as a fallback.

If you already run a full voice stack on a telephony provider and are happy with it, you do not need this. It is most useful when you want calls tied to the same agent identity as its email and Agent2Agent messages, with one event log for all of it.