To the end user, a callbot looks like an intelligent answering machine that asks questions and understands the answers. From the inside, it is a technology chain of five distinct components that must all work in perfect synchronisation, with a cumulative latency below 800 milliseconds to stay credible. Beyond one second, the caller thinks the line has dropped and hangs up.

This article takes the mechanics apart step by step, without needless jargon, to help you understand what happens between the moment you say "hello" and the moment the callbot answers you.

How does a callbot work?

A callbot works by chaining five components together: (1) SIP telephony which receives the call and provides the audio stream, (2) STT (Speech-to-Text) which transcribes the voice into text, (3) the LLM (Large Language Model) which understands the request and formulates a response, (4) TTS (Text-to-Speech) which turns the text response into a synthetic voice, and (5) business integrations (calendar, CRM, database) to carry out the actions. The whole loop must complete in under 800 milliseconds to stay natural.

Operating diagram

┌──────────┐
│  Caller  │ "Hello, I have a water leak"
└────┬─────┘
     │ Phone audio
     ▼
┌─────────────────┐
│  SIP telephony  │ (Asterisk, OVH SIP Trunk)
│  Reception      │
└────────┬────────┘
         │ Raw audio stream (PCM 8 kHz)
         ▼
┌─────────────────┐
│  STT (Voxtral)  │ → Silence detection
│  Transcription  │ → "Hello I have a water leak"
└────────┬────────┘
         │ Text
         ▼
┌─────────────────┐
│  LLM (Claude)   │ → Context analysis
│  Understanding  │ → Decides: URGENT, ask question
└────────┬────────┘
         │ Response text
         ▼
┌─────────────────┐
│  TTS (Piper)    │ → Cloned voice
│  Voice synthesis│ → "I understand, this is urgent..."
└────────┬────────┘
         │ Synthetic audio
         ▼
┌──────────┐
│  Caller  │ (hears the response)
└──────────┘

Step 1: SIP telephony

It all starts with receiving the call. The callbot needs a phone number to which calls are routed. Technically, this is a SIP number (Session Initiation Protocol), the standard for IP telephony. Carriers offer "SIP trunks": for a few euros a month, you rent a geographic number (09 72 xx xx xx for VoIP services in France) that points to your infrastructure.

The incoming call is received by an Asterisk server (the reference open-source software), which handles answering, internal routing, and provides the raw audio stream to the rest of the chain. The latency at this stage is minimal (10-30 ms).

Codec and audio quality

Traditional phones use low-resolution codecs (G.711 at 8 kHz, or G.722 at 16 kHz). For a callbot, G.722 is preferred when the carrier supports it, as it offers double the audio bandwidth (up to 7 kHz vs 3.4 kHz for G.711), with far better intelligibility for speech recognition.

Step 2: speech recognition (STT)

The audio stream is streamed to an STT (Speech-to-Text) engine, which transcribes speech into text as it goes. The main engines in 2026:

  • OpenAI Whisper and its derivatives (faster-whisper, distil-whisper): the open-source benchmark, excellent quality, 200-500 ms latency
  • Voxtral (Mistral AI): a French open-source alternative, optimised for French, low latency on GPU
  • NVIDIA Parakeet: an ultra-fast transducer (latency <200 ms), excellent for real-time streaming
  • Google Speech-to-Text, Azure Speech: commercial, high quality, cloud dependency

The main technical challenge: end-of-speech detection. How does the AI know the caller has finished their sentence and that it can respond? A VAD (Voice Activity Detection) is used, coupled with a semantic detector. Too early = the callbot cuts the caller off; too late = awkward silences.

Step 3: understanding (LLM)

Once the text is transcribed, it is sent to an LLM (Large Language Model) which analyses the request, consults the context (conversation history, business profile, business rules), and formulates the appropriate response. The LLMs used in production in 2026:

  • Anthropic Claude (Sonnet, Haiku): excellent French conversational quality, reasonable latency
  • OpenAI GPT-4o / GPT-4o-mini: the long-standing benchmark, high quality
  • Mistral (Large, Medium): a French alternative, sovereignty
  • Local LLMs (Llama 3, fine-tuned Mistral 7B): for full sovereignty, lower but acceptable quality

The system prompt is the key element: it defines the callbot's personality, its trade, its rules ("if life-threatening emergency, transfer immediately to a human"), the fields to collect, and the available actions (book an appointment, send an SMS, transfer the call).

Tool use and concrete actions

A modern LLM does not just return text: it can call functions (tool use). Concrete examples: check_availability(date) which queries Google Calendar, book_appointment(date, name, phone) which creates the appointment, send_sms_summary(content) which notifies the professional. This is what turns the callbot from a simple voice FAQ into an assistant capable of real actions.

Step 4: voice synthesis (TTS)

The LLM's text response then has to be converted into a synthetic voice. The TTS engines in 2026:

  • ElevenLabs: top quality, photorealistic cloned voices, variable latency (300-700 ms when streaming)
  • OpenAI TTS: high-quality neutral voices, low latency
  • Piper: open-source, CPU-optimised, very low latency (<200 ms), good quality after fine-tuning
  • XTTS-v2 (Coqui): open-source, zero-shot cloning, variable quality

For a callbot, streaming TTS is crucial: starting to speak as soon as the first words are synthesised, without waiting for the full sentence to finish. Otherwise, the cumulative latency goes past 2 seconds and the experience becomes frustrating.

Step 5: business integrations

The callbot must also connect to your business ecosystem:

  • Calendars: Google Calendar, Outlook 365, Doctolib
  • CRM: HubSpot, Pipedrive, Salesforce, or a simple internal database
  • Notifications: SMS via Twilio/OVH, emails via SendGrid/Mailgun, mobile push
  • Business databases: to check a customer file, a status, a history

These integrations are often what separates a "toy" callbot from a "production" callbot. Without a calendar integration, no real appointment booking. Without an SMS integration, no smooth notification to the professional.

The latency constraint

Golden rule: the total latency between the end of the caller's sentence and the start of the callbot's response must stay under 800 milliseconds. Beyond that, the caller thinks there is a line problem and starts talking again or hangs up. Under 500 ms, the experience is genuinely natural.

A typical breakdown:

ComponentTarget latency
End-of-speech detection (VAD)100-200 ms
STT (transcription)100-300 ms (streaming)
LLM (response)200-500 ms (first token)
TTS (first audio)100-300 ms (streaming)
Total500-1300 ms

To stay under 800 ms, streaming is used heavily at every stage (streaming STT, streaming LLM, streaming TTS), and each component is optimised individually. Without streaming, the total regularly exceeds 2 seconds: unusable.

Hosting: cloud or local?

Two philosophies coexist:

  • All cloud: cloud STT (Whisper API), cloud LLM (Claude API), cloud TTS (ElevenLabs). Simple to set up, pay-as-you-go, dependent on the providers and their availability, problematic for GDPR if servers are outside the EU.
  • Local + cloud mix: local STT (faster-whisper on GPU), cloud or local LLM (Claude API or local Mistral), local TTS (fine-tuned Piper). More complex, infrastructure to maintain, but sovereignty and very low marginal cost per use.

Accueil IA uses the mix: local STT (Voxtral GPU + faster-whisper CPU fallback), cloud LLM (Claude API) or local (fine-tuned Mistral as an option), local TTS (fine-tuned Piper cloned voice). 100% France-based hosting.

Going further

Related articles: GDPR compliance for a voice AI, clone your voice for a callbot, AI for business inbound calls.

To hear the result of this technical chain first-hand, call 09 72 10 55 19 (Accueil IA demo). To test it with your own trade, create a free account in 2 minutes.