Building Real-Time Voice AI Agents with Twilio and Deepgram in Node.js
Step-by-step guide to connecting Twilio Media Streams, Deepgram transcription, and LLMs over low-latency WebSockets.
Conversational Voice AI agents are transforming customer experiences by making citizen and customer interactions instant. Building a low-latency, real-time voice bot requires connecting multiple async pipelines. This article details how to channel Twilio audio streams over WebSockets to Deepgram, route transcribed speech to an LLM, and return immediate synthesized speech.
Architecture Overview
The voice pipeline is structured into three asynchronous loops: 1. Twilio establishes a phone call and streams inbound audio over a persistent WebSocket connection in raw mu-law format. 2. The Node.js WebSocket server forwards this audio stream to the Deepgram Transcription API for sub-second Speech-to-Text. 3. The transcribed text is sent to an LLM (such as GPT-4 or Claude via OpenRouter) to generate a response, which is converted to audio and sent back to Twilio.
Handling Twilio Media Streams
Twilio uses TwiML to initiate a WebSocket connection. Once active, Twilio sends JSON frames containing base64 encoded audio payloads. We parse these frames and stream the binary buffers to Deepgram.
// Initiating Twilio WebSocket connection in Node.js/Express
app.post('/incoming-call', (req, res) => {
res.type('text/xml');
res.send(`
<Response>
<Connect>
<Stream url="wss://yourdomain.com/media-stream" />
</Connect>
</Response>
`);
});Deepgram Low-Latency Streaming
Deepgram's real-time streaming endpoint accepts raw audio buffers over WebSockets. It returns transcriptions in real-time, firing callback events containing interim and final transcripts.
// Forwarding raw buffers to Deepgram
const { createClient } = require("@deepgram/sdk");
const deepgram = createClient(process.env.DEEPGRAM_API_KEY);
const connection = deepgram.listen.live({
encoding: "mulaw",
sample_rate: 8000,
channels: 1,
});
connection.on("transcript", (data) => {
const text = data.channel.alternatives[0].transcript;
if (data.is_final && text.trim().length > 0) {
console.log("Transcribed Text:", text);
// Send text to LLM pipeline...
}
});// Summary
By chaining Twilio, Deepgram, and LLM routes together in Node.js, we create responsive conversational systems. Minimizing package hops and using fast WebSockets ensures the overall audio loop response remains well under 1.5 seconds, mimicking natural human conversations.