Streaming & realtime
Transcribe live audio and generate speech as text arrives.
Realtime audio usually starts with transcription: send microphone or call audio as it arrives, and receive partial and final text. You can then send generated text to TTS and play the audio response as it is produced.
For the exact client and server frames used by each WebSocket route, see the realtime WebSocket reference.
Stream transcriptions
The ElevenLabs and OpenAI SDKs both connect to AllModels over WebSockets. You can also connect directly if you do not need an SDK.
For the lowest latency, send mono, signed 16-bit PCM without a WAV header. Use 16 kHz audio with the ElevenLabs-compatible endpoint and 24 kHz audio with the OpenAI-compatible endpoint. The examples send 100 ms at a time: 3,200 bytes at 16 kHz or 4,800 bytes at 24 kHz. If your source is 8 kHz telephony audio, ulaw_8000 is also available with models that support it.
Stream audio with the ElevenLabs JavaScript SDK
Send 16 kHz PCM and receive partial and committed transcripts.
import { readFile } from "node:fs/promises";
import { AudioFormat, CommitStrategy, ElevenLabsClient } from "@elevenlabs/elevenlabs-js";
const client = new ElevenLabsClient({
apiKey: process.env.ALLMODELS_API_KEY,
baseUrl: "https://api.allmodels.io/el"
});
const connection = await client.speechToText.realtime.connect({
modelId: "deepgram/nova-3",
audioFormat: AudioFormat.PCM_16000,
sampleRate: 16000,
languageCode: "en",
commitStrategy: CommitStrategy.MANUAL
});
connection.on("partial_transcript", ({ text }) => {
process.stdout.write(text);
});
connection.on("committed_transcript", ({ text }) => {
console.log(`\n${text}`);
connection.close();
});
const audio = await readFile("speech-16khz.pcm");
for (let offset = 0; offset < audio.length; offset += 3200) {
connection.send({
audioBase64: audio.subarray(offset, offset + 3200).toString("base64")
});
}
connection.commit();Stream generated speech
Use ElevenLabs-compatible or native WebSockets when text arrives incrementally. OpenAI speech takes complete text per request, so its example buffers generated text into speakable chunks and streams the requests in order. Every example writes raw 24 kHz PCM to reply.pcm.
Stream generated text with the ElevenLabs Python SDK
import os
from elevenlabs.client import ElevenLabs
client = ElevenLabs(
api_key=os.environ["ALLMODELS_API_KEY"],
base_url="https://api.allmodels.io/el",
)
def generated_text():
# Yield text here as your LLM or application produces it.
yield "Your order has shipped. "
yield "It should arrive tomorrow afternoon."
audio = client.text_to_speech.convert_realtime(
"eve",
text=generated_text(),
model_id="grok/grok-tts",
output_format="pcm_24000",
)
with open("reply.pcm", "wb", buffering=0) as output:
# pcm_24000 is headerless mono, signed 16-bit little-endian PCM.
for chunk in audio:
output.write(chunk)