OpenAI realtime transcription
Stream PCM audio and receive realtime transcription events with the OpenAI SDK.
Connect to wss://api.allmodels.io/oai/realtime?intent=transcription, configure the session, and send base64-encoded PCM audio. Commit the audio buffer when an utterance is complete. This endpoint returns transcripts only; to turn a transcript into streamed speech, see Stream generated speech.
SDK clients
Use the official OpenAI JavaScript or Python SDK.
import { once } from "node:events";
import { readFile } from "node:fs/promises";
import OpenAI from "openai";
import { OpenAIRealtimeWebSocket } from "openai/realtime/websocket";
const client = new OpenAI({
apiKey: process.env.ALLMODELS_API_KEY,
baseURL: "https://api.allmodels.io/oai"
});
const realtime = await OpenAIRealtimeWebSocket.create(client, {
model: "openai/gpt-realtime-whisper"
});
if (realtime.socket.readyState !== 1) await once(realtime.socket, "open");
realtime.on("conversation.item.input_audio_transcription.delta", (event) => {
process.stdout.write(event.delta);
});
realtime.on("conversation.item.input_audio_transcription.completed", (event) => {
console.log("\nfinal:", event.transcript);
});
realtime.on("error", console.error);
realtime.send({
type: "session.update",
session: {
type: "transcription",
audio: {
input: {
format: { type: "audio/pcm", rate: 24000 },
transcription: {
model: "openai/gpt-realtime-whisper",
language: "en"
},
turn_detection: null
}
}
}
});
const pcm = await readFile("chunk.pcm");
realtime.send({
type: "input_audio_buffer.append",
audio: pcm.toString("base64")
});
realtime.send({ type: "input_audio_buffer.commit" });Connection details
wss://api.allmodels.io/oai/realtime?intent=transcriptionOpenAIThe OpenAI SDK authenticates with the API key passed to OpenAI or AsyncOpenAI. Browser connections can pass realtime and openai-insecure-api-key.<key> as WebSocket subprotocols, but the SDK examples use a backend connection.
Browser credentials are visible
A key placed in a WebSocket subprotocol is available to the browser application and developer tools. Prefer a backend connection for long-lived tenant keys.
Events and buffer flow
| Direction | Event | Purpose |
|---|---|---|
| Client → server | session.update | Configure model, language, audio format, turn detection, provider, and provider options. |
| Client → server | input_audio_buffer.append | Add one base64-encoded audio chunk. |
| Client → server | input_audio_buffer.commit | Finalize the current utterance. |
| Client → server | input_audio_buffer.clear | Drop buffered audio. |
| Server → client | session.created / session.updated | Confirm effective session configuration. |
| Server → client | input_audio_buffer.committed | Confirm a committed item and its ID. |
| Server → client | conversation.item.input_audio_transcription.delta | Partial transcript text. |
| Server → client | conversation.item.input_audio_transcription.completed | Final transcript and duration usage. |
| Server → client | error | Protocol, provider, billing, or connection failure. |
Choose the provider, model, and audio format before sending audio. These settings cannot be changed after the first audio chunk.
Provider selection
Set provider preferences with session.update:
realtime.send({
type: "session.update",
session: {
type: "transcription",
provider: "deepgram",
provider_options: {
keyterms: ["AllModels"],
smart_format: true
},
audio: {
input: {
format: { type: "audio/pcm", rate: 24000 },
transcription: { model: "deepgram/nova-3", language: "en" },
turn_detection: null
}
}
}
});