ElevenLabs Speech Engine

Speech Engine connects the ElevenLabs conversation client directly to AllModels for microphone input and audio playback. Your application receives completed transcripts at its WebSocket URL and streams the agent's response text back.

Create the Speech Engine on your server, keep the AllModels API key there, and return only the signed URL to the client.

Create a Speech Engine

Set the ElevenLabs client base URL to https://api.allmodels.io/el and authenticate with an AllModels API key. This example uses Deepgram for transcription and Cartesia for synthesis.

The speech_engine.ws_url value must be your publicly reachable wss:// application endpoint.

import { ElevenLabsClient } from "@elevenlabs/elevenlabs-js";

const apiKey = process.env.ALLMODELS_API_KEY;
const baseUrl = "https://api.allmodels.io/el";
const client = new ElevenLabsClient({ apiKey, baseUrl });

const engine = await client.speechEngine.create({
  name: "Support agent", // optional
  speechEngine: {
    wsUrl: "wss://app.example.com/voice-agent"
  },
  asr: {
    userInputAudioFormat: "pcm_16000" // optional
  },
  tts: {
    voiceId: "YOUR_CARTESIA_VOICE_ID",
    agentOutputAudioFormat: "pcm_16000" // optional
  }
}, {
  queryParams: {
    stt: { model: "deepgram/nova-3" },
    tts: { model: "cartesia/sonic-3" }
  }
});

const { signedUrl } =
  await client.conversationalAi.conversations.getSignedUrl({
    agentId: engine.engineId
  });

The stt and tts query options are AllModels extensions accepted by the ElevenLabs SDK's request-options argument. The JavaScript SDK serializes these nested objects as bracketed query parameters. A canonical model ID automatically selects an eligible provider. Add an optional provider, provider_options, or portable input settings inside either object only when needed.

Start the conversation

Give the signed URL to the browser and start the official ElevenLabs conversation client. The browser sends microphone audio directly to AllModels and plays synthesized audio as it arrives.

import { Conversation } from "@elevenlabs/client";

const conversation = await Conversation.startSession({
  signedUrl,
  connectionType: "websocket" // optional
});

// Later, when the user ends the call:
await conversation.endSession();

Use any STT and TTS models marked as streaming-capable in the model catalog. See the API reference for creating a Speech Engine, creating a signed URL, and connecting the client.

Configure AllModels settings

Pass AllModels configuration through the ElevenLabs SDK's queryParams request option. Keep standard ElevenLabs settings such as the voice and audio formats in the request body.

const engine = await client.speechEngine.create({
  speechEngine: {
    wsUrl: "wss://app.example.com/voice-agent"
  },
  asr: {
    userInputAudioFormat: "pcm_16000" // optional
  },
  tts: {
    voiceId: "YOUR_VOICE_ID",
    agentOutputAudioFormat: "pcm_16000", // optional
    speed: 1.1 // optional
  },
  language: "en" // optional
}, {
  queryParams: {
    stt: {
      model: "deepgram/nova-3",
      provider: "deepgram", // optional
      input: {
        turn_detection: "acoustic",
        silence_duration_ms: 500
      }, // optional
      provider_options: {
        numerals: true
      } // optional
    },
    tts: {
      model: "cartesia/sonic-3",
      provider: "cartesia", // optional
      input: {
        text_buffer_mode: "sentence"
      } // optional
    }
  }
});

Each stt or tts object accepts:

• model: canonical AllModels model ID.
• provider: optional provider pin. Omit it to let AllModels select an eligible provider.
• input: optional portable streaming settings that do not have a standard Speech Engine field.
• provider_options: optional settings passed to the selected provider.

The SDK serializes the nested objects into query parameters such as stt[model] and stt[provider_options][numerals]. Model selection and these settings are saved with the Speech Engine when it is created or updated.

Use a standard Speech Engine field whenever one exists. In the example, language and tts.speed are native to the ElevenLabs SDK. Deepgram's numerals option has no Speech Engine equivalent and therefore belongs in stt.provider_options; AllModels text buffering likewise belongs in tts.input. Query parameters are only needed for AllModels model selection, portable settings without a Speech Engine field, and provider-specific extensions. If the same setting is supplied both ways, the native Speech Engine field takes precedence.

Supported native parameters

Pass these standard ElevenLabs SDK fields in the first argument to speechEngine.create() or speechEngine.update():

asr.quality, asr.provider, and the deprecated tts.optimizeStreamingLatency are accepted for ElevenLabs SDK compatibility but do not select an AllModels model or provider. tts.supportedVoices, tts.expressiveMode, tts.suggestedAudioTags, and tts.textNormalisationType: "system_prompt" are not currently supported; signing a session with one of these settings returns an error instead of silently ignoring it.

Supported query parameters

The second argument to the JavaScript SDK method accepts nested queryParams. Python uses request_options.additional_query_parameters with the serialized bracketed names.

Supported stt.input fields:

• commit_strategy
• turn_detection
• context
• channels
• silence_duration_ms
• min_speech_duration_ms
• min_silence_duration_ms
• turn_timeout_ms
• speech_threshold
• turn_threshold
• eager_end_threshold
• interim_results
• include_timestamps
• include_language_detection
• diarization
• formatting
• include_filler_words

Supported tts.input fields:

• text_buffer_mode: adaptive, sentence, or none
• text_buffer_min_characters
• text_buffer_target_characters
• text_buffer_max_characters

For example, JavaScript:

{
  queryParams: {
    stt: {
      model: "deepgram/nova-3",
      input: { silence_duration_ms: 500 },
      provider_options: { numerals: true }
    },
    tts: {
      model: "cartesia/sonic-3",
      input: { text_buffer_mode: "sentence" }
    }
  }
}

The equivalent Python request options are:

request_options={
    "additional_query_parameters": {
        "stt[model]": "deepgram/nova-3",
        "stt[input][silence_duration_ms]": 500,
        "stt[provider_options][numerals]": True,
        "tts[model]": "cartesia/sonic-3",
        "tts[input][text_buffer_mode]": "sentence",
    }
}

Provider options are forwarded to the selected provider. Consult that provider's documentation for accepted names and values. Do not repeat native fields such as language, audio format, voice, or speed in provider_options; native Speech Engine fields take precedence.