/realtime

Browser audio over WebSocket

The OpenAI Realtime SDK wires microphone and speaker automatically only for its WebRTC transport. ElevenLabs Realtime is WebSocket-only, so the app captures and plays audio itself. Both directions use PCM16, 24 kHz, mono.

This reference uses wavtools, the helper OpenAI uses in its own WebSocket examples. Any capture path that produces 24 kHz PCM16 ArrayBuffer chunks works, for example an AudioWorklet the app already has.

npm install @openai/agents-realtime wavtools

Implementation

Framework-agnostic; in React, hold the returned handle in a useRef and call stop() on unmount.

import {
  OpenAIRealtimeWebSocket,
  RealtimeAgent,
  RealtimeSession,
} from "@openai/agents-realtime";
import { WavRecorder, WavStreamPlayer } from "wavtools";

const SAMPLE_RATE = 24000;

export async function startVoiceSession() {
  const tokenResponse = await fetch("/api/token");
  if (!tokenResponse.ok)
    throw new Error(`Token request failed (${tokenResponse.status})`);
  const { value: clientSecret } = await tokenResponse.json();

  const recorder = new WavRecorder({ sampleRate: SAMPLE_RATE });
  const player = new WavStreamPlayer({ sampleRate: SAMPLE_RATE });
  await recorder.begin();
  await player.connect();

  const session = new RealtimeSession(
    new RealtimeAgent({
      name: "Assistant",
      instructions:
        "You are a helpful, friendly voice assistant. Keep replies concise.",
    }),
    {
      transport: new OpenAIRealtimeWebSocket({
        url: "wss://api.elevenlabs.io/v1/convai/realtime",
      }),
      model: "eleven_realtime_v1_mini",
      config: {
        outputModalities: ["audio"],
        audio: { output: { voice: "brian" } },
      },
    },
  );

  session.on("audio", (event) => {
    player.add16BitPCM(event.data, event.responseId);
  });
  session.on("audio_interrupted", () => {
    player.interrupt();
  });
  session.on("error", (event) => {
    console.error(event.error);
  });

  try {
    await session.connect({ apiKey: clientSecret });
    await recorder.record((data) => {
      session.sendAudio(data.mono as unknown as ArrayBuffer);
    });
  } catch (err) {
    await stop();
    throw err;
  }

  async function stop() {
    await recorder.quit().catch(() => {});
    player.interrupt();
    session.close();
  }

  return { session, stop };
}

Microphone audio streams continuously; server-side VAD decides when the user's turn ends, so there's no push-to-talk logic to port unless the app had one.

Pitfalls

• data.mono is mistyped. wavtools types the record() callback's mono as Int16Array, but at runtime it's already the PCM16 ArrayBuffer. Pass it straight to sendAudio. Reading .buffer off it, or wrapping it in a new typed array, sends garbage.
• Start from a user gesture. recorder.begin() prompts for the microphone and player.connect() creates an AudioContext. Browsers block audio contexts created outside a click or keypress, so call startVoiceSession() from the handler that used to start the WebRTC session.
• Stop playback on interruption. audio_interrupted fires when the user barges in. player.interrupt() discards audio already queued; without it, the assistant keeps talking over the user.
• Match the sample rate everywhere. Recorder, player, and the mint body's audio.input.format.rate all use 24000. A mismatch plays back too fast, too slow, or as noise.
• Tear down all three. Quit the recorder (releases the microphone indicator), interrupt the player, and close the session. A leftover recorder keeps the microphone open after the conversation ends.

Replacing WebRTC-specific code