Build a real-time voice assistant loop

Now that we've seen speech-to-text, text generation, and text-to-speech separately, we're going to put them together.

A voice assistant is a loop: listen, transcribe, answer, speak. Each piece lives in a chapter we've already done. This lesson wires the three pieces into a streaming conversation.

Three models to load: ASR with Whisper + Silero VAD, LLM with Llama 3.2 1B, TTS with Supertonic English. You would load them in this sequence:

const asrModelId = await loadModel({
  modelSrc: WHISPER_TINY,
  modelConfig: {
    vadModelSrc: VAD_SILERO_5_1_2,
    audio_format: "f32le",
    language: "en",
  },
});
const llmModelId = await loadModel({
  modelSrc: LLAMA_3_2_1B_INST_Q4_0,
  modelConfig: { ctx_size: 4096 },
});
const ttsModelId = await loadModel({
  modelSrc: TTS_EN_SUPERTONIC_Q8_0,
  modelConfig: {
    ttsEngine: "supertonic",
    language: "en",
    voice: "F1",
    ttsSpeed: 1.05,
    ttsNumInferenceSteps: 5,
  },
});

The system prompt is what tells the model how to behave: short answers (TTS takes ~1s per sentence), no markdown (asterisks sound bad read aloud), no lists (hard to follow mid-task). The system prompt might look as follows:

const history: Array<{
  role: "system" | "user" | "assistant";
  content: string;
}> = [{ role: "system", content: SYSTEM_PROMPT }];

transcribeStream({ modelId }) returns a Promise<TranscribeStreamSession>. await it to get the session, then iterate with for await (const rawText of session). The session yields plain text strings.

Audio capture is a child ffmpeg process. The startMicrophone() helper spawns the system ffmpeg with the right -i args for the current platform (avfoundation on macOS, pulse on Linux, dshow on Windows) and pipes 16 kHz mono f32le PCM to stdout. Each chunk on stdout is a frame we feed into the session:

const ffmpeg = startMicrophone({ sampleRate: 16000, format: "f32le" });
const session = await transcribeStream({ modelId: asrModelId });

ffmpeg.stdout.on("data", (chunk: Buffer) => {
  session.write(chunk);
});

A startup check on ffmpeg -version and ffplay -version makes the failure mode obvious if either is missing. The for (const tool of ['ffmpeg', 'ffplay']) loop keeps it to a few lines.

The main loop is one async for await over the session. Each iteration produces a user turn, runs the LLM, then speaks the answer:

for await (const rawText of session) {
  const userText = rawText.trim();
  if (userText.length === 0) continue;

  history.push({ role: "user", content: userText });

  const llmResult = completion({
    modelId: llmModelId,
    history,
    stream: true,
  });
  let assistantText = "";
  for await (const token of llmResult.tokenStream) {
    process.stdout.write(token);
    assistantText += token;
  }
  history.push({ role: "assistant", content: assistantText });

  const ttsResult = textToSpeech({
    modelId: ttsModelId,
    text: assistantText.trim(),
    inputType: "text",
    stream: false,
  });
  const samples = await ttsResult.buffer;
  if (samples.length > 0) {
    const wavBuffer = Buffer.concat([
      createWavHeader(samples.length * 2, TTS_SAMPLE_RATE),
      int16ArrayToBuffer(samples),
    ]);
    playAudio(wavBuffer);
  }
}

textToSpeech() returns the audio as raw 16-bit signed PCM samples at 44.1 kHz mono. To play it we wrap the samples in a minimal WAV header and pipe the buffer into ffplay, which ships with ffmpeg.

A cleanup() handler kills the ffmpeg child, ends the session, and unloads all three models. Wire it to SIGINT and SIGTERM so Ctrl+C exits cleanly:

async function cleanup() {
  if (shuttingDown) return;
  shuttingDown = true;
  ffmpeg.kill();
  try { session.end(); } catch {}
  await unloadModel({ modelId: ttsModelId }).catch(() => {});
  await unloadModel({ modelId: llmModelId }).catch(() => {});
  await unloadModel({ modelId: asrModelId }).catch(() => {});
  process.exit(0);
}

process.on("SIGINT", () => void cleanup());
process.on("SIGTERM", () => void cleanup());

The SDK installs its own SIGINT / SIGTERM handler that aborts in-flight RPC streams on shutdown. The abort rejects any pending session.write() or unloadModel() call with a WorkerShutdownError. The bare-rpc socket also emits an RPCError with code: 'CHANNEL_CLOSED' on the same teardown path. Both surface as unhandled stream errors. The process.on("uncaughtException", ...) filter ignores the shutdown noise and re-throws anything else:

process.on("uncaughtException", (err) => {
  if (err instanceof WorkerShutdownError) return;
  if (err?.code === "CHANNEL_CLOSED") return;
  throw err;
});

Note: the system prompt bans markdown and lists because the output is spoken aloud. A ### heading or a 1. list reads as a stuttery mess through TTS.

Put it to the test

1. Make three loadModel() calls for the ASR (Whisper + Silero VAD), LLM (Llama 3.2 1B), and TTS (Supertonic English) models.
2. Declare history with the system prompt as the first message.
3. Gate on ffmpeg and ffplay being on PATH at startup. Define the WAV helpers (createWavHeader, int16ArrayToBuffer, playAudio) and the mic helper (startMicrophone) above main.
4. Open a const session = await transcribeStream({ modelId: asrModelId }) session, then const ffmpeg = startMicrophone(...), then ffmpeg.stdout.on("data", (chunk) => session.write(chunk)).
5. Iterate with for await (const rawText of session). Each iteration: trim the transcript, push it as a user turn into history, call completion() with stream: true, then push the assistant turn into history.
6. After completion() returns, call textToSpeech() and await ttsResult.buffer. Wrap the samples in a WAV header and playAudio(wavBuffer).
7. After the loop, run the cleanup handler. Wire the WorkerShutdownError / CHANNEL_CLOSED filter to uncaughtException so the SDK's shutdown abort doesn't dump a stack.