Stop the voice assistant from hearing itself

The voice assistant from the previous lesson works for a turn or two, then the feedback loop takes over: the TTS output gets picked up by the mic, Whisper transcribes it as a new user turn, and the LLM answers it. Each turn triggers the next with no real user input.

Four things drive that loop:

• The default VAD commits segments too eagerly for long-running use.
• The mic keeps recording through TTS, so the TTS output comes back as a new user turn.
• VAD occasionally commits phantom transcripts from near-silence, single tokens like "you" or "Thanks." with no real speech behind them.
• The TTS audio rings through the speaker into the mic for a moment after the model finishes, getting transcribed as the tail of the user's next turn.

The defaults are tuned for one-shot dictation, not a loop, so the first fix is to override vad_params in the ASR modelConfig:

const vad_params = {
  threshold: 0.6,
  min_speech_duration_ms: 300,
  min_silence_duration_ms: 700,
  max_speech_duration_s: 15.0,
  speech_pad_ms: 200,
};

min_silence_duration_ms: 700 is the value that matters. VAD uses it to decide when the user stopped talking, so a longer quiet window keeps the TTS tail ringing through the speaker from getting folded into the user's turn.

VAD handles the first issue. The other three happen in the loop body, so the fix is three helpers defined right before the main loop:

• isSpeaking flag: the loop checks it at the top of each iteration to skip frames while TTS plays
• isMeaningfulTranscript: filter that drops the empty and phantom transcripts before they reach the LLM
• sleep(ms) helper: the main loop calls it to wait for the post-playback cooldown.

Here's how that looks in code:

const POST_PLAYBACK_COOLDOWN_MS = 300;
const MIN_UTTERANCE_CHARS = 3;

function isMeaningfulTranscript(text: string): boolean {
  const trimmed = text.trim();
  if (trimmed.length === 0) return false;
  if (trimmed.includes("[No speech detected]")) return false;
  if (/^\[[^\]]+\]$/.test(trimmed)) return false;
  const letters = trimmed.replace(/[^\p{L}\p{N}]/gu, "");
  return letters.length >= MIN_UTTERANCE_CHARS;
}

function sleep(ms: number) {
  return new Promise((resolve) => setTimeout(resolve, ms));
}

let isSpeaking = false;

The mic side gets a new gate. The previous lesson's data handler pipes every frame into the session; this one drops frames while the assistant is talking so the speaker output never reaches Whisper:

ffmpeg.stdout.on("data", (chunk: Buffer) => {
  if (isSpeaking) return;
  session.write(chunk);
});

The session is the async iterable; transcribeStream({ modelId }) returns a Promise<session>, so you await it before the for await. The loop checks isSpeaking and isMeaningfulTranscript at the top of each iteration to skip frames, and the try/finally around the LLM+TTS block flips isSpeaking back to false even if the LLM throws, so a crashed turn doesn't leave the mic muted:

const session = await transcribeStream({ modelId: asrModelId });
for await (const rawText of session) {
  if (isSpeaking) continue;
  if (!isMeaningfulTranscript(rawText)) continue;
  const userText = rawText.trim();

  history.push({ role: "user", content: userText });

  isSpeaking = true;
  try {
    const llmResult = completion({ modelId: llmModelId, history, stream: true });
    let assistantText = "";
    for await (const token of llmResult.tokenStream) {
      process.stdout.write(token);
      assistantText += token;
    }
    history.push({ role: "assistant", content: assistantText });

    const spoken = assistantText.trim();
    if (spoken.length > 0) {
      const ttsResult = textToSpeech({
        modelId: ttsModelId,
        text: spoken,
        inputType: "text",
        stream: false,
      });
      const samples = await ttsResult.buffer;
      if (samples.length > 0) {
        const wavBuffer = Buffer.concat([
          createWavHeader(samples.length * 2, TTS_SAMPLE_RATE),
          int16ArrayToBuffer(samples),
        ]);
        playAudio(wavBuffer);
      }
      await sleep(POST_PLAYBACK_COOLDOWN_MS);
    }
  } finally {
    isSpeaking = false;
  }
}

Two tuning knobs to revisit if the loop still misbehaves. If VAD commits segments while the user is still talking, raise min_silence_duration_ms. If VAD commits segments out of near-silence, raise threshold to 0.7.

Note: the isSpeaking flag drops the transcripts after Whisper processes them, but the mic is still recording the whole time. Pausing the ffmpeg pipe would let the buffer pile up, so we keep the pipe draining and drop the transcripts in software. The trade-off is a little extra VAD work on audio we'll throw away, in exchange for never stalling on a full buffer.

The SDK installs its own SIGINT / SIGTERM handler that aborts in-flight RPC streams on shutdown. The abort rejects any pending session.write() or unloadModel() call with a WorkerShutdownError. The bare-rpc socket also emits an RPCError with code: 'CHANNEL_CLOSED' on the same teardown path. Both surface as unhandled stream errors. The process.on("uncaughtException", ...) filter ignores the shutdown noise and re-throws anything else:

process.on("uncaughtException", (err) => {
  if (err instanceof WorkerShutdownError) return;
  if (err?.code === "CHANNEL_CLOSED") return;
  throw err;
});

Put it to the test

1. Add vad_params (threshold 0.6, min_speech_duration_ms 300, min_silence_duration_ms 700, max_speech_duration_s 15, speech_pad_ms 200) to the ASR modelConfig.
2. Define POST_PLAYBACK_COOLDOWN_MS = 300, MIN_UTTERANCE_CHARS = 3, isMeaningfulTranscript(text), sleep(ms), and let isSpeaking = false.
3. Gate on ffmpeg and ffplay being on PATH at startup. Define the WAV helpers (createWavHeader, int16ArrayToBuffer, playAudio) and the mic helper (startMicrophone) above main.
4. Open a const session = await transcribeStream({ modelId: asrModelId }) session, then const ffmpeg = startMicrophone(...), then ffmpeg.stdout.on("data", (chunk) => { if (isSpeaking) return; session.write(chunk) }).
5. Iterate with for await (const rawText of session). Skip transcripts while isSpeaking, skip non-meaningful ones, push user turn into history, wrap completion + textToSpeech in a try/finally that flips isSpeaking = true at the start and false at the end. After await ttsResult.buffer, wrap the samples in a WAV header, playAudio(wavBuffer), then await sleep(POST_PLAYBACK_COOLDOWN_MS). Wire the WorkerShutdownError filter to uncaughtException so the SDK's shutdown abort doesn't dump a stack trace.