Diarize speakers with Parakeet Sortformer

When a single audio file has more than one speaker, we want a transcript that says who said what. Sortformer diarizes the audio into speaker-attributed segments, then TDT transcribes each segment into text.

The pipeline runs in two passes:

• Sortformer returns a text block where each line is Speaker N: <start>s - <end>s.
• TDT transcribes each segment and we attach the text to the speaker metadata.

Each Speaker N: <start>s - <end>s line becomes a { speaker, start, end } object. The Sortformer-to-TDT chain starts with the diarization pass. The first pass would look like the following:

const sfModelId = await loadModel({
  modelSrc: PARAKEET_SORTFORMER_4SPK_V2_1_Q8_0,
  modelType: "parakeet-transcription",
});

const diarization = await transcribe({
  modelId: sfModelId,
  audioChunk: audioFilePath,
});
await unloadModel({ modelId: sfModelId });

const segments = diarization
  .split("\n")
  .map((line) => line.match(/Speaker (\d+): ([\d.]+)s - ([\d.]+)s/))
  .filter((m): m is RegExpMatchArray => m !== null)
  .map((m) => ({ speaker: +m[1]!, start: +m[2]!, end: +m[3]! }))
  .sort((a, b) => a.start - b.start);

Second pass: TDT takes over. The slicing reads the source WAV once and writes a new WAV per start/end range; transcribe consumes each slice:

const tdtModelId = await loadModel({
  modelSrc: PARAKEET_TDT_0_6B_V3_Q8_0,
  modelType: "parakeet-transcription",
});

const pcm = readPcm(audioFilePath);
const sliceDir = join(tmpdir(), `qvac-diarize-${Date.now()}`);
mkdirSync(sliceDir, { recursive: true });

const results: { speaker: number; start: number; end: number; text: string }[] = [];
for (let i = 0; i < segments.length; i++) {
  const seg = segments[i]!;
  const slicePath = join(sliceDir, `seg-${i}.wav`);

  if (!writeWavSlice(pcm, seg.start, seg.end, slicePath)) {
    results.push({ ...seg, text: "[No speech detected]" });
    continue;
  }

  const text = await transcribe({
    modelId: tdtModelId,
    audioChunk: slicePath,
  });
  results.push({ ...seg, text: text.trim() || "[No speech detected]" });
}
await unloadModel({ modelId: tdtModelId });

One log line per diarized result, with the speaker label and time range:

for (const r of results) {
  console.log(`Speaker ${r.speaker} (${r.start.toFixed(2)}s - ${r.end.toFixed(2)}s): ${r.text}`);
}

readPcm and writeWavSlice (prefilled in the scaffold) handle the WAV byte-level work: skipping the 44-byte header and rebuilding a 16 kHz / 16-bit / mono header in front of each PCM range. The slicing runs once per diarized segment, so the inference cost is N × TDT_inference rather than one shot on the full file.

Note: Sortformer v2.1 supports up to 4 speakers. For more speakers, split the audio or run the diarization in overlapping windows.

Put it to the test

1. Load PARAKEET_SORTFORMER_4SPK_V2_1_Q8_0 with modelType: "parakeet-transcription", call await transcribe({ modelId, audioChunk: audioFilePath }), then unloadModel. Parse the output lines into { speaker, start, end } objects.
2. Load PARAKEET_TDT_0_6B_V3_Q8_0. For each segment, call writeWavSlice(pcm, seg.start, seg.end, slicePath) and then await transcribe({ modelId: tdtModelId, audioChunk: slicePath }). Push the text onto the segment.
3. Console.log one line per result: Speaker N (start - end): text.