# QVAC (/courses/qvac/en)

The QVAC track teaches you to build with the local-first, peer-to-peer AI SDK from Tether.

The lessons mirror the upstream QVAC SDK examples one-to-one. You write the code, hit Check Answer, and the runner animates the same expected output the SDK would produce.

[Start Lesson 1 →](/courses/qvac/en/getting-started/load-model)

### What you'll cover

* **Loading a model** with `loadModel` and the model constants in the SDK.
* **Running a completion** with `completion()` over a `history` array.
* **Streaming typed events**: content deltas, thinking deltas, completion-done.
* **Reading `stopReason`** to tell natural EOS apart from a token-budget hit.
* **Unloading the model** with `unloadModel` to free device resources.
* **Watching the download** with `onProgress` callbacks.


# Batch decode a neural signal file (/courses/qvac/en/bci/bci-filesystem)

This lesson looks at a much more unusual capability, the brain-computer interface.

BCI models interpret neural signals into something the rest of the SDK can consume. The shape is the same as every other capability: load a model, hand it input, read back the structured result. The inputs and outputs are unusual, but the workflow isn't.

The model that drives this chapter is `BCI_WINDOWED`. It bundles a Whisper-style decoder with a brain-computer-interface projection layer that turns a raw neural-signal `.bin` file into the same audio-token space Whisper expects. The output is the same kind of timed transcript you'd get from a microphone recording.

The BCI model has TWO configs in one `modelConfig`: `whisperConfig` for the decoder, `bciConfig` for the neural data. The two-config block would look like the following:

```ts
const modelId = await loadModel({
  modelSrc: BCI_WINDOWED,
  modelConfig: {
    whisperConfig: {
      language: "en",
      n_threads: 4,
      temperature: 0.0,
    },
    bciConfig: {
      day_idx: 1,
    },
  },
});
```

`bciTranscribe` returns an array of segments, each with timestamp, id, `append` flag, and decoded text. You would call it like so:

```ts
const segments = await bciTranscribe({
  modelId,
  neuralData: neuralFilePath,
  metadata: true,
});
```

Each segment carries a timestamp and a metadata block. We iterate and log the text, the start/end in seconds, and the confidence:

```ts
for (const segment of segments) {
  const start = (segment.startMs / 1000).toFixed(2);
  const end = (segment.endMs / 1000).toFixed(2);
  console.log(
    `  [${start}s → ${end}s] (id=${segment.id}, append=${segment.append}) ${segment.text}`,
  );
}
```

The `bciConfig.day_idx` field picks which day-specific projection matrices the model uses. Set it to match the recording session your neural file came from. Day 1 is the example default.

The `append` field on each segment tells you whether the new text continues the previous segment or starts a new one. A live UI uses it to decide between overwriting the last caption and appending.

> Note: the Whisper half of the BCI pipeline takes the same `whisperConfig` knobs the standalone Whisper model does (language, n\_threads, temperature). For batch decode, `n_threads: 4` and `temperature: 0.0` are sensible defaults.

## Put it to the test

1. Call `loadModel` with `modelSrc: BCI_WINDOWED`, a `whisperConfig` block (`language: "en"`, `n_threads: 4`, `temperature: 0.0`), and `bciConfig: { day_idx: 1 }`.
2. Call `await bciTranscribe({ modelId, neuralData: neuralFilePath, metadata: true })`.
3. Iterate the segments with a `for` loop and console.log each with its `[start → end]` timestamp and `id` and `append` fields.


# Stream-transcribe a sliding window over a neural signal (/courses/qvac/en/bci/bci-streaming)

The previous lesson decoded a neural file in one shot. This one feeds it chunk by chunk to simulate a live stream.

`bciTranscribeStream` opens a duplex session. Writing bytes into `session.write(chunk)` feeds the sliding window, and iterating the session reads the decoded text as the window advances. The session is fully streaming on both sides: writes don't have to wait for the previous decode to finish, and the reader doesn't have to wait for a full file.

Loading the BCI model is the first step. The `whisperConfig` and `bciConfig` tell the engine which day of neural data we're transcribing. The load is the same shape as the previous lesson:

```ts
const modelId = await loadModel({
  modelSrc: BCI_WINDOWED,
  modelConfig: {
    whisperConfig: { language: "en", n_threads: 4, temperature: 0.0 },
    bciConfig: { day_idx: 1 },
  },
});
```

For this, `emit: "delta"` is the flag that switches the session from "emit the full transcript on every read" to "emit just the new text since the last read". The session open looks like this:

```ts
const session = await bciTranscribeStream({ modelId, emit: "delta" });
```

Sequential read-then-write would deadlock, so the consume task runs as an IIFE in parallel with the write loop. Writes run in main flow, with `await consume` after `session.end()`. The 64KB-chunk pipeline looks like:

```ts
const consume = (async () => {
  for await (const text of session) {
    process.stdout.write(text);
  }
})();

const data = readFileSync(neuralFilePath);
for (let offset = 0; offset < data.length; offset += CHUNK_SIZE) {
  const chunk = data.subarray(offset, offset + CHUNK_SIZE);
  session.write(chunk);
  await new Promise((resolve) => setTimeout(resolve, 10));
}
session.end();
await consume;

await unloadModel({ modelId });
```

`emit: "delta"` is the mode that streams the running transcript (each iteration is the new text since the last read). The alternative mode emits the full transcript on every read.

> Note: a 64KB chunk is a reasonable starting size for the example, but a real device driver usually has a buffer of its own. If your source is producing frames continuously, the write loop forwards each frame as it comes in.

## Put it to the test

1. Call `loadModel` with `BCI_WINDOWED` and the same `whisperConfig` + `bciConfig` as the previous lesson.
2. Open `bciTranscribeStream({ modelId, emit: "delta" })` and start the consumer in the background.
3. Read the file with `readFileSync`, write `CHUNK_SIZE` chunks to `session.write(chunk)` in a loop with a 10ms sleep, call `session.end()`, `await consume`, then `unloadModel`.


# BCI (/courses/qvac/en/bci)

BCI models are a much more unusual capability, the brain-computer interface.

BCI models take a neural signal buffer and produce a text transcription. The output plugs into the same completion / event APIs the rest of the SDK uses. The shape is the same: load a model, hand it input, read back the structured result. The inputs and outputs are unusual, but the workflow isn't.

[Start Lesson 1 →](/courses/qvac/en/bci/bci-filesystem)

### All lessons in this chapter

1. [Batch decode a neural signal file](/courses/qvac/en/bci/bci-filesystem)
2. [Stream-transcribe a sliding window over a neural signal](/courses/qvac/en/bci/bci-streaming)


# Connect a delegated-inference consumer (/courses/qvac/en/delegated-inference/delegated-consumer)

The previous lesson booted a provider. This one consumes it.

The runner captures the provider's public key from the previous run's output and passes it to this snippet as `process.argv[2]`. You don't have to copy anything. If you want to connect to a different provider (a peer you don't own, a test instance on another machine), type the key into the **Provider public key** field above the editor. That value takes precedence over the captured one and persists across runs, so you only set it once.

The consumer side is the same `loadModel()` / `completion()` interface as every other lesson. The only new field is `delegate`, which routes the inference to a peer instead of running it locally.

The `loadModel()` call takes the same `modelSrc` and `modelType` parameters. It also takes `delegate`, which carries the routing fields (`providerPublicKey`, `timeout`, `fallbackToLocal`):

```ts
const modelId = await loadModel({
  modelSrc: LLAMA_3_2_1B_INST_Q4_0,
  delegate: {
    providerPublicKey,
    timeout: 60_000,
    fallbackToLocal: true,
  },
});
```

The `completion()` call uses the same `modelId`, `history`, and `stream: true` shape as the text-generation lessons. The `modelId` here is the delegate's handle, so the call routes through the peer over the DHT:

```ts
const response = completion({
  modelId,
  history: [{ role: "user", content: "Hello!" }],
  stream: true,
});
```

The loop iterates `response.tokenStream` and writes each token to stdout. The `response.stats` Promise resolves once the stream ends, so logging it after the loop gives the per-call metrics:

```ts
for await (const token of response.tokenStream) {
  process.stdout.write(token);
}
console.log("\n▸ Stats:", await response.stats);
```

The `delegate` block takes the provider's public key and a generous timeout. The first call on a cold DHT needs 15 to 45 seconds: bootstrapping hyperdht, looking up the provider's key, opening the connection. The SDK gives that headroom via `timeout: 60_000`. Once the DHT is warm, subsequent connections in the same process are sub-second.

`fallbackToLocal: true` is the safety net. If the provider is unreachable (restarting, offline, behind a firewall the consumer can't traverse), the consumer falls back to running the model on its own hardware. The call still succeeds, though with a different latency profile.

> Note: the consumer's `modelSrc` has to match the provider's loaded model. If the provider has Llama 3.2 1B loaded, the consumer has to ask for the same constant, or the provider's router will reject the request with a model-mismatch error.

## Put it to the test

1. Call `loadModel` with `modelSrc: LLAMA_3_2_1B_INST_Q4_0` and a `delegate: { providerPublicKey, timeout: 60_000, fallbackToLocal: true }` block.
2. Call `completion({ modelId, history: [{ role: "user", content: "Hello!" }], stream: true })`.
3. Iterate `response.tokenStream` to stdout, then console.log `response.stats` when the stream ends.


# Run a delegated-inference provider (/courses/qvac/en/delegated-inference/delegated-provider)

Now that we've seen how models download over P2P, we're going to look at the other side of the same network.

Delegated inference sends a completion request to a peer and reads back the response, instead of running the model locally. Useful when the requester's hardware is too small for the model, or when the data needs to stay on the closer-to-source peer.

The provider side is a single long-running process. It advertises itself on the Hyperswarm DHT under a public key, and any consumer that knows that key can route completion calls through it.

The seed is optional. Without it, Hyperswarm uses a fresh random key per run. Consider the following seed setup:

```ts
const seed = process.argv[2];
const allowedConsumerPublicKey = process.argv[3];

if (seed) {
  process.env["QVAC_HYPERSWARM_SEED"] = seed;
}
```

Calling `startQVACProvider()` is what boots the local provider. If a consumer key was passed, we lock the firewall to that one consumer like so:

```ts
const response = await startQVACProvider({
  firewall: allowedConsumerPublicKey
    ? {
        mode: "allow" as const,
        publicKeys: [allowedConsumerPublicKey],
      }
    : undefined,
});
console.log(`▸ Provider Public Key: ${response.publicKey}`);
console.log("");
console.log("▸ Consumer command:");
console.log(`   node consumer.ts ${response.publicKey}`);
```

A seed makes the provider's identity deterministic: the same seed always boots the same public key, so a consumer configured with that key can reconnect across provider restarts. A random seed (no argument) generates a fresh identity each run.

A consumer public key passed as the second argument locks the provider down. The firewall is allowlist-only; consumers not on the list are rejected at the network layer. Useful for staging, demos, and any setup where the provider is reachable from the public DHT.

The runner watches the `▸ Provider Public Key: ...` line and captures the value into the state store. The next lesson reads it back as `process.argv[2]`, so you don't have to copy the key. Stop the provider with Ctrl+C when you're done, then head to the next lesson.

> Note: `startQVACProvider()` doesn't return until the provider is listening on the DHT. Once it returns, consumers can connect. The provider process stays alive until you Ctrl+C.

## Put it to the test

1. Read the optional seed and consumer public key from `process.argv`. If a seed is present, set `process.env["QVAC_HYPERSWARM_SEED"]` to it before the provider call.
2. Call `startQVACProvider({ firewall: { mode: "allow", publicKeys: [consumerKey] } })` if a consumer key was given, or `startQVACProvider({})` otherwise.
3. Log `response.publicKey` and a copy-pasteable `node consumer.ts <publicKey>` command line.


# Delegated inference (/courses/qvac/en/delegated-inference)

Now that we've seen how models download over P2P, we're going to look at the other side of the same network.

Delegated inference sends a completion request to a peer and reads back the response, instead of running the model locally. Useful when the requester's hardware is too small for the model, or when the data needs to stay on the closer-to-source peer. The two halves of the flow are a long-running provider and a short-lived consumer.

[Start Lesson 1 →](/courses/qvac/en/delegated-inference/delegated-provider)

### All lessons in this chapter

1. [Run a delegated-inference provider](/courses/qvac/en/delegated-inference/delegated-provider)
2. [Connect a delegated-inference consumer](/courses/qvac/en/delegated-inference/delegated-consumer)


# Check if a model is fine-tunable (/courses/qvac/en/fine-tuning/check-eligibility)

We're starting a new chapter on fine-tuning.

A trained LoRA adapter only helps if we can stack it on a base model that supports fine-tuning. The QVAC SDK accepts `Q4_K_M` for inference but rejects that quantization for training. We'd see the error halfway through a multi-hour run.

`getModelInfo({ name })` returns the catalog model's `quantization` string. Two things to know before calling it:

1. The catalog constant (`QWEN3_600M_INST_Q4`) is an object whose `.name` field is the string. Pass `QWEN3_600M_INST_Q4.name`.
2. The SDK returns the quantization in lowercase and drops the `_0` suffix, so the 600M Q4 model comes back as `"q4"` instead of `"Q4_0"`.

Pick the base model first, call `getModelInfo` to confirm, then start the trainer. The check looks like this:

```ts
const modelId = await loadModel({ modelSrc: QWEN3_600M_INST_Q4 });
const info = await getModelInfo({ name: QWEN3_600M_INST_Q4.name });

console.log("Quantization:", info.quantization);

const quantization = info.quantization.toUpperCase().replace(/^Q(\d)$/, "Q$1_0");
const fineTunable = ["F32", "F16", "Q4_0", "Q8_0", "TQ1_0", "TQ2_0"].includes(quantization);

console.log("Fine-tunable:", fineTunable ? "yes" : "no");
```

Swap `QWEN3_600M_INST_Q4` for a `Q4_K_M` constant and the second line flips to `no`. Pick a fine-tunable model before you start training.

> Note: the allowlist covers the quantizations the trainer knows how to update. Other quantizations might work someday but aren't supported in this version of the SDK.

## Put it to the test

1. Call `getModelInfo({ name: QWEN3_600M_INST_Q4.name })` and read `info.quantization` into a local variable.
2. Normalize the string (uppercase + restore `_0` suffix) and check it against the allowlist `["F32", "F16", "Q4_0", "Q8_0", "TQ1_0", "TQ2_0"]`. Log the verdict.


# Fine-tuning (/courses/qvac/en/fine-tuning)

Nice, the whole RAG flow works.

Now that we can search a workspace, we're going to train our own adapter.

Fine-tuning takes a base model and produces a small LoRA (low-rank adaptation) file alongside it. The adapter is a small file with the trained weight deltas; the base model stays untouched. Loading the base model with `modelConfig.lora` pointing at the adapter file routes completions through the trained behavior.

[Start Lesson 1 →](/courses/qvac/en/fine-tuning/check-eligibility)

### All lessons in this chapter

1. [Check if a model is fine-tunable](/courses/qvac/en/fine-tuning/check-eligibility)
2. [Run a fine-tune](/courses/qvac/en/fine-tuning/run-finetune)
3. [Pause, resume, and cancel a fine-tune](/courses/qvac/en/fine-tuning/pause-resume-cancel)


# Pause, resume, and cancel a fine-tune (/courses/qvac/en/fine-tuning/pause-resume-cancel)

Now that we've started a fine-tune, we're going to see how to control it.

A fine-tune takes minutes to hours. We don't want to wait around to find out the loss is exploding. The QVAC SDK exposes a small operation surface on `finetune()` itself:

* `pause` saves the current state, stops the trainer, and resolves a promise we can await
* `resume` picks up from the latest checkpoint, the same call shape as the original `finetune()`, just with `operation: "resume"`
* `cancel` is the hard kill: it frees GPU memory immediately and resolves with `{ status: "CANCELLED" }`.

The worker can only run one fine-tune at a time, so each control call has to arrive while the prior run is alive (the pause) and can only fire after it has fully ended (the resume). `finetune()` returns a `handle` with a `progressStream` we iterate, and a `result` promise that resolves when the run ends. We need both.

The setup mirrors the SDK's `llamacpp-finetune` example. The `finetuneParams` wrapper keeps the model + options together, so the resume spreads it and adds `operation: "resume"`:

```ts
const finetuneParams = { modelId, options: baseOptions };
const handle = finetune(finetuneParams);
```

The progress stream runs in an IIFE so the resume after the awaits sees a worker slot that's free. We fire the pause from a callback inside the loop, then wait for the run and the stream to drain before resuming:

```ts
let pauseRequested = false;
let pauseResultPromise;
const progressTask = (async () => {
  for await (const tick of handle.progressStream) {
    // 1: pause from a callback
  }
})();

const initialResult = await handle.result;
await progressTask;
```

Now the three control calls. Pause, fire from a callback once training is rolling so the trainer sees it before the run ends:

```ts
if (!pauseRequested && tick.global_steps >= 4) {
  pauseRequested = true;
  pauseResultPromise = finetune({ operation: "pause", modelId });
}
```

Resume, same params + `operation: "resume"`, after `await handle.result` and `await progressTask` confirm the worker slot is free:

```ts
if (initialResult.status === "PAUSED") {
  const resumed = finetune({ ...finetuneParams, operation: "resume" });
  await resumed.result;
  console.log("▸ Resumed status: COMPLETED");
}
```

Cancel, same as pause but synchronous, returns the final status:

```ts
const cancelResult = await finetune({ operation: "cancel", modelId });
console.log("▸ Cancelled status:", cancelResult.status);
```

Pause and resume both keep our saved checkpoints under `checkpointSaveDir`. Cancel drops the in-flight run but leaves any completed checkpoints intact, so we don't lose what we've already trained.

> Note: pause and cancel are both fire-and-forget at the SDK level. We `await` them to confirm the operation completed, but the returned promise resolves as soon as the trainer acknowledges the request.

## Put it to the test

1. Inside the for-await loop, call `finetune({ operation: "pause", modelId })` from a callback after a few training steps. Set `pauseRequested = true` first so it only fires once.
2. After `await handle.result` and `await progressTask` confirm the worker slot is free, call `finetune({ ...finetuneParams, operation: "resume" })` and await the new handle. Gate it on `initialResult.status === "PAUSED"` so we only resume a run that paused.
3. After the resume, call `finetune({ operation: "cancel", modelId })` and log `result.status`.


# Run a fine-tune (/courses/qvac/en/fine-tuning/run-finetune)

Now that we've confirmed the base model is fine-tunable, it's time to actually run a training job.

`finetune({ modelId, options })` starts a LoRA training run against a chat dataset. The handle exposes a `progressStream` we can iterate to watch training tick by tick, with a `result` promise that resolves with the final status.

The simplest input format is a HuggingFace chat JSONL: one JSON object per line, each with a `messages` array of `{role, content}` pairs. The trainer handles tokenization internally.

`finetune()` returns a handle with a `progressStream` and a `result` promise. You would call it as follows:

```ts
const handle = finetune({
  modelId,
  options: {
    trainDatasetDir: "./examples/qvac/fine-tuning/input/small_train_HF.jsonl",
    validation: { type: "dataset", path: "./examples/qvac/fine-tuning/input/small_eval_HF.jsonl" },
    numberOfEpochs: 1,
    learningRate: 1e-4,
    loraModules: "attn_q,attn_k,attn_v,attn_o,ffn_gate,ffn_up,ffn_down",
    assistantLossOnly: true,
    outputParametersDir: "../../apps/desktop/output/finetune/",
  },
});
```

The `progressStream` ticks once per training step, each item carrying `global_steps`, `loss`, `accuracy`, `current_epoch`, `total_batches`, and `eta_ms`. `await handle.result` returns the final status (`COMPLETED`, `CANCELLED`, or a failure mode):

```ts
for await (const tick of handle.progressStream) {
  const phase = tick.is_train ? "train" : "val";
  console.log(
    `▸ epoch=${tick.current_epoch + 1} step=${tick.global_steps} ` +
      `batch=${tick.current_batch}/${tick.total_batches} ${phase} ` +
      `loss=${tick.loss?.toFixed(4)} acc=${tick.accuracy?.toFixed(4)} ` +
      `eta=${Math.round(tick.eta_ms / 1000)}s`,
  );
}

const result = await handle.result;
console.log("▸ Result status:", result.status);
```

The data files at `./examples/qvac/fine-tuning/run-finetune/small_train_HF.jsonl` live next to this lesson's code. The output adapters go to `../../apps/desktop/output/finetune/` (the desktop app is the runtime, generated files live there).

## Known SDK quirk

The bare worker that runs the trainer can crash with `SIGABRT` during cleanup after the last batch. Training and the adapter write complete before the crash, so the loss/accuracy you see are real. The answer wraps the `progressStream` loop in `try/catch` and swallows the SDK's `WORKER_CRASHED` error so the user just sees a clean "Adapter written" line. If something fails before any tick lands, the error is rethrown.

> Note: `learningRate: 1e-4` is a reasonable starting point for LoRA on a Qwen3 600M. If you're training a larger model or a smaller one, scale by the parameter count or follow the dataset author's recommendation.

## Put it to the test

1. Call `finetune({ modelId, options: { trainDatasetDir, numberOfEpochs: 1, learningRate: 1e-4, loraModules, assistantLossOnly, outputParametersDir } })` and store the handle.
2. Loop through `handle.progressStream`, logging the epoch/step/batch/phase/loss/accuracy/eta for each tick.
3. Await `handle.result` inside a `try/catch` and log `result.status` on success, or the last tick's loss/accuracy on the cleanup crash.


# Getting started (/courses/qvac/en/getting-started)

Welcome to the Tether Academy! This first chapter gets a model running on your machine, fires a completion at it, and tears the model back down. Each lesson builds on the one before it, so it's worth working through them in order.

By the end of this chapter you'll have watched the model download, run a completion, read the stop reason, and freed the model and its worker. The next chapter covers the richer event stream.

[Start Lesson 1 →](/courses/qvac/en/getting-started/load-model)

### All lessons in this chapter

1. [Load your first model](/courses/qvac/en/getting-started/load-model)
2. [Run a completion](/courses/qvac/en/getting-started/run-completion)
3. [Read the stop reason from a completion](/courses/qvac/en/getting-started/stop-reasons)
4. [Unload the model from memory](/courses/qvac/en/getting-started/unload-model)
5. [Show download progress](/courses/qvac/en/getting-started/show-download-progress)


# Load your first model (/courses/qvac/en/getting-started/load-model)

In this first lesson, we're going to load a model into memory and get back a `modelId` you can reuse. Every other lesson in this chapter builds on top of this one, so let's get it right.

Before any completion can run, the model has to live somewhere the SDK can reach it. `loadModel()` does that for us. We give it a model constant, it downloads the model files if they're not already cached, and hands us back an id we can pass to other calls.

Every QVAC lesson starts with the same load idiom. The canonical `loadModel({ modelSrc })` call looks like this:

```ts
const modelId = await loadModel({
  modelSrc: LLAMA_3_2_1B_INST_Q4_0,
});
console.log("modelId:", modelId);
```

`LLAMA_3_2_1B_INST_Q4_0` is one of the model constants `@qvac/sdk` exports. It's small (about 740 MB) and downloads in a few minutes on a fast connection.

> Note: the `await` is important. `loadModel` returns a Promise, and we have to wait for it to resolve before we can use `modelId`.

In the next lesson, we'll use this `modelId` to run our first completion. For now, let's make sure the load itself works.

## Put it to the test

1. Call `loadModel({ modelSrc: LLAMA_3_2_1B_INST_Q4_0 })` and assign the result to `modelId`.
2. Log `modelId` to stdout.


# Run a completion (/courses/qvac/en/getting-started/run-completion)

Now that we have a `modelId`, it's time to ask the model to say something.

`completion()` takes a `modelId` and a `history` (an array of messages), and gives us back a result we can either await fully or stream token by token. We pass `stream: true` so we can watch the tokens arrive.

The `history` array is a list of `{ role, content }` messages, like a chat log. We're starting with one user message, but later lessons will add assistant and system turns.

A `history` array with one user message would look like this:

```ts
const history = [
  { role: "user", content: "Explain quantum computing in one sentence." },
];
```

Now we wrap that history in a `completion()` call like so:

```ts
const result = completion({ modelId, history, stream: true });
```

Finally, we drain the token stream token-by-token using `process.stdout.write`:

```ts
for await (const token of result.tokenStream) {
  process.stdout.write(token);
}
```

> Note: when the stream finishes, the result is also available as a single string via `await result.text`. We'll use that in later lessons.

## Put it to the test

1. Build a `history` array with one user message asking the model to explain quantum computing in one sentence.
2. Call `completion({ modelId, history, stream: true })` and store the result.
3. Iterate `result.tokenStream` and write each token to stdout.


# Show download progress (/courses/qvac/en/getting-started/show-download-progress)

In the very first lesson we called `loadModel({ modelSrc: ... })` and waited. From the user's point of view the script sat there with no output while a multi-hundred-megabyte file downloaded. `onProgress` exists to make that wait readable.

`onProgress` is a callback we pass alongside `modelSrc`. The SDK calls it repeatedly while the model downloads. Each call hands us `{ percentage, downloaded, total }`. We print every call, throttle to every few percent, or draw a progress bar.

Let's take a closer look at the pattern from the [QVAC quickstart example](https://github.com/tetherto/qvac/blob/main/packages/sdk/examples/quickstart.ts). The callback goes inside the same `loadModel()` options object that already carries `modelSrc`.

`onProgress` runs many times during a download. The full callback wired into `loadModel()` would look like the following:

```ts
const modelId = await loadModel({
  modelSrc: LLAMA_3_2_1B_INST_Q4_0,
  onProgress: (p) => {
    const mb = (n: number) => (n / 1e6).toFixed(1);
    const line = `▸ Downloading ${p.percentage.toFixed(0)}% (${mb(p.downloaded)}/${mb(p.total)} MB)`;
    process.stderr.write(process.stderr.isTTY ? `\r${line}` : `${line}\n`);
    if (p.percentage >= 100) process.stderr.write("\n");
  },
});
console.log("modelId:", modelId);
```

A few details worth pointing at:

* `process.stderr.write` is the right call here. Stdout is reserved for the model's actual response in the next lesson; progress is logging, and stderr is the standard place for it.
* The `process.stderr.isTTY` flag picks the rendering mode. When the terminal is a TTY, the script writes `\r` so each tick overwrites the same line. When output is piped to a file, it writes `\n` so each tick ends up on its own row in the log.
* The final `process.stderr.write("\n")` after `p.percentage >= 100` makes sure the next log starts on a fresh row.

## Put it to the test

1. Add `modelSrc: LLAMA_3_2_1B_INST_Q4_0` and an `onProgress` callback to the `loadModel` call. Inside the callback, write `▸ Downloading X% (Y/Z MB)` to `process.stderr`, choosing between overwriting the line (`isTTY`) and appending a newline (pipe). When `p.percentage >= 100`, write a trailing newline so the next log starts on a fresh row.


# Read the stop reason from a completion (/courses/qvac/en/getting-started/stop-reasons)

Now that we know how to iterate the event stream, let's look at what we get when the stream ends.

Every `CompletionFinal` carries a `stopReason` that explains how the model stopped generating:

* `undefined`, natural end of sequence (EOS). The model finished on its own. This is the common case.
* `"length"`, the `predict` token budget was exhausted. Output is truncated, the model did not reach a natural stopping point.
* `"cancelled"`, the request was cancelled via `cancel({ requestId })`.

Setting `predict: 10` forces the truncated path. You would call it in the following way:

```ts
const result = completion({
  modelId,
  history: [{ role: "user", content: "Say hi in one word." }],
  captureThinking: true,
  generationParams: { predict: 10 },
  stream: true,
});
```

The drain is necessary even when we only care about `final`. Note that without it, the stream backs up and `result.final` never resolves. You would write the drain like so:

```ts
for await (const token of result.tokenStream) process.stdout.write(token);
```

After the drain, we read the aggregate. `result.final` is the canonical surface for it. Note that the `await` is what fetches the aggregated `contentText`, `thinkingText`, `toolCalls`, `stats`, and `stopReason`:

```ts
const final = await result.final;
```

Branching on `stopReason === "length"` is how we surface the truncation. Note that the only "length" value here is the budget-truncation path; `"cancelled"` is a separate branch. Consider the following example:

```ts
if (final.stopReason === "length") {
  console.log("▸ truncated: model hit the token budget");
}
```

> Note: a tight `predict` budget on a short prompt is the easiest way to see the truncation path. In production you usually want a generous budget and rely on EOS, but reading `stopReason` is how you tell the two apart after the fact.

## Put it to the test

1. Use `captureThinking: true` and `generationParams: { predict: 10 }` so the response truncates before it finishes and the model's `thinking` stays out of the output.
2. Iterate `result.tokenStream` to completion.
3. After the loop, await `result.final` and read `final.stopReason`.
4. Branch on `final.stopReason`. For `"length"`, log that the token budget cut off the response.


# Unload the model from memory (/courses/qvac/en/getting-started/unload-model)

Now that we've run our completion, let's free the model and tear down the worker.

`unloadModel` is the symmetric counterpart to `loadModel`. We pass it the `modelId` and it frees the loaded model from device memory. With `autoClose: true`, the underlying worker process is also torn down.

`autoClose: true` shuts down the worker (frees the OS process); `autoClose: false` only frees the weights. We can fully unload the session like so:

```ts
await unloadModel({ modelId, autoClose: true });
console.log("▸ model unloaded");
```

If you skip this call, the model stays in memory until the process exits. For long-running workloads (servers, desktop apps, mobile sessions) unload the moment the work is done. The model takes hundreds of megabytes, and the GPU or CPU stays warm as long as the worker is alive.

> Note: calling `unloadModel()` mid-stream would cut tokens off. Always wait for the events loop to finish before unloading.

## Put it to the test

1. After the completion loop ends, call `unloadModel({ modelId, autoClose: true })`.
2. Log a confirmation line, e.g., `"▸ model unloaded"`.


# Classify an image (/courses/qvac/en/image-classification/classify-image)

We're going to add one more image-based capability, separate from generation.

Image classification takes an image and returns one or more category labels with confidence scores. The SDK includes a small bundled MobileNetV3-Small model that produces three classes: `food`, `report`, `other`. It's useful for routing (which model to invoke), filtering (does this image contain X?), and tagging at scale.

The flow is similar to the other capabilities. The full lifecycle, with the provider started, the model loaded, the function called, and the provider stopped.

Booting the provider is required before any model loads. The classification model is bundled in the addon, so `loadModel` takes no `modelSrc`:

```ts
await startQVACProvider({});
const modelId = await loadModel({ modelType: "ggml-classification" });
```

The result is sorted by score descending, so `predictions[0]` is the top guess. The SDK returns a `confidence` score between 0 and 1, so multiplying by 100 will get you a percentage:

```ts
const image = fs.readFileSync("./examples/image/basic_test.jpg");
const results = await classify({ modelId, image });

for (const { label, confidence } of results) {
  console.log(`  ${label}: ${(confidence * 100).toFixed(1)}%`);
}
```

Freeing the model and shutting the provider back down is the cleanup. Both are explicit so the next caller doesn't see a half-running backend. You would clean up like so:

```ts
await unloadModel({ modelId });
await stopQVACProvider();
```

The notable difference from other lessons: `loadModel` takes no `modelSrc`. The classification model is bundled inside the `@qvac/classification-ggml` addon, not downloaded from the registry. The `modelType: "ggml-classification"` flag tells the SDK which addon to route through.

> Note: the bundled MobileNetV3-Small model is small on purpose. It runs in tens of milliseconds on a single CPU core. If you need a bigger or domain-specific classifier, you'd bring your own GGUF and add it to a custom addon.

## Put it to the test

1. Call `startQVACProvider({})` and `loadModel({ modelType: "ggml-classification" })`.
2. Read the image with `fs.readFileSync`, call `classify({ modelId, image })`, and log each label with confidence.
3. Call `unloadModel({ modelId })` and `stopQVACProvider()`.


# Image classification (/courses/qvac/en/image-classification)

Image classification is one more image-based capability, separate from generation.

Image classification takes an image and returns one or more category labels with confidence scores. It's useful for routing (which model to invoke), filtering (does this image contain X?), and tagging at scale.

[Start Lesson 1 →](/courses/qvac/en/image-classification/classify-image)

### All lessons in this chapter

1. [Classify an image](/courses/qvac/en/image-classification/classify-image)


# Upscale a generated image in the same call (/courses/qvac/en/image-generation/esrgan-postprocess)

Diffusion can produce a sharp image at the resolution you ask for, but the absolute ceiling is bounded by VRAM. To go past it, you pair the diffusion model with an ESRGAN upscaler at load time. The diffusion call then runs the model and the upscaler in sequence, returning the upscaled PNG in the same call.

`modelConfig.upscaler` is the wiring. It's an object with a `type`, a `model_src`, and an optional `tile_size`. Once it's set, every `diffusion()` call in that session can opt in by passing an `upscale` option.

The `upscale` option takes one of three forms:

* `upscale: true` runs a single pass at the model's native scale factor. For an `x4` ESRGAN, that's `4x` linear in each dimension.
* `upscale: { repeats: 1 }` is the same thing, spelled out.
* `upscale: { repeats: N }` compounds the scale factor across N sequential passes. `repeats: 2` on an `x4` model is `16x` linear.

The upscaler block has three fields: `type`, `model_src`, `tile_size`. Wiring it on `loadModel` once is the canonical upscaler setup:

```ts
upscaler: {
  type: "esrgan",
  model_src: REALESRGAN_X4PLUS_ANIME_6B,
  tile_size: 128,
},
```

A single native-scale pass is `upscale: true`. For an `x4` ESRGAN, that's `4x` linear in each dimension. You would call it like so:

```ts
const x4 = diffusion({ ...baseParams, upscale: true });
const x4Buffers = await x4.outputs;
```

A compounded 2-pass upscale is `upscale: { repeats: 2 }`. Each pass is internal, so only the final 16x result comes back. Let's look at how to make that call:

```ts
const x16 = diffusion({ ...baseParams, upscale: { repeats: 2 } });
const x16Buffers = await x16.outputs;
```

The source `width` and `height` are intentionally small. Each ESRGAN pass multiplies the dimensions, so a `128x128` input at `repeats: 2` ends up at `2048x2048`. The model doesn't need to do the heavy lifting of a high-resolution diffusion pass; the upscaler does the enlargement in a separate, cheaper pass.

> Note: the upscaler adds time and memory on top of the diffusion call. For a `512x512` source with no upscaling, you only pay the diffusion cost. With `repeats: 2`, you pay diffusion plus two ESRGAN passes. Watch your VRAM on a 24GB card at `repeats: 2` for a `1024x1024` source.

## Put it to the test

1. Add `upscaler: { type: "esrgan", model_src: REALESRGAN_X4PLUS_ANIME_6B, tile_size: 128 }` to `modelConfig`.
2. Call `diffusion({ ...baseParams, upscale: true })` and `await result.outputs`. The result is written to `fox_x4.png`.
3. Call `diffusion({ ...baseParams, upscale: { repeats: 2 } })` and `await result.outputs`. The result is written to `fox_x16.png`. The final line logs `"Generated 1 image"`.


# Generate image with FLUX.2-klein split layout (/courses/qvac/en/image-generation/flux2-split-layout)

We've used a single model file so far. FLUX.2-klein needs three: a diffusion model, a text encoder, and a VAE.

The diffusion model is the inference engine. It iteratively refines a noise field step by step until the result matches your prompt. The text encoder produces the tokens the engine reads. The VAE encodes the working state in latent space and decodes the final image back to pixels.

The three files are independent. The SDK downloads each one on its own schedule, then wires them together at load time.

FLUX.2-klein split-layout is three files in one `loadModel` call: diffusion model, LLM encoder, VAE. The pipeline: LLM → tokens → diffusion → latent → VAE → pixels. The combined load would look like the following:

```ts
const modelId = await loadModel({
  modelSrc: FLUX_2_KLEIN_4B_Q4_0,
  modelType: "sdcpp-generation",
  modelConfig: {
    llmModelSrc: QWEN3_4B_Q4_K_M,
    vaeModelSrc: FLUX_2_KLEIN_4B_VAE,
  },
});
```

From the caller's perspective, split-layout vs single-file is invisible. The `outputs` array is the same shape as for a single-file model: a `Promise<Uint8Array[]>` of PNG bytes:

```ts
const result = diffusion({ modelId, prompt: "a quiet harbor at dawn" });
const outputs = await result.outputs;
```

`outputs[0]` is the first PNG, `fs.writeFileSync` writes raw bytes (no header needed), and an empty array means the call failed silently. Here's how an example would look like:

```ts
const first = outputs[0];
if (first) fs.writeFileSync("../../apps/desktop/output/image-gen/harbor.png", first);
console.log(`Generated ${outputs.length} image`);
```

> Note: the SDK only downloads the files that aren't already on disk. The second run with the same constants skips the download and goes straight to inference.

## Put it to the test

1. Call `loadModel` with `modelType: "sdcpp-generation"`, `modelSrc: FLUX_2_KLEIN_4B_Q4_0`, and a `modelConfig` block holding the LLM encoder and VAE.
2. Call `diffusion({ modelId, prompt })` and `await result.outputs`.
3. Write `outputs[0]` to a PNG and log the image count.


# Generate image with img2img (/courses/qvac/en/image-generation/img2img)

Now that we can do txt2img, we're going to learn img2img.

With txt2img, the model by default starts from noise. With img2img, the model starts from an image we supply and iteratively refines it until the result matches our prompt.

`init_image` is a `Uint8Array` of PNG or JPEG bytes. We read it off disk with `fs.readFileSync`. The `strength` number tells the model how much room it has. At `0` it keeps the source image untouched, and at `1` it ignores it and behaves like txt2img.

Common uses include rough sketch to colored illustration, screenshot to wireframe turned into a design mock, or an existing photo restyled.

Reading the source image into a `Uint8Array` is what hands the bytes to `diffusion()`. Let's look at how to handle that:

```ts
const initImage = fs.readFileSync("./examples/qvac/image-generation/input/sketch.png");
```

Calling `diffusion()` with `init_image` set to the source bytes is the img2img path. The call we'd run looks like:

```ts
const result = diffusion({
  modelId,
  prompt: "an oil painting of a fox in a snowy forest",
  init_image: initImage,
  strength: 0.6,
  width: 512,
  height: 512,
  steps: 25,
});
```

Once `await result.outputs` resolves, write `outputs[0]` and log the count:

```ts
const outputs = await result.outputs;
const first = outputs[0];
if (!first) throw new Error("No image returned from diffusion");
fs.writeFileSync("../../apps/desktop/output/image-gen/fox-painting.png", first);
console.log(`Generated ${outputs.length} image`);
```

We tweak `strength` until the balance between "preserves the source" and "rewrites everything" feels right.

> Note: `init_image` dimensions are the lower bound. The output resolution is governed by `width` and `height`, not the source size.

## Put it to the test

1. Read an input image with `fs.readFileSync` into a `Uint8Array`.
2. Call `diffusion({ modelId, prompt, init_image, strength: 0.6, width, height, steps })` and `await result.outputs`.
3. Write the first PNG to disk and console.log the count.


# Image generation (/courses/qvac/en/image-generation)

Now that a model can take images as input, we're going to add the reverse direction: a model that produces images from text.

Diffusion is a different kind of model: instead of running a language model to predict tokens, it starts from noise and iteratively refines the image until the result matches the prompt. The shape is the same as `completion()`, but the result is a `Uint8Array` of PNG bytes instead of a token stream.

[Start Lesson 1 →](/courses/qvac/en/image-generation/txt2img)

### All lessons in this chapter

1. [Generate image with txt2img](/courses/qvac/en/image-generation/txt2img)
2. [Set image width, height, and steps](/courses/qvac/en/image-generation/size-and-steps)
3. [Track diffusion progress](/courses/qvac/en/image-generation/progress)
4. [Generate image with img2img](/courses/qvac/en/image-generation/img2img)
5. [Generate image with FLUX.2-klein split layout](/courses/qvac/en/image-generation/flux2-split-layout)
6. [Generate image with Stable Diffusion](/courses/qvac/en/image-generation/stable-diffusion)
7. [Upscale a generated image in the same call](/courses/qvac/en/image-generation/esrgan-postprocess)


# Track diffusion progress (/courses/qvac/en/image-generation/progress)

Now that we're running diffusion calls that take real time, let's see how to show progress.

`diffusion()` runs for tens of seconds. Without feedback, the user thinks it crashed. The same `result` we used in the previous lesson has a second property: `progressStream`.

`progressStream` ticks once per denoising step. Each item is `{ step, totalSteps }`. The total tells us the upper bound. The step tells us where the model is right now.

We loop the stream in parallel with the final `await result.outputs`, since they don't block each other.

The per-step progress stream carries `{ step, totalSteps }` once per denoising tick, same `for await` shape as the `tokenStream` drain in the text lessons like so:

```ts
if (result.progressStream) {
  for await (const progress of result.progressStream) {
    console.log(`${progress.step}/${progress.totalSteps}`);
  }
}
```

The progress stream and the output promise don't block each other, so you can run them in parallel:

```ts
const outputs = await result.outputs;
const first = outputs[0];
if (first) fs.writeFileSync("../../apps/desktop/output/image-gen/skyline.png", first);
console.log(`Generated ${outputs.length} image`);
```

We wire this stream into our app's progress bar and the user sees "5/20, 6/20, ..." tick up while the model works.

> Note: `progressStream` is optional in the type system. The check `if (result.progressStream)` is defensive; older model versions didn't expose it. New models all do.

## Put it to the test

1. After calling `diffusion`, iterate `result.progressStream` with `for await`.
2. Log each tick as `${progress.step}/${progress.totalSteps}`, then `await result.outputs` and write the PNG.


# Set image width, height, and steps (/courses/qvac/en/image-generation/size-and-steps)

The previous lesson made our first image. This one tunes the size and the step count.

`diffusion()` takes a handful of options besides `prompt`. The three you'll reach for first are `width`, `height`, and `steps`. Width and height control the output resolution in pixels. `steps` controls how many denoising iterations the model runs. More steps means more refine work, and eventually the image stops changing much.

A 512×512 image at 20 steps runs in a few seconds on a GPU. Doubling resolution roughly quadruples the work. The `seed` option lets us pin the random noise, so we can pass any integer and get the same image twice.

Four knobs on the same `diffusion()` call: `width` / `height` (multiples of 16) set resolution, `steps` is denoising iterations, `seed` pins the noise. The four-knob call would look like:

```ts
const result = diffusion({
  modelId,
  prompt: "a watercolor cat on a sunny windowsill",
  width: 512,
  height: 512,
  steps: 20,
  seed: 42,
});
```

Same seed and same prompt, two calls produce byte-identical output like so:

```ts
const outputs = await result.outputs;
const first = outputs[0];
if (first) fs.writeFileSync("../../apps/desktop/output/image-gen/cat-watercolor.png", first);
console.log(`Generated ${outputs.length} image`);
```

Run it twice with the same seed and we get byte-identical output. Change the prompt and we get a different image from the same starting noise.

> Note: the seed is per-call, not per-model. Two `diffusion()` calls with the same `seed` and the same `prompt` produce identical output; changing either breaks the reproducibility.

## Put it to the test

1. Call `diffusion({ modelId, prompt, width: 512, height: 512, steps: 20, seed: 42 })`.
2. `await result.outputs` and write `outputs[0]` to disk.


# Generate image with Stable Diffusion (/courses/qvac/en/image-generation/stable-diffusion)

FLUX.2 is one model family. Stable Diffusion is another, and the older of the two. SD 1.x and SD 2.x are available as a single all-in-one GGUF. There's no LLM encoder and no VAE to download alongside it.

That single-file layout is the trade. SD 2.1 in this format is much smaller on disk than FLUX.2-klein, so it loads faster on a fresh device and fits in less VRAM. The generation quality is lower than FLUX.2-klein at the same step count, but for short prompts and quick iteration it's the path of least resistance.

Stable Diffusion supports two sampling targets: `epsilon` (default) and `v` (the velocity, the change in noise). At high guidance, `v` is the cleaner choice with fewer artifacts. The `modelConfig` block setting `v` would look like so:

```ts
const modelId = await loadModel({
  modelSrc: SD_V2_1_1B_Q8_0,
  modelType: "sdcpp-generation",
  modelConfig: { prediction: "v" },
});
```

The generation call is the same `diffusion({ modelId, prompt })` as the FLUX.2 lessons:

```ts
const result = diffusion({
  modelId,
  prompt: "a photo of a cat sitting on a windowsill",
});
```

Once the awaited PNG is in hand, write the first one and log the count:

```ts
const outputs = await result.outputs;
const first = outputs[0];
if (!first) throw new Error("No image returned from diffusion");
fs.writeFileSync("../../apps/desktop/output/image-gen/cat.png", first);
console.log(`Generated ${outputs.length} image`);
```

SD 2.1 was trained for v-prediction; epsilon would produce a noisier result on this model.

> Note: SD 2.1 doesn't support the in-context `init_image` path that FLUX.2 uses. For img2img with SD, set `strength` instead. The next chapter covers that.

## Put it to the test

1. Call `loadModel` with `modelType: "sdcpp-generation"`, `modelSrc: SD_V2_1_1B_Q8_0`, and `modelConfig: { prediction: "v" }`.
2. Call `diffusion({ modelId, prompt: "..." })` and `await result.outputs`.
3. Write `outputs[0]` to a PNG file and console.log the image count.


# Generate image with txt2img (/courses/qvac/en/image-generation/txt2img)

We're starting a new chapter on image generation, and we're going to make our first PNG.

Diffusion models generate images by starting from noise and step-by-step turning it into what your prompt describes. The QVAC SDK wraps one inference pass into a single call: `diffusion({ modelId, prompt })`.

The model needs to be loaded with the right `modelType`. `sdcpp-generation` is the official constant for the Diffusion engine. Anything else, and `diffusion()` either rejects the call or runs against the wrong backend.

Result is a `DiffusionResult`. `outputs` is a `Promise<Uint8Array[]>`. Each array entry is one PNG. The default is a single image unless we ask for more.

Standard FLUX.2 setup: a small diffusion model, a prompt encoder, a VAE. The whole pipeline (LLM → tokens → diffusion → latent → VAE → pixels) wires at load time. The combined `loadModel` would look like below:

```ts
const modelId = await loadModel({
  modelSrc: FLUX_2_KLEIN_4B_Q4_0,
  modelType: "sdcpp-generation",
  modelConfig: {
    llmModelSrc: QWEN3_4B_Q4_K_M,
    vaeModelSrc: FLUX_2_KLEIN_4B_VAE,
  },
});
```

Default is a single image. Same `diffusion()` shape as every other model in this chapter:

```ts
const result = diffusion({ modelId, prompt: "a cat sitting on a sofa" });
```

`outputs[0]` is the first image; `fs.writeFileSync` writes raw PNG bytes with no header needed. Note that the write needs the `if (firstImage)` guard to avoid a crash if the model fails to return an image:

```ts
const outputs = await result.outputs;
const firstImage = outputs[0];
if (!firstImage) throw new Error("No image returned from diffusion");
fs.writeFileSync("../../apps/desktop/output/image-gen/cat.png", firstImage);

console.log(`Generated ${outputs.length} image`);
```

Open `cat.png` after running. That's the model's answer to your prompt. The next lesson tunes the size and the step count.

> Note: diffusion models are memory-heavy. A 4B-parameter Flux model in FP16 needs roughly 8 GB of VRAM. Make sure your machine has headroom before loading.

## Put it to the test

1. Call `loadModel` with `modelType: "sdcpp-generation"` and the split-layout `modelConfig`.
2. Call `diffusion({ modelId, prompt })` and `await result.outputs`.
3. Write `outputs[0]` to `cat.png` and console.log the count.


# Compare multiple images in a completion (/courses/qvac/en/multimodal/compare-images)

The previous lesson attached a single image. This one attaches two and asks the model to compare them.

The model can hold more than one image in context. The `attachments` array is the only knob: we add more objects, point each at a different file on disk, and the projection model handles the rest.

We use it for comparisons, "spot the difference", describing a sequence of frames, or any task where the answer is "across" multiple images rather than "about" one.

Each image in `attachments` becomes one entry in the array. The order in the array is the order the SDK passes the images to the model. We keep that in mind if our prompt references "the first image" or "the second image".

Two image attachments on the same user message, SDK passes them to the projection in `attachments` order (index 0 is the first image):

```ts
const history = [
  {
    role: "user",
    content: "Compare the two newspaper articles. Which one is older?",
    attachments: [
      { path: "./examples/qvac/multimodal/input/article-a.jpg" },
      { path: "./examples/qvac/multimodal/input/article-b.jpg" },
    ],
  },
];
```

Two images in the history is what makes the call 'compare'. You would call it like so:

```ts
const result = completion({ modelId: multimodalId, history, stream: true });
```

Once the call returns, drain the tokens same as text-only:

```ts
for await (const token of result.tokenStream) {
  process.stdout.write(token);
}
process.stdout.write("\n");
```

The history grows by one entry per extra image. `completion()` and `result.tokenStream` work the same as a text-only call; only the history entries carry images.

> Note: multimodal assistants use `modelConfig.projectionModelSrc`, not `modelConfig.lora`. The two are unrelated options.

## Put it to the test

1. Build a `history` array with one user message that has two image attachments in the `attachments` array and asks the model to compare them.
2. Call `completion({ modelId: multimodalId, history, stream: true })`.
3. Iterate `result.tokenStream` and write each token to stdout. Confirm the model compares the two articles.


# Multimodal (/courses/qvac/en/multimodal)

Now that we've done text and fine-tuning, we're going to add images to the mix.

A multimodal model takes a prompt with both images and text and produces a text completion. We attach images to history messages, and the resulting completion text can reference them. The setup needs an LLM with a small projection model that translates image pixels into the same vector space the LLM uses for token embeddings.

[Start Lesson 1 →](/courses/qvac/en/multimodal/load-pair)

### All lessons in this chapter

1. [Load a multimodal model and projection](/courses/qvac/en/multimodal/load-pair)
2. [Send a single image to a completion](/courses/qvac/en/multimodal/send-image)
3. [Compare multiple images in a completion](/courses/qvac/en/multimodal/compare-images)


# Load a multimodal model and projection (/courses/qvac/en/multimodal/load-pair)

We're starting a new chapter on multimodal models, and we're going to load a model that takes images alongside text.

Text-only LLMs load from one file. Multimodal LLMs need two: the language model itself, and a small "projection" model that turns an image into the same kind of vector space the language model operates in.

`loadModel()` accepts both at once. The main file goes in `modelSrc`. The projector goes in `modelConfig.projectionModelSrc`.

Multimodal = LLM in `modelSrc` + mmproj projector in `projectionModelSrc`. Both files must be from the same model family. You would load them like so:

```ts
const multimodalId = await loadModel({
  modelSrc: SMOLVLM2_500M_MULTIMODAL_Q8_0,
  modelConfig: {
    projectionModelSrc: MMPROJ_SMOLVLM2_500M_MULTIMODAL_Q8_0,
  },
});
console.log("multimodalId:", multimodalId);
```

`multimodalId` looks the same as every other `modelId`. The difference is hidden in how `completion()` will treat `attachments` on history messages, which we'll see in the next two lessons.

> Note: the two constants have to come from the same model family. `SMOLVLM2` pairs with `MMPROJ_SMOLVLM2`. Mixing an LLM constant with the wrong projector (or a non-multimodal LLM with a projector) produces nonsense output.

## Put it to the test

1. Call `loadModel` with `modelSrc` set to the LLM constant and `modelConfig.projectionModelSrc` set to the matching mmproj.
2. Store the result in a variable called `multimodalId` and log it.


# Send a single image to a completion (/courses/qvac/en/multimodal/send-image)

Now that we have a multimodal model in memory, it's time to give it an image.

The multimodal model takes a prompt that includes an image as well as text. We tell it what to look at by attaching an image file to a user message. `attachments[].path` is the only thing it needs, a string path to a real file on disk.

We don't swap the chat shape. We add `attachments` to the existing `history` message we already use for text completions.

`attachments[].path` is the multimodal engine's API surface. The SDK reads the bytes; the projector runs before the message reaches the LLM. Adding an attachment to a user message is one extra field on the existing history entry:

```ts
attachments: [{ path: "./examples/qvac/multimodal/input/cat.png" }],
```

Calling `completion()` with the same shape as a text call is what kicks off the multimodal run. The only new field is the `attachments` array on the user message:

```ts
const result = completion({ modelId: multimodalId, history, stream: true });
```

Once the call returns, drain the tokens same as text-only. Note that the drain uses `process.stdout.write("\n")` at the end to flush a newline. The drain looks like:

```ts
for await (const token of result.tokenStream) {
  process.stdout.write(token);
}
process.stdout.write("\n");
```

The path is relative to where we run the script from, or absolute if we prefer. We drop any local `.jpg` or `.png` at the path we pass in.

> Note: the multimodal model needs the projector loaded (from the previous lesson) to interpret the image. If `projectionModelSrc` was missing, the model would silently ignore the attachment and respond as if no image was attached.

## Put it to the test

1. Add `attachments: [{ path: "./examples/qvac/multimodal/input/cat.png" }]` to the user message in `history`.
2. Call `completion({ modelId: multimodalId, history, stream: true })`.
3. Iterate `result.tokenStream` and write each token to stdout.


# OCR (/courses/qvac/en/ocr)

OCR is one more capability: extracting text that's already printed in an image.

OCR (optical character recognition) takes an image and returns text blocks with bounding boxes and confidence scores. The `OCR_LATIN` model handles any Latin-script language.

[Start Lesson 1 →](/courses/qvac/en/ocr/ocr-image)

### All lessons in this chapter

1. [Extract text from an image](/courses/qvac/en/ocr/ocr-image)


# Extract text from an image (/courses/qvac/en/ocr/ocr-image)

We're starting a new chapter on OCR, and we're going to extract text from an image.

OCR (optical character recognition) extracts printed text from images. The `OCR_LATIN` model in the SDK handles any Latin-script language. The result is an array of text blocks, each with the recognized string, a bounding box on the image, and a confidence score.

`paragraph: false` asks for one block per visual line, useful for a UI that highlights one line at a time. With `true`, the engine merges adjacent lines into paragraphs. You would call it like so:

```ts
const { blocks } = ocr({
  modelId,
  image: "./examples/qvac/ocr/input/basic_test.jpg",
  options: { paragraph: false },
});
const result = await blocks;
```

Each block in `result` carries `text`, a `bbox` (the rectangle in pixel coordinates, for drawing a highlight overlay), and a `confidence` score:

```ts
for (const block of result) {
  console.log(block.text);
  if (block.bbox) console.log(`BBox: [${block.bbox.join(", ")}]`);
  if (block.confidence !== undefined) {
    console.log(`Confidence: ${block.confidence.toFixed(4)}`);
  }
}
```

`paragraph: false` returns one block per visual line. Set it to `true` and the SDK groups lines into paragraphs based on spacing.

> Note: the `bbox` field is in pixel coordinates relative to the input image. We use it to draw highlight overlays or to extract individual words for downstream processing.

## Put it to the test

1. Call `ocr({ modelId, image: <path>, options: { paragraph: false } })` and `await blocks`.
2. Loop through the blocks and log the text, bbox, and confidence.


# Connect through blind relays (/courses/qvac/en/p2p/blind-relays)

The previous lesson pre-downloaded a model from a regular Hyperdrive seed. This one makes sure peers behind NATs can find each other.

Most peers are behind NATs or firewalls, so they can't accept incoming connections, which makes peer-to-peer model downloads hard. Blind relays solve this: they're public Hyperswarm nodes that help two peers find each other without seeing the actual traffic.

The SDK reads its config during initialization. The `QVAC_CONFIG_PATH` env var has to be set *before* the `import "@qvac/sdk"` line, since module evaluation reads the config at import time.

The configuration file is a regular JavaScript module that exports the relay list. `swarmRelays` is the only field: an array of hex-encoded public keys (one per relay). The vendored `qvac.config.js` for this lesson uses placeholder keys; real deployments need a relay you control or a trusted public relay:

```js
// qvac.config.js
export default {
  swarmRelays: [
    "0000000000000000000000000000000000000000000000000000000000000001",
    "0000000000000000000000000000000000000000000000000000000000000002",
    "0000000000000000000000000000000000000000000000000000000000000003",
  ],
};
```

`QVAC_CONFIG_PATH` must be set BEFORE `import '@qvac/sdk'`, since module evaluation reads the config at import time. The env-then-import sequence looks like:

```ts
process.env["QVAC_CONFIG_PATH"] = "./qvac.config.js";

async function main() {
  const modelId = await loadModel({ modelSrc: LLAMA_3_2_1B_INST_Q4_0 });
  await unloadModel({ modelId });

  await downloadAsset({
    assetSrc: LLAMA_3_2_1B_INST_Q4_0,
    onProgress: (p) => {
      const mb = (n: number) => (n / 1e6).toFixed(1);
      process.stderr.write(`▸ ${p.percentage.toFixed(0)}% (${mb(p.downloaded)}/${mb(p.total)} MB)\n`);
    },
  });
}

main().catch(console.error);
```

The relays help establish the peer connection through NAT. They don't see the actual model data. The download happens peer-to-peer once the connection is set up.

> Note: blind relays only help with NAT traversal. They do not anonymize traffic or change what's downloaded.

## Put it to the test

1. Set up the relay config (file + env var) and call `loadModel` followed by `downloadAsset`. Both should succeed via the configured relays.


# Pre-download a model with downloadAsset (/courses/qvac/en/p2p/download-asset)

We're starting a new chapter on peer-to-peer (P2P), and we're going to decouple download from load.

`loadModel()` does two things: download the model file (if not cached), then load it into memory. For multi-hundred-megabyte models, that's a long wait on the first user request.

`downloadAsset()` separates the two steps. We call it once at install or app startup, then `loadModel()` skips straight to the in-memory part.

`downloadAsset()` pre-caches the model without loading it. Next `loadModel()` skips the download and goes straight to the in-memory load. You would call it like so:

```ts
await downloadAsset({
  assetSrc: LLAMA_3_2_1B_INST_Q4_0,
  onProgress: (p) => {
    const mb = (n: number) => (n / 1e6).toFixed(1);
    const line = `▸ Downloading ${p.percentage.toFixed(0)}% (${mb(p.downloaded)}/${mb(p.total)} MB)`;
    process.stderr.write(process.stderr.isTTY ? `\r${line}` : `${line}\n`);
  },
});
```

`downloadAsset` writes the file to the SDK's cache. The `loadModel()` below reuses it; `clearStorage: false` keeps the file around for the next run:

```ts
const modelId = await loadModel({ modelSrc: LLAMA_3_2_1B_INST_Q4_0 });
await unloadModel({ modelId, clearStorage: false });
```

The `onProgress` callback uses the same `{ percentage, downloaded, total }` shape as `loadModel()`.

> Note: re-running `downloadAsset` against an already-cached model is a no-op. The SDK checks the local cache before hitting the network.

## Put it to the test

1. Call `downloadAsset({ assetSrc, onProgress })` and `await` it. The model is now cached on disk.
2. Call `loadModel({ modelSrc: <same constant> })` and `unloadModel({ modelId, clearStorage: false })` to confirm the cached file loads into memory and frees cleanly.


# P2P (/courses/qvac/en/p2p)

Now that we've covered all the model surfaces, we're going to look at how the models actually get to your machine.

The peer-to-peer (or P2P) surface is about model distribution. QVAC pulls model files from a peer-to-peer Hyperdrive instead of a single CDN, which means downloads can come from whoever has the file already and can serve as a seeder. This chapter covers the two patterns you'll use most often: pre-downloading a model and configuring blind relays for NAT traversal.

[Start Lesson 1 →](/courses/qvac/en/p2p/download-asset)

### All lessons in this chapter

1. [Pre-download a model with downloadAsset](/courses/qvac/en/p2p/download-asset)
2. [Connect through blind relays](/courses/qvac/en/p2p/blind-relays)


# Chunk documents for RAG (/courses/qvac/en/rag/chunk-documents)

Now that we know how to ingest and search, let's handle longer documents.

A 1024-token embedding model can't take a 5000-word article in one shot. `chunk: true` tells `ragIngest` to split documents before embedding. The search results we saw before were whole documents. Chunked search returns the *passage* that matched, which is what we actually want for question-answering.

The default strategy is `paragraph`: the SDK splits on blank lines, then groups paragraphs to roughly hit `chunkSize` tokens. `chunkOverlap` keeps a few tokens of shared context at every boundary so a sentence that straddles two chunks isn't lost.

`chunk: true` splits each document before embedding. The default `paragraph` strategy splits on blank lines, then groups to roughly hit `chunkSize` tokens like so:

```ts
const result = await ragIngest({
  modelId,
  workspace: "tech",
  documents: samples,
  chunk: true,
  chunkOpts: {
    chunkSize: 200,
    chunkOverlap: 20,
  },
});
console.log(`Created ${result.processed.length} chunks from ${samples.length} documents`);
```

`processed` is now a list of chunks, not documents. Search against this workspace returns the specific passage that matched, not the whole document.

> Note: `chunkSize` is in tokens, not characters. A typical English word is roughly 1.3 tokens, so `chunkSize: 200` gives you chunks of about 150 words.

## Put it to the test

1. Call `ragIngest()` with `chunk: true` and `chunkOpts: { chunkSize: 200, chunkOverlap: 20 }`.
2. Log both `result.processed.length` (chunk count) and `samples.length` (document count).


# Delete documents from a RAG workspace (/courses/qvac/en/rag/delete-embeddings)

The previous lessons added documents to a workspace. This one removes them.

`ragDeleteEmbeddings({ workspace, ids })` takes the workspace name and a list of ids to remove. The ids come from the `processed` array returned by `ragIngest`. The editor prefills the ingest call and the id collection; the new piece is the search + delete + re-search flow.

Without a baseline, the second search is just a number. Counting search results before the delete would look as follows:

```ts
const before = await ragSearch({ modelId, workspace, query: "machine learning", topK: 5 });
console.log(`▸ Before delete: ${before.length} matches`);
```

Now the delete. `ragDeleteEmbeddings({ workspace, ids })` removes each id in the list, and the call is idempotent: passing an id that doesn't exist is a no-op. Here's how that looks in code:

```ts
await ragDeleteEmbeddings({ workspace, ids: [ids[0]!] });
console.log(`▸ Deleted embedding ${ids[0]}`);

const after = await ragSearch({ modelId, workspace, query: "machine learning", topK: 5 });
console.log(`▸ After delete: ${after.length} matches`);
```

In the call, `ids` is `string[]`. Pass one entry to delete a single document, or several to delete them in one call. After the delete, the workspace's index is updated. `ragSearch` returns one fewer result on the next call. If you're running repeated delete + ingest cycles, follow up with `ragReindex` after a few hundred writes to keep scores tight.

> Note: `ragDeleteEmbeddings` removes the documents but doesn't shrink the on-disk index until you reindex. The next `ragSearch` skips the deleted ids, but the index file still occupies the original size until `ragReindex` is called.

## Put it to the test

1. The editor prefills the `ragIngest` + ids collection. Add a search before the delete and log the count.
2. Call `ragDeleteEmbeddings({ workspace, ids: [firstId] })`, search again, and log the new count.


# Delete a RAG workspace and its data (/courses/qvac/en/rag/delete-workspace)

The list-and-close lesson closed the in-memory handle for a workspace without touching the data. This one drops the workspace and its on-disk files in one call.

`ragCloseWorkspace({ workspace, deleteOnClose: true })` releases the in-memory handle and removes the on-disk folder in a single pass. The flag is the only thing that flips the call from "close" to "delete":

```ts
await ragCloseWorkspace({ workspace, deleteOnClose: true });
```

The setup is the same as the other RAG lessons: a named workspace, an ingest to give it content, then the teardown. Right call for "throw away the index and start over":

```ts
const workspace = "delete-workspace-demo";
const samples = [
  "A temporary workspace holds documents we want to remove in one go.",
];

const modelId = await loadModel({ modelSrc: GTE_LARGE_FP16 });
await ragIngest({ modelId, workspace, documents: samples, chunk: false });
console.log(`▸ Created workspace '${workspace}' with ${samples.length} document`);

await ragCloseWorkspace({ workspace, deleteOnClose: true });
console.log(`▸ Deleted workspace '${workspace}' and its on-disk files`);

await unloadModel({ modelId });
```

After the call returns, the workspace is gone. The next `ragIngest` against the same name starts fresh; the next `ragSearch` returns "workspace not found" instead of empty results.

> Note: `deleteOnClose: true` is the destructive flag. Omit it and the same `ragCloseWorkspace` call only releases the in-memory handle, leaving the on-disk data intact for the next `ragOpen` (when one is available).

## Put it to the test

1. Call `ragIngest` to add a few documents to a named workspace, then log the workspace name and document count.
2. Call `ragCloseWorkspace({ workspace, deleteOnClose: true })` and log the deletion.


# Build RAG with SQLite-Vector (/courses/qvac/en/rag/external-vector-db)

Production setups usually want their own vector store. SQLite-Vector is the example here, but the same pattern works with pgvector, LanceDB, or ChromaDB. This lesson builds the RAG flow against SQLite-Vector directly: `embed()` and `loadModel()` are the SDK pieces, the storage layer and the scan are yours.

The pipeline has four steps: initialize SQLite with the vector extension, embed + insert each document, register and quantize the index, then run a top-K scan on the query.

The setup is two parts. `sqlite3InitModule()` registers the vector extension so SQLite knows about the `FLOAT32[1024]` type, and `loadModel()` brings in the embedder. The CREATE TABLE is a vanilla SQLite schema with `id`, `text`, and an `embedding` BLOB:

```ts
const sqlite3 = await sqlite3InitModule();
const db = new sqlite3.oo1.DB(":memory:", "c");
const modelId = await loadModel({ modelSrc: GTE_LARGE_FP16 });

db.exec(`
  CREATE TABLE IF NOT EXISTS documents (
    id INTEGER PRIMARY KEY,
    text TEXT NOT NULL,
    embedding BLOB NOT NULL
  )
`);
```

Each document gets one row. `embed()` returns `number[]`, and `vector_as_f32(?)` takes a JSON-stringified array as the bind parameter, so the loop wraps the embed and INSERT together like this:

```ts
for (const sample of samples) {
  const { embedding } = await embed({ modelId, text: sample.text });
  db.exec({
    sql: "INSERT INTO documents VALUES (?, ?, vector_as_f32(?))",
    bind: [sample.id, sample.text, JSON.stringify(embedding)],
  });
}
```

Before any scan, the index needs two calls. `vector_init` registers the column as a `FLOAT32[1024]` vector type, and `vector_quantize` builds the actual search index. Without these, scans fall back to byte-level comparison:

```ts
db.exec(`SELECT vector_init('documents', 'embedding', 'type=FLOAT32,dimension=1024')`);
db.exec(`SELECT vector_quantize('documents', 'embedding')`);
```

The search path mirrors the ingest: embed the query, run `vector_quantize_scan` to find the top-K nearest by distance, then log each result. The scan joins back to the row so we get the readable text:

```ts
const { embedding: queryEmbedding } = await embed({ modelId, text: query });
const results: { id: number; text: string; distance: number }[] = [];
db.exec({
  sql: `
    SELECT d.id, d.text, v.distance
    FROM documents d
    JOIN vector_quantize_scan('documents', 'embedding', vector_as_f32(?), 3) v
    ON d.id = v.rowid
  `,
  bind: [JSON.stringify(queryEmbedding)],
  rowMode: "object",
  callback: (row) => {
    results.push(row as { id: number; text: string; distance: number });
  },
});

for (const [i, r] of results.entries()) {
  console.log(`${i + 1}. [ID: ${r.id}] (distance: ${r.distance.toFixed(4)})`);
  console.log(`   ${r.text}`);
}
```

The dimension in `vector_init` must match the embedding model. `GTE_LARGE_FP16` produces 1024-dim vectors. A different model would need a different `dimension=N`. Mismatches fail at index time. The error shows up on `vector_init`, not on the first `vector_quantize_scan`.

> Note: SQLite-Vector is one option. The same pattern works with pgvector, LanceDB, ChromaDB, or any store that takes a `number[]` per row and exposes a top-K-by-distance query.

## Put it to the test

1. Initialize SQLite with `sqlite3InitModule()` and `new sqlite3.oo1.DB(":memory:", "c")`. `loadModel({ modelSrc: GTE_LARGE_FP16 })` and `CREATE TABLE documents (id INTEGER PRIMARY KEY, text TEXT NOT NULL, embedding BLOB NOT NULL)`.
2. For each `samples[i]`, call `await embed({ modelId, text: samples[i].text })` and `INSERT INTO documents VALUES (?, ?, vector_as_f32(?))` with the JSON-stringified embedding.
3. Run `SELECT vector_init('documents', 'embedding', 'type=FLOAT32,dimension=1024')` and `SELECT vector_quantize('documents', 'embedding')`.
4. `await embed({ modelId, text: query })` for the query, then a `vector_quantize_scan` join for top-3, and console.log each result's id, distance, and text.


# RAG (/courses/qvac/en/rag)

Now that we can do an in-memory search, we're going to add persistence.

The tiny search engine from chapter 3 lived entirely in memory, and the corpus was rebuilt every run. RAG (Retrieval-Augmented Generation) gives us a workspace: a folder on disk where vectors are stored between runs, indexed by name, and ready to query. The `rag*` family covers the full lifecycle: ingest, search, chunk, reindex, list, save pre-computed vectors, delete individual entries, drop a whole workspace. When the built-in store isn't enough, the same `embed()` call feeds an external vector DB.

[Start Lesson 1 →](/courses/qvac/en/rag/ingest-documents)

### All lessons in this chapter

1. [Ingest documents into a workspace](/courses/qvac/en/rag/ingest-documents)
2. [Search a RAG workspace](/courses/qvac/en/rag/search-workspace)
3. [Chunk documents for RAG](/courses/qvac/en/rag/chunk-documents)
4. [Reindex a RAG workspace after many writes](/courses/qvac/en/rag/reindex)
5. [List and close workspaces](/courses/qvac/en/rag/list-and-close)
6. [Save pre-computed embeddings to a RAG workspace](/courses/qvac/en/rag/save-embeddings)
7. [Delete documents from a RAG workspace](/courses/qvac/en/rag/delete-embeddings)
8. [Delete a RAG workspace and its data](/courses/qvac/en/rag/delete-workspace)
9. [Build RAG with an external vector DB](/courses/qvac/en/rag/external-vector-db)


# Ingest documents into a workspace (/courses/qvac/en/rag/ingest-documents)

Now that we've built the smallest useful search engine in chapter 3, we're going to scale it up to a workspace that lives on disk.

Real search needs persistence: we build the index once, query it many times. The chapter 3 search kept everything in memory; `ragIngest` writes the vectors to a folder under the SDK's data directory and reads them back on the next call.

`ragIngest({ modelId, documents, workspace, chunk: false })` runs the whole pipeline in one call: chunk (skip, in this lesson), embed, save.

Without chunking, each document becomes one entry. This is especially useful for long documents. You would call it like so:

```ts
const result = await ragIngest({
  modelId,
  workspace,
  documents: samples,
  chunk: false,
});
console.log(`Ingested ${result.processed.length} documents`);
console.log("First entry:", result.processed[0]);
```

`chunk: false` tells the SDK to skip splitting. The next lesson uses chunking for longer text.

> Note: re-running `ragIngest` against the same workspace doesn't double-ingest. The SDK identifies already-embedded documents and skips them. The first run takes the longest; the second is faster.

## Put it to the test

1. Call `ragIngest({ modelId, workspace, documents: samples, chunk: false })`.
2. Console.log `result.processed.length` and inspect the first entry to confirm chunking was off.


# List and close workspaces (/courses/qvac/en/rag/list-and-close)

Workspaces live on disk under the SDK's data directory. `ragListWorkspaces()` reads them back so we can see what's there. `ragCloseWorkspace({ workspace, deleteOnClose })` tears one down.

The list comes back as an array of `{ name, open }` objects, where the `open` flag tells us if the workspace is still loaded or already torn down. The loop prints one line per workspace:

```ts
for (const ws of workspaces) {
  console.log(`▸ ${ws.name} (${ws.open ? "open" : "closed"})`);
}
```

Closing a workspace is a single call. `deleteOnClose: true` drops the in-memory handle and wipes the on-disk folder. The call itself only needs the workspace name:

```ts
await ragCloseWorkspace({ workspace: "tech", deleteOnClose: true });
```

If we pass `deleteOnClose: false`, the workspace files stay on disk but the in-memory handle is dropped. A workspace auto-closes when the process exits, so this call is for explicit cleanup during long-running sessions.

> Note: `deleteOnClose` is the right name for "remove from disk." Passing `false` is the right call when you want to keep the data but release the in-memory handle.

## Put it to the test

1. The editor prefills the `ragListWorkspaces()` call. Loop through and log each workspace's name and open status.
2. Call `ragCloseWorkspace({ workspace, deleteOnClose: true })` to remove the workspace from disk.


# Reindex a RAG workspace after many writes (/courses/qvac/en/rag/reindex)

After enough `ragDeleteEmbeddings` and re-ingest cycles, scores can drift. The index no longer reflects the source text. `ragReindex({ workspace })` processes the stored text for that workspace and regenerates every embedding from scratch.

`ragReindex` runs k-means over the stored embeddings. K-means needs at least `K` samples to produce `K` centroids. HyperDB's `NUM_CENTROIDS` defaults to 4 and is not configurable through the SDK, so any workspace with fewer than 4 documents gets a no-op reindex with `reason: "insufficient documents"`.

The reindex is idempotent: running it twice in a row is safe, the second run finds nothing to do.

The result is `{ reindexed, details }`. `reindexed: true` means anything was rebuilt; `false` carries a `reason` you can log. The call is two lines:

```ts
const result = await ragReindex({ workspace });
console.log("Reindexed:", result.reindexed);
if (!result.reindexed) {
  console.log("Reason:", result.details?.reason ?? "unknown");
}
```

For this 4-recipe demo the reindex deliberately bails. The point of the lesson is to read the result and know what to do with it: in a real workspace with thousands of documents, this branch never fires, you just log the success and proceed with the updated embeddings.

> Note: a reindex on a large workspace takes roughly the same time as the original ingest. Plan to run it during a quiet window or as a one-shot maintenance task.

## Put it to the test

1. Call `await ragReindex({ workspace })` and console.log `result.reindexed` plus the reason from `result.details?.reason` when it's `false`.


# Save pre-computed embeddings to a RAG workspace (/courses/qvac/en/rag/save-embeddings)

Now that we have the chunk-embed-save pipeline running through `ragIngest`, we're going to break it apart. This lesson covers the third step on its own: saving vectors you've already computed.

`ragSaveEmbeddings` stores one `RagEmbeddedDoc` per entry. Each one has an `id`, the `content`, the `embedding` (a `number[]`), and the `embeddingModelId` that produced the vector. The `embeddingModelId` must match the `modelId` passed to `ragSearch` later, since cosine similarity is only meaningful between vectors from the same model.

`ragSaveEmbeddings({ workspace, documents })` writes the `RagEmbeddedDoc` array to disk and returns `processed`, a per-entry `{ status, id, error }` array. Counting fulfilled entries gives you the saved count:

```ts
const saveResult = await ragSaveEmbeddings({
  workspace: "save-embeddings-demo",
  documents: embeddedDocs,
});

const saved = saveResult.filter((r) => r.status === "fulfilled").length;
console.log(`▸ Saved ${saved}/${saveResult.length} embeddings to the workspace`);
```

> Note: `ragSaveEmbeddings` is a storage-only operation. The SDK doesn't need the model on hand to save. It only needs the model to search. Re-running with the same `id` is idempotent; the existing entry is overwritten.

## Put it to the test

1. Call `ragSaveEmbeddings({ workspace, documents })` with the prefilled `embeddedDocs` and log the count of fulfilled saves out of `saveResult.length`.


# Search a RAG workspace (/courses/qvac/en/rag/search-workspace)

We've got a workspace from the previous lesson. This lesson queries it.

`ragSearch({ modelId, workspace, query, topK })` returns an array of `{ score, content }`. The `modelId` must match the one used at ingest, because different models put vectors in different spaces and cross-model similarity scores are meaningless.

Result is sorted by score descending. The call passes all four options to `ragSearch`:

```ts
const results = await ragSearch({
  modelId,
  workspace,
  query: "How do I make a peanut butter sandwich?",
  topK: 3,
});
```

`slice(0, 80)` clips the preview to one line. 80 chars is arbitrary, pick what fits your terminal. Also, the loop uses `toFixed(4)` for the score so keep that in mind:

```ts
let i = 0;
for (const result of results) {
  console.log(`Score ${result.score.toFixed(4)}: ${result.content.slice(0, 80)}...`);
  i += 1;
}
```

Each `result` carries the same `content` we put in, with a `score` field added (higher means more similar).

> Note: `topK` is an integer, not a "score threshold". If you want to filter by confidence, sort by `score` and drop anything below your threshold.

## Put it to the test

1. Call `ragSearch({ modelId, workspace, query, topK })` and store the results.
2. Add a `for (const result of results)` loop that console.logs `result.score` and the first 80 chars of `result.content`.


# Compare embeddings with cosine similarity (/courses/qvac/en/text-embeddings/cosine-similarity)

We have three vectors from the last lesson. Two are similar (both about a fox). One is unrelated (Python). The comparison is one short function: multiply matching positions, then sum them up.

For two 1024-dimension vectors `a` and `b`, the similarity is the dot product `Σᵢ aᵢ × bᵢ`, the sum of the element-wise products. That's all there is to it. Strictly speaking, "cosine similarity" divides by the magnitudes too, but for normalized embeddings like these the dot product alone ranks similarity correctly.

Same batch `embed()` call, but destructured into three named variables. Different destructuring pattern than the previous lesson, since we want three separate handles rather than one batch. You would destructure like so:

```ts
const { embedding: [emb1, emb2, emb3] } = await embed({ modelId, text: texts });
```

The `cosineSimilarity` helper is one-line math: a loop that multiplies matching positions and sums. Output range is `-1` to `1`, but `GTE_LARGE_FP16` typically produces `0` to `1`. Note that the `?? 0` is defensive against sparse arrays. The function looks as follows:

```ts
function cosineSimilarity(vecA: number[], vecB: number[]) {
  let dotProduct = 0;
  for (let i = 0; i < vecA.length; i++) {
    dotProduct += (vecA[i] ?? 0) * (vecB[i] ?? 0);
  }
  return dotProduct;
}
```

With `cosineSimilarity` defined, the side-by-side compare is two calls and two logs. The similar pair (text 1 vs text 2) should land high (around 0.87), the unrelated one (text 1 vs text 3) should land low (around 0.11). `.toFixed(4)` rounds each to four decimal places so the numbers stay readable on one line:

```ts
const similarity1 = cosineSimilarity(emb1, emb2);
const similarity2 = cosineSimilarity(emb1, emb3);
console.log(`Similarity between texts 1 and 2 (similar meaning): ${similarity1.toFixed(4)}`);
console.log(`Similarity between texts 1 and 3 (different topics): ${similarity2.toFixed(4)}`);
```

> Note: `vecA[i] ?? 0` defends against `undefined` if the array is ever sparse, which doesn't happen for our case. Drop the `?? 0` and use `vecA[i]!` if you want the leaner read.

## Put it to the test

1. Call `embed({ modelId, text: texts })` and destructure the batch as `emb1`, `emb2`, `emb3`.
2. Define `cosineSimilarity(vecA: number[], vecB: number[])` that returns the dot product.
3. Call `cosineSimilarity(emb1, emb2)` and `cosineSimilarity(emb1, emb3)`, then console.log both with `.toFixed(4)`.


# Embed many strings at once (/courses/qvac/en/text-embeddings/embed-many-strings)

In the previous lesson we embedded one string at a time. That works for learning, but if we've got a thousand documents we don't want a thousand round-trips.

`embed()` accepts a string or a string array. Pass an array, get back an array of vectors.

The array overload of `embed()` is the same call with `text` as `string[]`. Pass a `string[]` and the batch call returns `{ embedding: number[][] }` like so:

```ts
const { embedding: batchEmbeddings } = await embed({
  modelId,
  text: texts,
});
console.log(`Input: ${texts.length} texts`);
console.log(`Output: ${batchEmbeddings.length} embeddings`);
console.log(`Each embedding dimensions: ${batchEmbeddings[0]!.length}`);
```

Each inner array is one 1024-number vector, in the same order as the input. The model stays loaded, so the second call is much faster than the first (only the embedding step runs).

> Note: `batchEmbeddings[0]` is the first vector, `batchEmbeddings[0][0]` is the very first number of that vector. Mind your brackets if you start indexing.

## Put it to the test

1. Call `await embed({ modelId, text: texts })` and destructure `{ embedding: batchEmbeddings }`.
2. `console.log` the input count (`texts.length`), the output count (`batchEmbeddings.length`), and the dimension of the first vector (`batchEmbeddings[0]!.length`).


# Embed a single string (/courses/qvac/en/text-embeddings/embed-single-text)

Now that we have a model in memory, let's hand it a string and see what comes back.

`embed({ modelId, text: "..." })` resolves to `{ embedding: number[] }`. For `GTE_LARGE_FP16` the array has 1024 numbers, one per feature the model learned during training. Each number is a small float, mostly between -1 and 1.

We don't need to understand what each number means yet. We're going to compare vectors in the next lessons. For now, the part to remember is that every input produces a 1024-number array on `GTE_LARGE_FP16`.

The `embed()` call for one text takes a modelId and a single `text` string. For `GTE_LARGE_FP16` the return is a 1024-number array. You would call it like so:

```ts
const { embedding } = await embed({
  modelId,
  text: "Hello, world!",
});
console.log("Input:", "Hello, world!");
console.log("Embedding dimensions:", embedding.length);
console.log("First 10 values:", embedding.slice(0, 10));
```

`embedding` is a `number[]`. Use `.length` for the dimension and `.slice(0, 10)` to peek at the first few values without spamming the console.

> Note: the SDK always returns the same `number[]` shape regardless of how long the input text is. Short sentences and paragraphs both produce a 1024-number vector for `GTE_LARGE_FP16`. The numbers are different, but the shape is identical.

## Put it to the test

1. Call `embed({ modelId, text: "Hello, world!" })` and destructure `embedding` from the result.
2. Log the input, the dimensions, and the first 10 values.


# Text embeddings (/courses/qvac/en/text-embeddings)

Now that we know how to call a model, we're going to start working with text as numbers.

An embedding is a list of numbers derived from a piece of text. Two texts with similar meaning produce similar number lists, even when the words differ.

[Start Lesson 1 →](/courses/qvac/en/text-embeddings/load-embedding-model)

### All lessons in this chapter

1. [Load an embedding model](/courses/qvac/en/text-embeddings/load-embedding-model)
2. [Embed a single string](/courses/qvac/en/text-embeddings/embed-single-text)
3. [Embed many strings at once](/courses/qvac/en/text-embeddings/embed-many-strings)
4. [Compare embeddings with cosine similarity](/courses/qvac/en/text-embeddings/cosine-similarity)
5. [Build a tiny semantic search](/courses/qvac/en/text-embeddings/tiny-search)


# Load an embedding model (/courses/qvac/en/text-embeddings/load-embedding-model)

Now that we've finished chapter 2, we're going to start a new line of work: embeddings.

An embedding is a list of numbers that encodes what a piece of text means. A real model maps a sentence to a 1024-number array. We can't read the numbers, but we can compare them with math, and two similar sentences end up with similar numbers.

The QVAC SDK loads embedding models through the same `loadModel()` we used in chapter 1. Same import, same options, same `modelId` pattern. The only thing that changes is the constant we hand to `modelSrc`.

`GTE_LARGE_FP16` is the SDK's embedding model. The call is the same `loadModel` from chapter 1, with `GTE_LARGE_FP16` in `modelSrc`:

```ts
const modelId = await loadModel({ modelSrc: GTE_LARGE_FP16 });
console.log("modelId:", modelId);
```

Save the `modelId`. The next lessons vectorize text against the same loaded model, and reloading between calls would be wasteful.

> Note: the import line changes too. We'll have both `loadModel` and `GTE_LARGE_FP16` in the same `import { ... }` statement, since they come from the same `@qvac/sdk` package.

## Put it to the test

1. Inside `main()`, call `loadModel` with `{ modelSrc: GTE_LARGE_FP16 }`.
2. `await` the result and store it in a variable called `modelId`. Log `modelId` to stdout.


# Build a tiny semantic search (/courses/qvac/en/text-embeddings/tiny-search)

Now that we can compare two vectors, we're ready to build the smallest useful search engine.

A semantic search is just three steps: embed a small corpus once, embed the query separately, then loop through the corpus picking the highest score. The corpus doesn't have to be perfect. Three documents are enough to see the pattern.

Embedding the whole corpus in a single batch call would look like this:

```ts
const { embedding: corpusVectors } = await embed({ modelId, text: corpus });
```

The query goes through the same `embed()` API with a single string, destructured into a one-element array. Note that the `[queryEmbedding]` array-destructure is the same shape as the corpus destructure, just with a single element:

```ts
const { embedding: [queryEmbedding] } = await embed({ modelId, text: query });
```

With the corpus and query both embedded, the loop does the work: it scores every corpus vector against the query, tracks the highest score, and remembers its index. The whole thing runs in O(N·d), N cosine calls each touching d dimensions, and that's the brute-force shape `ragSearch` generalizes to a vector index for O(log N):

```ts
let bestIdx = 0;
let bestScore = -Infinity;
for (let i = 0; i < corpusVectors.length; i++) {
  const score = cosineSimilarity(queryEmbedding, corpusVectors[i]!);
  if (score > bestScore) {
    bestScore = score;
    bestIdx = i;
  }
}
console.log(`Query: ${query}`);
console.log(`Best match: ${titles[bestIdx]} (score ${bestScore.toFixed(4)})`);
```

After the loop, `bestIdx` holds the position of the highest-scoring corpus vector, and `titles[bestIdx]` is the document title closest in meaning to the query.

> Note: the same pattern scales to thousands of documents. The only thing that changes is where the vectors live. In memory for a few hundred, on disk for the rest. Chapter 4 shows the on-disk version.

## Put it to the test

1. Call `embed({ modelId, text: corpus })` to embed the corpus as a single batch.
2. Call `embed({ modelId, text: query })` to embed the query as a separate single call.
3. Loop through the corpus vectors, computing `cosineSimilarity(queryEmbedding, corpusVectors[i])` and tracking `bestIdx` and `bestScore`.
4. Log the query and the best-matching title with the score formatted to 4 decimals.


# Run multiple completions in parallel (/courses/qvac/en/text-generation/concurrent)

In the previous lessons we've been calling `completion()` once at a time. Now let's see what happens when we fire two calls simultaneously.

A loaded model is one native context under the hood: one KV-cache, one decode loop. Two completions on the same model cannot literally run at the same time. The SDK uses a per-`(kind, modelId)` FIFO admission queue, so the second request waits its turn instead of being rejected with `RequestRejectedByPolicyError`.

The trick is calling `completion()` twice in the same tick before awaiting either.

First we'd need to fire both `completion()` calls in the same tick, no `await` between. The two back-to-back calls look as follows:

```ts
const r1 = completion({ modelId, history, stream: false, captureThinking: true });
const r2 = completion({ modelId, history, stream: false, captureThinking: true });
```

Next, we'd use the canonical pattern to await the results. `Promise.all` is the surface; the wait over both resolves to:

```ts
const [text1, text2] = await Promise.all([r1.text, r2.text]);
console.log(`▸ req-A: ${text1}`);
console.log(`▸ req-B: ${text2}`);
console.log(`▸ Both completed.`);
```

Both `completion()` calls fire synchronously, so both end up "in flight" against the SDK's queue. `Promise.all` then waits for them. On the same model, the queue runs them in order. On different models, the per-model key means they run in parallel.

> Note: if you `await` the first call before the second `completion()`, the second call never sees contention and the queue is bypassed. Fire both first, await together.

## Put it to the test

1. Call `completion({ modelId, history, stream: false, captureThinking: true })` twice in a row without awaiting either, assigning each to a variable.
2. Wrap the awaits in `Promise.all([r1.text, r2.text])` and log the result of each with a label like `req-A` and `req-B`.
3. After both finish, log `▸ Both completed.` so the output has a final line.


# Handle event types in a completion stream (/courses/qvac/en/text-generation/event-types)

Earlier chapters used `result.tokenStream`, a flat string iterable of the model's text. The stream gives you raw output, but content and thinking look identical without type tags. The `events` surface dispatches each by type, so they arrive separately.

On a completion run, this surface is the canonical one. Each item carries a payload: a content delta, a thinking block, a tool call, a stats frame, the terminal `done`, or the raw text.

The previous lesson covered `contentDelta` and `thinkingDelta` firing side by side. This one covers the full set, plus the `result.final` promise that joins them all into one object.

Here's how you'd dispatch the events in a streaming loop:

```ts
for await (const event of result.events) {
  switch (event.type) {
    case "contentDelta":
      process.stdout.write(event.text);
      break;
    case "thinkingDelta":
      process.stderr.write(`[think] ${event.text}`);
      break;
    case "toolCall":
      console.log(`▸ tool ${event.call.name}(${JSON.stringify(event.call.arguments)})`);
      break;
    case "completionStats":
      console.log(`▸ ${event.stats.tokensPerSecond?.toFixed(1)} tok/s`);
      break;
    case "completionDone":
      break;
  }
}
```

`contentDelta` is the model's text token-by-token, written to stdout. `thinkingDelta` is the chain-of-thought stream, written to stderr with a `[think]` prefix so the reasoning doesn't get mixed into the user-visible response. `toolCall` marks a function-call emission in the response stream. `completionStats` carries throughput numbers.

After the loop, `await result.final` joins them into one object: `contentText`, `thinkingText`, `toolCalls`, `stats`, `stopReason`, `raw.fullText`. Reading the aggregate would look like this:

```ts
console.log();
const final = await result.final;
console.log(`▸ Final contentText: ${final.contentText}`);
console.log(`▸ Stop reason: ${final.stopReason}`);
```

> Note: `tokenStream` still works for simple cases, but new code should consume `events` for streaming and `final.contentText` for the aggregated result.

## Put it to the test

1. Iterate `result.events` with a `for await ... { switch (event.type) { ... } }`. Cases: `contentDelta` writes `event.text` to stdout, `thinkingDelta` writes `[think] ${event.text}` to stderr, `completionDone` breaks.
2. After the loop, `await result.final` and log `final.contentText` and `final.stopReason`.


# Text generation (/courses/qvac/en/text-generation)

Now that we can run a single completion, it's time to look at the structural knobs behind it.

This chapter covers the chat completion surface in depth: the kinds of events the model can emit, how to keep a conversation going across multiple turns, how to wire tools in, and how to format the response as schema-valid JSON.

[Start Lesson 1 →](/courses/qvac/en/text-generation/thinking-content)

### All lessons in this chapter

1. [Capture thinking content from a completion](/courses/qvac/en/text-generation/thinking-content)
2. [Send a multi-turn conversation](/courses/qvac/en/text-generation/multi-turn)
3. [Use tool calls from a completion](/courses/qvac/en/text-generation/tool-calls)
4. [Plug MCP into a completion](/courses/qvac/en/text-generation/mcp)
5. [Stream raw tokens from a completion](/courses/qvac/en/text-generation/raw-output)
6. [Cache conversation state across turns](/courses/qvac/en/text-generation/kv-cache)
7. [Run multiple completions in parallel](/courses/qvac/en/text-generation/concurrent)
8. [Handle event types in a completion stream](/courses/qvac/en/text-generation/event-types)


# Cache conversation state across turns (/courses/qvac/en/text-generation/kv-cache)

Generating the first response from a multi-turn history reprocesses every prior turn. The KV cache is the trick that skips that: save the model's internal state after the first turn, then replay it on the second turn with only the new user message.

`kvCache: true` on the second `completion()` call skips the prior turn's reprocessing. The same `history` prefix is required; change the prefix and the cache misses, falling back to a full reprocess.

Setting `kvCache: true` asks the SDK to save the model's internal state after this turn. Consider the first call with the flag on:

```ts
const r1 = completion({ modelId, history, stream: true, kvCache: true });
for await (const token of r1.tokenStream) process.stdout.write(token);
const final1 = await r1.final;
```

The cache key is the history prefix. To keep that prefix intact for the next hit, push `cacheableAssistantContent` back into `history`:

```ts
history.push({
  role: "assistant",
  content: final1.cacheableAssistantContent ?? final1.contentText,
});
history.push({ role: "user", content: "What about Germany?" });
```

Same flag, same `history` reference. The second turn replays the cache; comparing `stats` after proves it. You would call it like so:

```ts
const r2 = completion({ modelId, history, stream: true, kvCache: true });
for await (const token of r2.tokenStream) process.stdout.write(token);
const final2 = await r2.final;

console.log(`\n▸ First: ${JSON.stringify(final1.stats)}`);
console.log(`▸ Second (cached): ${JSON.stringify(final2.stats)}`);
```

The two `stats` objects show the speedup. On a long-running assistant with thousands of prior turns, the cached path runs ten to a hundred times faster than the cold path.

> Note: `final.cacheableAssistantContent` is the exact text the cache was saved against. Fall back to `final.contentText` if it's `undefined` (some models and tool-using flows don't expose it).

## Put it to the test

1. Build a `history` array starting with one user turn. First turn: call `completion({ modelId, history, kvCache: true, stream: true })`, drain `tokenStream`, then `await r1.final`.
2. Push the assistant turn back into `history` using `final1.cacheableAssistantContent ?? final1.contentText`. Append the next user turn.
3. Second turn: call `completion()` again with the same `history` and `kvCache: true`. Compare `final.stats` for both runs.


# Plug MCP into a completion (/courses/qvac/en/text-generation/mcp)

Now that we know how to wire tools in by hand, let's see how MCP makes it automatic.

MCP (Model Context Protocol) is a standard way for the model to call external tools. We run an MCP server (search, file access, a database client), hand the SDK a `Client` from `@modelcontextprotocol/sdk/client/index.js`, and the SDK routes the model's tool calls through it automatically.

With an MCP server, the SDK reads the tool list at call time and adapts on its own, with no per-model schema rewrites. The lesson prefills the client setup and the `loadModel` call, so the only thing left to figure out is how the client threads into `completion()`.

The `mcp` field is the only piece of wiring that ties an MCP client into a completion call. Once it's in, the SDK treats MCP-backed tools the same as native ones. A wired-up call looks like:

```ts
const result = completion({
  modelId,
  history: [
    { role: "user", content: "What's the current weather in New York?" },
  ],
  mcp: [{ client: mcpClient, includeResources: false }],
  stream: true,
  captureThinking: true,
});
```

The event loop is the same as in the tool-calls lesson, since the SDK fires identical `toolCall` events whether the tool is native or MCP-backed:

```ts
for await (const event of result.events) {
  if (event.type === "toolCall") {
    console.log(`▸ Tool: ${event.call.name}(${JSON.stringify(event.call.arguments)})`);
  }
  if (event.type === "contentDelta") {
    process.stdout.write(event.text);
  }
}
```

When the sampled output is a tool call from the MCP server, the SDK routes the call through `mcpClient`. We see the same `toolCall` events as for native tools.

> Note: `captureThinking: true` is the option the first lesson in this chapter introduced. The model's `thinking` arrives on `thinkingDelta` events and stays out of the `contentDelta` stream the tool call comes from. The lesson's loop ignores `thinkingDelta` so the runner's OUTPUT panel only shows the tool calls and the final answer.

> Note: MCP needs `@modelcontextprotocol/sdk` installed in your project. The SDK does not bundle it.

## Put it to the test

1. Install `@modelcontextprotocol/sdk`. The editor prefills a `Client` connected to `npx -y @oevortex/ddg_search` as a starter.
2. Call `completion({ modelId, history, stream: true, captureThinking: true, mcp: [{ client: mcpClient, includeResources: false }] })` and iterate `result.events`, logging `toolCall` events and writing `contentDelta` tokens to stdout. The loop ignores `thinkingDelta` so the runner's OUTPUT panel only shows the tool calls and the final answer. Then `await mcpClient.close()`.


# Send a multi-turn conversation (/courses/qvac/en/text-generation/multi-turn)

The previous lesson made a single `completion()` call. This one uses it for a real conversation.

A single call is stateless on the SDK side. Only what we put in `history` reaches the model. To get a real conversation going we keep the `history` array and append each turn before the next call.

The SDK doesn't keep conversation state, so we own the `history` array as follows:

```ts
const history: Array<{ role: string; content: string }> = [
  { role: "user", content: "What is the capital of France?" },
];
```

Pushing the assistant's reply back into the same `history` array is what makes the next call see it as context. You would write the first turn like so:

```ts
const r1 = completion({ modelId, history, stream: true, captureThinking: true });
for await (const event of r1.events) {
  if (event.type === "contentDelta") process.stdout.write(event.text);
}
const text1 = await r1.text;
history.push({ role: "assistant", content: text1 });
```

Same `history` reference, with the assistant turn already in. The follow-up user turn, then the second `completion`:

```ts
history.push({ role: "user", content: "And which river runs through it?" });
const r2 = completion({ modelId, history, stream: true, captureThinking: true });
for await (const event of r2.events) {
  if (event.type === "contentDelta") process.stdout.write(event.text);
}
```

The same `history` reference goes into both calls. The first question and the model's answer from the first turn are both in the array by the time we make the second call, so the model can answer the follow-up.

> Note: `captureThinking: true` is the option the first lesson in this chapter introduced. It splits the model's `thinking` from the answer, so `contentDelta` only carries the final response. The thinking tokens arrive on `thinkingDelta` events too; we ignore them here so the runner's OUTPUT panel stays clean. The first lesson in this chapter shows how to surface the thinking when you want it.

> Note: the SDK never mutates `history` for you. If you forget to push the assistant turn back, the next call's input contains only the user turns and the prior assistant answer is gone, so a follow-up that asks about it gets an answer without that context.

## Put it to the test

1. Above the first `completion()` call, declare `const history: Array<{ role: string; content: string }> = [{ role: "user", content: "What is the capital of France?" }]`.
2. After awaiting the first response, push `{ role: "assistant", content: text1 }`, then push `{ role: "user", content: "And which river runs through it?" }`.
3. Call `completion()` a second time, passing the same `history`, and stream its `contentDelta` tokens to stdout.


# Generate structured JSON output (/courses/qvac/en/text-generation/raw-output)

Sometimes we want structured data. `responseFormat` is the option: tell the model the shape we need, and the engine enforces it through grammar.

The option takes one of three values, each giving a different strength of guarantee over the output:

* `text` (default). Free-form text. No constraints.
* `json_object`. The output is some valid JSON object, but the keys aren't pinned. Small models tend to emit `{}`.
* `json_schema`. The output matches a JSON Schema we provide. The grammar engine forces the keys, the types, and the required fields.

Telling TypeScript the schema is read-only lets inference run end-to-end. The schema constant declared with `as const` looks like this:

```ts
const PERSON_SCHEMA = {
  type: "object",
  properties: {
    name: { type: "string" },
    age: { type: "integer" },
    occupation: { type: "string" },
  },
  required: ["name", "age", "occupation"],
  additionalProperties: false,
} as const;
```

The `responseFormat` option hands the schema to the grammar engine. The engine constrains the keys, the types, and the required fields, so the streamed output is always valid JSON. The correct completion call would look like:

```ts
const result = completion({
  modelId,
  history: [
    { role: "system", content: "Extract structured info about people." },
    { role: "user", content: "Hi, I'm Alice, 30, data engineer." },
  ],
  captureThinking: true,
  responseFormat: {
    type: "json_schema",
    json_schema: { name: "person", schema: PERSON_SCHEMA },
  },
  stream: true,
});

let raw = "";
for await (const event of result.events) {
  if (event.type === "contentDelta") {
    raw += event.text;
    process.stdout.write(event.text);
  }
}
```

Reading the schema-valid output back is the last step of the pipeline. Note that the `JSON.parse` step requires the loop to finish first. Use it as follows:

```ts
const parsed = JSON.parse(raw.trim()) as {
  name: string;
  age: number;
  occupation: string;
};
console.log("\n▸ Parsed:", parsed);
```

The result is already valid JSON. No regex, no repair, no fallback parsing.

> Note: `as const` on the schema tells TypeScript the value is read-only. The runtime API does not care, but the type system is happier this way.

## Put it to the test

1. Define a `PERSON_SCHEMA` (or similar) as a `const` JSON Schema object with `type: "object"`, `properties`, and `required`.
2. Pass `responseFormat: { type: "json_schema", json_schema: { name: "person", schema: PERSON_SCHEMA } }` to `completion()`, stream `contentDelta` events.
3. After the loop, `await result.final` and `JSON.parse(final.contentText)`. Log the parsed object.


# Capture thinking content from a completion (/courses/qvac/en/text-generation/thinking-content)

Chapter 2 starts with one specific feature of the `completion()` surface: thinking content.

Some models emit a `<think>...</think>` reasoning block before the final answer. The reasoning arrives in the same token stream that produces the response. Without capture, that reasoning either ends up inside the response text, mixed into the user-visible answer, or the chat-template layer strips it and you can't recover it from the SDK surface.

`captureThinking: true` tells the SDK to parse `<think>` blocks out of the raw model output and surface them as their own event type, separate from the final answer. The call would look like:

```ts
const result = completion({
  modelId,
  history: [{ role: "user", content: "Why is the sky blue?" }],
  stream: true,
  captureThinking: true,
});
```

That single boolean changes the shape of the events stream. A model that emits `<think>The user is asking about Rayleigh scattering.</think>The sky appears blue because...` fires four event types: `thinkingDelta` for each reasoning token (before the answer), then `contentDelta` for each answer token (after the reasoning completes), then `completionStats` with throughput numbers, then `completionDone` with `stopReason: "eos"`.

`contentDelta` goes to stdout, `thinkingDelta` routes to stderr with a `[think]` prefix so it doesn't mix with the response. Both event types carry the same `text` field. The event loop for this kind of call would look like:

```ts
for await (const event of result.events) {
  switch (event.type) {
    case "contentDelta":
      process.stdout.write(event.text);
      break;
    case "thinkingDelta":
      process.stderr.write(`[think] ${event.text}`);
      break;
  }
}
```

After the loop, the joined reasoning sits in `final.thinkingText`. Reading the full reasoning block at once is one `await` away:

```ts
const final = await result.final;
if (final.thinkingText) {
  console.log(`\n▸ Thinking: ${final.thinkingText}`);
}
```

For convenience, the aggregated `final.thinkingText` joins every reasoning token into one string. We read it after the events loop exits.

> Note: `process.stdout` is reserved for the user-visible answer across this curriculum. Reasoning is part of the model's behaviour but not the answer, so it goes to `process.stderr`. On small models the SDK surfaces thinking as a stream of short tokens (single words and punctuation), so a per-token `process.stderr.write` in the event loop duplicates the label on every fragment. Read `final.thinkingText` after the events loop and emit one labeled block instead. If you'd rather show both interleaved to the user, write both to stdout with different prefixes.

## Put it to the test

1. Set `captureThinking: true` in the `completion()` options.
2. Handle `thinkingDelta` by writing `event.text` to `process.stderr` with a `[think] ` prefix.
3. After the events loop, await `result.final` and log `final.thinkingText` with a `▸ Thinking:` prefix.


# Use tool calls from a completion (/courses/qvac/en/text-generation/tool-calls)

Now that we know how to hold a conversation, let's give the model a tool it can reach for.

A tool is a function the model is allowed to call. We describe what the function does and what arguments it takes. The model calls it when the sampled output points to it.

A tool definition has three fields: `name`, `description`, and `parameters` (a JSON Schema describing the arguments). The model emits `toolCall` events when the sampled output is a tool call. We handle the call ourselves and push the result back into `history` so the next call has the tool result in its input.

The shape declared with the SDK would look like the following:

```ts
const tools = [
  {
    name: "get_weather",
    description: "Get current weather for a city",
    parameters: {
      type: "object",
      properties: { city: { type: "string" } },
      required: ["city"],
    },
  },
];
```

Passing the `tools` array is what makes them available to the model:

```ts
const result = completion({
  modelId,
  history: [{ role: "user", content: "What's the weather in Tokyo?" }],
  tools,
  stream: true,
  captureThinking: true,
});
```

Arguments come back as a typed object matching the schema. Every time the model emits a tool call, we'd dispatch on `event.call.name` to run the matching function like so:

```ts
for await (const event of result.events) {
  if (event.type === "toolCall") {
    console.log(`▸ Tool: ${event.call.name}(${JSON.stringify(event.call.arguments)})`);
  }
}
```

The arguments come back as a parsed object matching our JSON Schema. We execute the function (call an API, query a database), then push `{ role: "tool", content: resultString }` back into history so the model can synthesize the final answer.

> Note: `captureThinking: true` is the option the first lesson in this chapter introduced. Without it, the model's `thinking` would land in the same `contentDelta` stream the tool call comes from and muddle the tool-call detection. The option keeps `thinking` on `thinkingDelta` events, so `contentDelta` only carries the model's prose around the tool call. The lesson's loop ignores `thinkingDelta` so the runner's OUTPUT panel only shows the tool calls.

> Note: tool support has to be enabled when loading the model. Set `modelConfig: { tools: true }` on the `loadModel()` call, otherwise the model will not understand how to use the tools array.

## Put it to the test

1. Define a `tools` array with one tool. Give it `name`, `description`, and `parameters` (JSON Schema).
2. Pass `tools` and `captureThinking: true` to `completion()` with a user question that requires the tool, stream `result.events`, and log `toolCall` events with the function name and arguments. `thinkingDelta` events arrive on the same stream and are ignored here so the runner's OUTPUT panel only shows the tool calls.
3. (Optional) After the loop, `await result.toolCalls` to get the full list and push tool results back as `{role: "tool"}` messages for the next turn.


# Clone a voice with Chatterbox TTS (/courses/qvac/en/text-to-speech/chatterbox)

Now that we can synthesize English speech with Supertonic, we're going to add voice cloning. Chatterbox takes a reference audio file and synthesizes speech in that voice.

Chatterbox is a two-file engine. A T3 GGUF handles the language side; an S3Gen GGUF handles the audio decoder. Both are available as registry constants. Load them together by setting `ttsEngine: "chatterbox"` and passing `s3genModelSrc` alongside the top-level `modelSrc`.

Chatterbox is a two-stage TTS pipeline: T3 (language → acoustic tokens), S3Gen (tokens → waveform). One `loadModel()` brings both, with `modelConfig` wiring the second stage. The split load would look like the following:

```ts
const modelId = await loadModel({
  modelSrc: TTS_T3_TURBO_EN_CHATTERBOX_Q8_0,
  modelConfig: {
    ttsEngine: "chatterbox",
    language: "en",
    s3genModelSrc: TTS_S3GEN_EN_CHATTERBOX.src,
    streamChunkTokens: 25,
    streamFirstChunkTokens: 10,
    cfmSteps: 1,
  },
});
```

Voice cloning is opt-in via `referenceAudioSrc`. Pass a path to a 16-bit mono WAV of the target speaker, and Chatterbox conditions the decoder on it. Without `referenceAudioSrc`, Chatterbox uses its bundled default voice. The reference clip should be a clean, single-speaker sample of 5 to 30 seconds.

`textToSpeech()` is fire-and-await. The call returns a 44.1 kHz mono `Int16Array`. Here's a simple example of synthesizing speech in the default voice:

```ts
const result = textToSpeech({
  modelId,
  text: "Hello, world.",
  inputType: "text",
  stream: false,
});
const audioBuffer = await result.buffer;
console.log(`▸ TTS complete. Total samples: ${audioBuffer.length}`);
```

The `textToSpeech` call matches Supertonic. The only differences are the loadModel `modelConfig` (Chatterbox needs `s3genModelSrc`) and the sample rate (24 kHz for Chatterbox, 44.1 kHz for Supertonic).

> Note: voice cloning inherits the *timbre* of the reference, not the words. The synthesized text is whatever you pass to `textToSpeech({ text })`. The reference only conditions the voice.

## Put it to the test

1. Call `loadModel` with `modelConfig.ttsEngine: "chatterbox"` and `s3genModelSrc: TTS_S3GEN_EN_CHATTERBOX.src`.
2. Optionally pass `referenceAudioSrc` to clone a voice from a WAV file on disk.
3. Call `textToSpeech({ modelId, text, inputType: "text", stream: false })` and `await result.buffer`. Log the sample count.


# Text-to-speech (/courses/qvac/en/text-to-speech)

Now that we can turn speech into text, we're going to turn text back into speech.

Text-to-speech takes a string and gives us back audio samples. Supertonic covers English with a fast single-file engine. Chatterbox layers voice cloning on top: feed it a reference WAV and it reads new text in that voice. Supertonic 3 covers 31 languages from a single multilingual GGUF. And `textToSpeech({ stream: true })` exposes `bufferStream` for incremental audio output, which is the shape you want for low-latency voice.

[Start Lesson 1 →](/courses/qvac/en/text-to-speech/tts-synthesize)

### All lessons in this chapter

1. [Synthesize speech from text](/courses/qvac/en/text-to-speech/tts-synthesize)
2. [Clone a voice with Chatterbox TTS](/courses/qvac/en/text-to-speech/chatterbox)
3. [Synthesize multilingual speech with Supertonic](/courses/qvac/en/text-to-speech/supertonic-multilingual)
4. [Stream TTS audio with bufferStream](/courses/qvac/en/text-to-speech/stream-tts-buffer)


# Stream TTS audio with bufferStream (/courses/qvac/en/text-to-speech/stream-tts-buffer)

Now that we can synthesize a full utterance, we're going to surface the audio as it synthesizes. For low-latency voice, the reader wants each chunk as soon as the engine produces it.

`textToSpeech({ stream: true })` returns `{ buffer, bufferStream, done }` instead of just `{ buffer }`. The samples live on `result.bufferStream` as an `AsyncGenerator<number>`.

Setting `stream: true` is what flips the result shape to `{ buffer, bufferStream, done }`. You would call it like so:

```ts
const result = textToSpeech({
  modelId,
  text: "Streaming chunks as the engine synthesizes them is the right latency for low-latency voice.",
  inputType: "text",
  stream: true,
});
```

Iterating `result.bufferStream` with `for await` is the canonical way to consume samples:

```ts
let totalSamples = 0;
for await (const sample of result.bufferStream) {
  void sample;
  totalSamples += 1;
}
console.log(`▸ Streamed ${totalSamples} samples`);
```

Each `number` is a single PCM sample. Iterating with `for await` yields samples in the order they were synthesized. `result.buffer` is empty when `stream: true`; the samples live on the generator, not the promise.

`result.done` is a `Promise<boolean>` that resolves when synthesis finishes. Await it from a separate branch if you need to know when the stream terminates.

> Note: this is a different shape from `textToSpeechStream`, which is a duplex session for piping tokens from a streaming LLM into TTS. Use `textToSpeech({ stream: true })` when you already have the full text and want a stream of audio chunks.

## Put it to the test

1. Call `textToSpeech({ modelId, text, inputType: "text", stream: true })`.
2. Iterate `result.bufferStream` with `for await (const sample of result.bufferStream)` and count the samples.
3. Log the total sample count. Don't await `result.buffer`; it stays empty when streaming.


# Synthesize multilingual speech with Supertonic (/courses/qvac/en/text-to-speech/supertonic-multilingual)

Now that we can synthesize English with Supertonic, we're going to add languages. Supertonic 3 covers 31 languages from a single multilingual GGUF.

The load is the same as the English Supertonic lesson. The only field that changes is `language`.

Loading the multilingual model is the same as the English lesson:

```ts
const modelId = await loadModel({
  modelSrc: TTS_MULTILINGUAL_SUPERTONIC3_Q8_0,
  modelConfig: {
    ttsEngine: "supertonic",
    language: "es",
    voice: "F1",
    ttsSpeed: 1.05,
    ttsNumInferenceSteps: 5,
  },
});
```

The voices are language-agnostic. `F1` is a female voice, `F2` a second female voice, `M1` and `M2` male voices. The same voice ID reads out the text in whatever language `modelConfig.language` selects. Switching from `"es"` to `"fr"` gives you the same voice reading French.

Same synth-call shape, with `language: "es"` carried into the options. Consider the following synth call:

```ts
const result = textToSpeech({
  modelId,
  text: "Hola mundo. Esta es una demostración de síntesis de voz con Supertonic en español.",
  inputType: "text",
  stream: false,
});
const audioBuffer = await result.buffer;
console.log(`▸ TTS complete. Total samples: ${audioBuffer.length}`);
```

> Note: language codes that aren't in the Supertonic model's vocabulary fall back to English. If you need a specific dialect or regional accent, check the upstream Supertonic docs for the supported language list.

## Put it to the test

1. Call `loadModel` with `modelConfig.ttsEngine: "supertonic"`, a non-English `language`, and a `voice`.
2. Call `textToSpeech({ modelId, text, inputType: "text", stream: false })` and `await result.buffer`.
3. Log the sample count.


# Synthesize speech from text (/courses/qvac/en/text-to-speech/tts-synthesize)

We're starting a new chapter on text-to-speech, and we're going to turn text into audio.

Text-to-speech runs the Supertonic engine on the supplied text and hands us back raw audio samples. The result is an `Int16Array` (one sample per array slot), 44100 Hz by default. We write it to disk as a WAV with a 44-byte header prepended.

Supertonic needs three knobs at first load: `ttsEngine` (backend), `language`, `voice` (F1/F2/M1/M2). All three are required. The first load with them set would look like the following:

```ts
const modelId = await loadModel({
  modelSrc: TTS_MULTILINGUAL_SUPERTONIC3_Q8_0,
  modelConfig: {
    ttsEngine: "supertonic",
    language: "en",
    voice: "F1",
  },
});
```

`textToSpeech()` is fire-and-await. Returns a 44.1 kHz mono `Int16Array`, the unit a WAV writer expects as follows:

```ts
const result = textToSpeech({
  modelId,
  text: "Hello, world.",
  inputType: "text",
  stream: false,
});
const audioBuffer = await result.buffer;
console.log(`▸ TTS complete. Total samples: ${audioBuffer.length}`);
```

To save it as a `.wav` file, we prepend a 44-byte header (see the supertonic example's `createWav` helper). The samples themselves are `Int16Array` values at the engine's sample rate.

> Note: the `ttsSpeed` and `ttsNumInferenceSteps` fields in `modelConfig` trade latency against quality. `ttsSpeed: 1.05` and `ttsNumInferenceSteps: 5` are the defaults in the SDK and a good starting point.

## Put it to the test

1. Call `loadModel` with `modelConfig.ttsEngine: "supertonic"`, a `language`, and a `voice`.
2. Call `textToSpeech({ modelId, text, inputType: "text", stream: false })` and `await result.buffer`.
3. Log the buffer length.


# Transcription (/courses/qvac/en/transcription)

Now that we've done text, image, and video, we're going to add audio. First, speech-to-text.

Whisper takes audio in and returns the spoken words as text. The result carries timestamps per segment, so we can render captions, search inside audio, or chunk the transcript by speaker turn. Parakeet TDT is the multilingual alternative: 25+ languages, no language code, no VAD. Parakeet CTC trades the multilingual coverage for lower latency on English-only audio. Sortformer attaches speaker IDs to segments, so a multi-speaker recording becomes a labelled transcript.

[Start Lesson 1 →](/courses/qvac/en/transcription/transcribe-file)

### All lessons in this chapter

1. [Transcribe an audio file](/courses/qvac/en/transcription/transcribe-file)
2. [Stream transcription from microphone](/courses/qvac/en/transcription/mic-transcribe)
3. [Transcribe an audio file with a Whisper prompt](/courses/qvac/en/transcription/whisper-prompt)
4. [Stream transcripts with VAD and end-of-turn events](/courses/qvac/en/transcription/whisper-vad)
5. [Transcribe multilingual audio with Parakeet TDT](/courses/qvac/en/transcription/parakeet-tdt)
6. [Transcribe English audio with Parakeet CTC](/courses/qvac/en/transcription/parakeet-ctc)
7. [Diarize speakers with Parakeet Sortformer](/courses/qvac/en/transcription/parakeet-sortformer)


# Stream transcription from microphone (/courses/qvac/en/transcription/mic-transcribe)

Now that we can transcribe files, we're going to stream transcription live.

Whisper doesn't have to wait for the user to finish talking. We feed it a live audio stream and it returns segments as soon as it has enough context to commit to a transcription. Each segment carries timestamps, so the UI can render captions in sync with the audio.

The streaming path requires a VAD model. The VAD decides when the speaker has paused long enough to commit a segment, so the engine refuses to open a streaming session without one. Pair `WHISPER_TINY` with `VAD_SILERO_5_1_2` in `loadModel` modelConfig:

```ts
const modelId = await loadModel({
  modelSrc: WHISPER_TINY,
  modelConfig: {
    vadModelSrc: VAD_SILERO_5_1_2,
    audio_format: "f32le",
    language: "en",
  },
});
```

The streaming API is the duplex `transcribeStream()` call. It returns a session object with two surfaces: `session.write(buffer)` to push audio in, and `for await (const segment of session)` to pull segments out:

```ts
const session = await transcribeStream({
  modelId,
  metadata: true,
});
```

Audio capture is a child ffmpeg process. The `startMicrophone()` helper spawns the system ffmpeg with the right `-i` args for the current platform (avfoundation on macOS, pulse on Linux, dshow on Windows) and pipes 16 kHz mono f32le PCM to stdout. Each chunk on stdout is a frame we feed into the session:

```ts
const ffmpeg = startMicrophone({ sampleRate: 16000, format: "f32le" });

ffmpeg.stdout.on("data", (chunk: Buffer) => {
  try { session.write(chunk); } catch {}
});
```

A startup check on `ffmpeg -version` makes the failure mode obvious if ffmpeg isn't on `PATH`. The try/catch around `session.write` ignores the abort when the worker is being torn down at shutdown.

The consuming side is a `for await` over the session itself. With `metadata: true`, each iteration yields a `TranscribeSegment` with `text`, `startMs`, `endMs`, `append`, and `id`. The `append` flag tells the UI whether to overwrite the previous caption or extend it. Render captions like so:

```ts
for await (const segment of session) {
  const start = (segment.startMs / 1000).toFixed(2);
  const end = (segment.endMs / 1000).toFixed(2);
  console.log(`[${start}s → ${end}s] ${segment.text}`);
}
```

Whisper's voice-activity detection commits to a segment when the speaker pauses or the buffer hits its maximum length. The `append` field on each segment tells us whether to overwrite the previous caption or append to it.

A `cleanup()` handler kills the ffmpeg child, ends the session, and unloads the model. Wire it to `SIGINT` and `SIGTERM` so Ctrl+C exits cleanly:

```ts
async function cleanup() {
  if (shuttingDown) return;
  shuttingDown = true;
  ffmpeg.kill();
  try { session.end(); } catch {}
  await unloadModel({ modelId }).catch(() => {});
  process.exit(0);
}

process.on("SIGINT", () => void cleanup());
process.on("SIGTERM", () => void cleanup());
```

The SDK installs its own `SIGINT` / `SIGTERM` handler that aborts in-flight RPC streams on shutdown. The abort rejects any pending `session.write()` with a `WorkerShutdownError`, and the bare-rpc socket emits a `CHANNEL_CLOSED` error. The `process.on("uncaughtException", ...)` filter ignores the shutdown noise and re-throws anything else:

```ts
process.on("uncaughtException", (err) => {
  if (err instanceof WorkerShutdownError) return;
  if (err?.code === "CHANNEL_CLOSED") return;
  throw err;
});
```

> Note: the running buffer has a maximum length. Long pauses or slow speakers may produce fewer segments than you'd expect. Tune the model's `vad_params.max_speech_duration_s` if you need more granularity.

## Put it to the test

1. Open `const session = await transcribeStream({ modelId, metadata: true })`. The `await` is required; iterating a Promise with `for await` is a syntax error.
2. Gate on `ffmpeg` being on `PATH` at startup. Define the mic helper (`getAudioInputArgs`, `startMicrophone`) above `main`.
3. Pipe `startMicrophone({ sampleRate: 16000, format: "f32le" }).stdout` into `session.write(chunk)` in a background task. Wrap the write in try/catch.
4. Iterate the session with `for await (const segment of session)` and log each segment's `[start → end]` text as it is finalized.
5. Wire `cleanup()` to `SIGINT` and `SIGTERM`, and add the `WorkerShutdownError` / `CHANNEL_CLOSED` filter to `uncaughtException` so the SDK's shutdown abort doesn't dump a stack.


# Transcribe English audio with Parakeet CTC (/courses/qvac/en/transcription/parakeet-ctc)

When you know the audio is English and latency matters, Parakeet CTC is faster than TDT. The trade is that CTC is English-only; Parakeet CTC rejects non-English audio rather than mistranslating it.

CTC is the English-only sibling of TDT. The model file is smaller and the inference path is shorter, so `transcribe` returns a few hundred milliseconds sooner.

The editor prefills the model load (Parakeet CTC with `modelType: "parakeet-transcription"`). The new piece is the transcribe call.

CTC is the English-only TDT variant: same call shape, same options, just a different `modelId`. The transcribe call we'd make would look like:

```ts
const text = await transcribe({
  modelId,
  audioChunk: "./examples/qvac/transcription/input/sample-16khz.wav",
});
console.log(text);
```

The call shape and the constants are the only differences from TDT.

> Note: CTC is also the right Parakeet variant for the `transcribeStream` duplex session when the audio is English. The next lesson covers diarization with Sortformer.

## Put it to the test

1. Call `transcribe({ modelId, audioChunk: "./examples/qvac/transcription/input/sample-16khz.wav" })`, await the result, and `console.log` the returned text.


# Diarize speakers with Parakeet Sortformer (/courses/qvac/en/transcription/parakeet-sortformer)

When a single audio file has more than one speaker, we want a transcript that says who said what. Sortformer diarizes the audio into speaker-attributed segments, then TDT transcribes each segment into text.

The pipeline runs in two passes:

* Sortformer returns a text block where each line is `Speaker N: <start>s - <end>s`.
* TDT transcribes each segment and we attach the text to the speaker metadata.

Each `Speaker N: <start>s - <end>s` line becomes a `{ speaker, start, end }` object. The Sortformer-to-TDT chain starts with the diarization pass. The first pass would look like the following:

```ts
const sfModelId = await loadModel({
  modelSrc: PARAKEET_SORTFORMER_4SPK_V2_1_Q8_0,
  modelType: "parakeet-transcription",
});

const diarization = await transcribe({
  modelId: sfModelId,
  audioChunk: audioFilePath,
});
await unloadModel({ modelId: sfModelId });

const segments = diarization
  .split("\n")
  .map((line) => line.match(/Speaker (\d+): ([\d.]+)s - ([\d.]+)s/))
  .filter((m): m is RegExpMatchArray => m !== null)
  .map((m) => ({ speaker: +m[1]!, start: +m[2]!, end: +m[3]! }))
  .sort((a, b) => a.start - b.start);
```

Second pass: TDT takes over. The slicing reads the source WAV once and writes a new WAV per `start`/`end` range; `transcribe` consumes each slice:

```ts
const tdtModelId = await loadModel({
  modelSrc: PARAKEET_TDT_0_6B_V3_Q8_0,
  modelType: "parakeet-transcription",
});

const pcm = readPcm(audioFilePath);
const sliceDir = join(tmpdir(), `qvac-diarize-${Date.now()}`);
mkdirSync(sliceDir, { recursive: true });

const results: { speaker: number; start: number; end: number; text: string }[] = [];
for (let i = 0; i < segments.length; i++) {
  const seg = segments[i]!;
  const slicePath = join(sliceDir, `seg-${i}.wav`);

  if (!writeWavSlice(pcm, seg.start, seg.end, slicePath)) {
    results.push({ ...seg, text: "[No speech detected]" });
    continue;
  }

  const text = await transcribe({
    modelId: tdtModelId,
    audioChunk: slicePath,
  });
  results.push({ ...seg, text: text.trim() || "[No speech detected]" });
}
await unloadModel({ modelId: tdtModelId });
```

One log line per diarized result, with the speaker label and time range:

```ts
for (const r of results) {
  console.log(`Speaker ${r.speaker} (${r.start.toFixed(2)}s - ${r.end.toFixed(2)}s): ${r.text}`);
}
```

`readPcm` and `writeWavSlice` (prefilled in the scaffold) handle the WAV byte-level work: skipping the 44-byte header and rebuilding a 16 kHz / 16-bit / mono header in front of each PCM range. The slicing runs once per diarized segment, so the inference cost is `N × TDT_inference` rather than one shot on the full file.

> Note: Sortformer v2.1 supports up to 4 speakers. For more speakers, split the audio or run the diarization in overlapping windows.

## Put it to the test

1. Load `PARAKEET_SORTFORMER_4SPK_V2_1_Q8_0` with `modelType: "parakeet-transcription"`, call `await transcribe({ modelId, audioChunk: audioFilePath })`, then `unloadModel`. Parse the output lines into `{ speaker, start, end }` objects.
2. Load `PARAKEET_TDT_0_6B_V3_Q8_0`. For each segment, call `writeWavSlice(pcm, seg.start, seg.end, slicePath)` and then `await transcribe({ modelId: tdtModelId, audioChunk: slicePath })`. Push the text onto the segment.
3. Console.log one line per result: `Speaker N (start - end): text`.


# Transcribe multilingual audio with Parakeet TDT (/courses/qvac/en/transcription/parakeet-tdt)

Whisper handles English well and several other languages adequately. For multilingual audio where accuracy matters, we switch to Parakeet TDT.

Parakeet TDT is a single GGUF that handles 25+ languages out of the box. The same model file transcribes French, German, Spanish, Mandarin, and more without a language code. The addon auto-detects TDT vs CTC vs Sortformer from the GGUF's internal metadata, so the load is the same as Whisper's.

`modelType: "parakeet-transcription"` tells the engine which Parakeet family to load: TDT (default), CTC (smaller), Sortformer (diarization). The load with that flag set would look like the following:

```ts
const modelId = await loadModel({
  modelSrc: PARAKEET_TDT_0_6B_V3_Q8_0,
  modelType: "parakeet-transcription",
});
```

TDT returns a single string per file, no per-segment data. Note that without overrides the call is minimal. For example:

```ts
const text = await transcribe({
  modelId,
  audioChunk: "./examples/qvac/transcription/input/sample-16khz.wav",
});
console.log(text);
```

No `modelConfig.language`. No prompt. No VAD. The only flag you need is `modelType: "parakeet-transcription"` so the SDK routes the call to the right addon.

> Note: Parakeet handles VAD internally, so no separate VAD model is needed. If you need explicit end-of-utterance detection for conversation, load `PARAKEET_EOU_120M_V1_Q8_0` alongside and pass `parakeetStreamingConfig` to `transcribeStream`.

## Put it to the test

1. Call `loadModel({ modelSrc: PARAKEET_TDT_0_6B_V3_Q8_0, modelType: "parakeet-transcription" })` and `transcribe({ modelId, audioChunk: "./examples/qvac/transcription/input/sample-16khz.wav" })`. No `language` or `prompt` field.
2. `console.log` the returned text.


# Transcribe an audio file (/courses/qvac/en/transcription/transcribe-file)

We're starting a new chapter on transcription, and we're going to turn a WAV file into text.

Whisper takes a WAV file and returns the spoken text. The SDK wraps it in one `transcribe()` call. The result is an array of segments with start and end timestamps, so we can render captions, search inside audio, or chunk the transcript by speaker turn.

`WHISPER_TINY` is the smallest Whisper model in the SDK.

Setting `metadata: true` is what brings back the per-segment timestamps. You would call it like so:

```ts
const segments = await transcribe({
  modelId,
  audioChunk: "./examples/qvac/transcription/input/sample-16khz.wav",
  metadata: true,
});

for (const segment of segments) {
  const start = (segment.startMs / 1000).toFixed(2);
  const end = (segment.endMs / 1000).toFixed(2);
  console.log(`[${start}s → ${end}s] ${segment.text}`);
}
```

The `metadata: true` flag keeps the per-segment timing fields. We set it to `false` if we only need the joined text.

> Note: the audio must be a WAV file at 16 kHz mono PCM. Other formats work with extra conversion, but the SDK takes the most common shape directly.

## Put it to the test

1. Call `transcribe({ modelId, audioChunk: <path>, metadata: true })`, await the segments, and loop through them logging each with its `[start → end]` timestamp.


# Transcribe an audio file with a Whisper prompt (/courses/qvac/en/transcription/whisper-prompt)

Now that we can transcribe a file, we're going to steer the decoder with a prompt.

Whisper takes a short prompt string before it commits to a transcription. The prompt isn't a system instruction. It's a list of words and phrases the decoder should prefer. Names, jargon, and formatting you put in the prompt appear correctly in the output, even when the audio mumbles them.

Whisper's decoder bias comes from the `prompt` option. You would call it like so:

```ts
const text = await transcribe({
  modelId,
  audioChunk: "./examples/qvac/transcription/input/sample-16khz.wav",
  prompt: "This is a test recording with clear speech and proper punctuation.",
});

console.log("▸ Transcription result:");
console.log(text);
```

`prompt` is a string. Whisper tokenizes it and conditions the first few decoder steps on it. Any word in the prompt gets a head start. Any word that conflicts with the audio gets overridden.

> Note: the prompt doesn't change the model's language. Set `modelConfig.language: "en"` (or whatever target language you need) at load time. The prompt only steers vocabulary.

## Put it to the test

1. Call `transcribe({ modelId, audioChunk: <path>, prompt: "..." })`, await the result, and log `"▸ Transcription result:"` followed by the text.


# Stream transcripts with VAD and end-of-turn events (/courses/qvac/en/transcription/whisper-vad)

Now that we can stream transcripts from a microphone, we're going to surface the VAD and end-of-turn events the engine already tracks.

`transcribeStream` returns a duplex session. Audio goes in via `session.write(chunk)`, and the session yields a discriminated union of events: text chunks, voice-activity state, and turn boundaries. A real-time voice assistant builds on top of those three event types.

Setting `emitVadEvents: true` and an `endOfTurnSilenceMs` of 800 surfaces VAD and end-of-turn events. The session open looks like this:

```ts
const session = await transcribeStream({
  modelId,
  emitVadEvents: true,
  endOfTurnSilenceMs: 800,
});

void (async () => {
  for await (const chunk of audioStream()) {
    session.write(chunk);
  }
})();
```

Three event types on the same duplex stream, one `for await` + `switch` is the canonical consume pattern:

```ts
for await (const event of session) {
  switch (event.type) {
    case "text":
      console.log(`> ${event.text.trim()}`);
      break;
    case "vad":
      console.log(`▸ [vad] speaking=${event.speaking} probability=${event.probability.toFixed(2)}`);
      break;
    case "endOfTurn":
      console.log(`▸ [endOfTurn] silence ${event.silenceDurationMs}ms\n`);
      break;
  }
}
```

The `vad` event fires while the speaker is talking. The `endOfTurn` event fires after the speaker pauses for `endOfTurnSilenceMs` milliseconds. That silence window is the conversation-equivalent of "they're done; now I can answer."

> Note: `endOfTurn` measures silence from the VAD, not from a parakeet EOU token. Pair the Whisper model with `VAD_SILERO_5_1_2` in `modelConfig.vadModelSrc` so the silence window is measured accurately.

## Put it to the test

1. Open `transcribeStream({ modelId, emitVadEvents: true, endOfTurnSilenceMs: 800 })`, pipe the mic stream's Float32Array frames into `session.write(chunk)` in a background task, then iterate `for await (const event of session)`.
2. Switch on `event.type`. Console.log `text` events as transcript lines, `vad` events as VAD ticks, and `endOfTurn` events as turn boundaries.


# Translation (/courses/qvac/en/translation)

Now that we've covered speech, we're going to look at translation.

Bergamot is the on-device translation engine. Each model covers one language pair, and the result is the translated string. Loaded and called the same way as an LLM, just with a different `modelType` flag.

[Start Lesson 1 →](/courses/qvac/en/translation/translate-text)

### All lessons in this chapter

1. [Translate text between languages](/courses/qvac/en/translation/translate-text)


# Translate text between languages (/courses/qvac/en/translation/translate-text)

We're starting a new chapter on translation, and we're going to translate a string between two languages.

Bergamot is the on-device translation engine. Loading it looks like loading an LLM: pass a model constant, hand the engine a string, and get a translation back. Bergamot models are tiny (single-language pairs, around 30 MB), so they're cheap to keep in memory alongside other models.

Bergamot is a tiny neural translation model from Mozilla, single-language pairs around 30 MB each. The `engine: "Bergamot"` flag picks the right backend. The Bergamot load would look like the following:

```ts
const modelId = await loadModel({
  modelSrc: BERGAMOT_EN_FR,
  modelConfig: {
    engine: "Bergamot",
    from: "en",
    to: "fr",
    beamsize: 1,
  },
});
```

The inference step picks up where `loadModel` left off: hand the `modelId` to `translate({ text })`, await `result.text`, get the translated string. The two new flags are `modelType` (which addon to route the call to) and `stream: false` (sync, no duplex session):

```ts
const result = translate({
  modelId,
  text: "Hello, world.",
  modelType: "nmtcpp-translation",
  stream: false,
});
const translatedText = await result.text;
console.log(`EN -> FR: "${translatedText}"`);
```

The `modelType: "nmtcpp-translation"` flag tells the SDK which addon to route the call through. Without it, the SDK can't pick the right engine for translation vs transcription vs text generation.

> Note: each translation model is a single language pair. For EN to DE you'd load `BERGAMOT_EN_DE` instead. The SDK doesn't translate between non-direct pairs without explicit pivot configuration.

## Put it to the test

1. Call `loadModel` with `modelSrc: BERGAMOT_EN_FR` and `modelConfig.engine: "Bergamot"` (plus `from: "en"`, `to: "fr"`).
2. Call `translate({ modelId, text, modelType: "nmtcpp-translation", stream: false })`, `await result.text`, and log the translated string.


# Build a video from a still image (/courses/qvac/en/video-generation/img2vid)

The previous lesson built a video from text. This one builds one from a still image.

`txt2vid` builds the clip from text. `img2vid` starts from a still and animates it forward. The first frame becomes the input; the rest of the clip is generated from the prompt.

The still comes in via `init_image`. `strength` controls how much motion happens, and at `0` we get a frozen frame, and at `1` we lose all resemblance to the source. Most clips sit somewhere around `0.6` to `0.85`.

For img2vid, the model needs a vision encoder alongside the diffusion model. `WAN2_1_I2V_14B_Q4_K_M` is the I2V model, with `CLIP_VISION_H` as the vision encoder:

The I2V model needs a vision tower as a side file (CLIP-style encoder), in the same slot T2V uses for T5-XXL. You would load it like so:

```ts
const videoId = await loadModel({
  modelSrc: WAN2_1_I2V_14B_Q4_K_M,
  modelType: "sdcpp-generation",
  modelConfig: {
    mode: "video",
    t5XxlModelSrc: UMT5_XXL_FP16,
    vaeModelSrc: WAN_2_1_COMFYUI_REPACKAGED_VAE,
    clipVisionModelSrc: CLIP_VISION_H,
  },
});
```

`init_image` is the first frame the diffusion model starts denoising from; the vision tower encodes it, the diffusion model adds motion, and the VAE decodes:

```ts
const initImage = new Uint8Array(fs.readFileSync("./examples/qvac/video-generation/input/portrait.png"));
```

`video({ mode: "img2vid", ... })` kicks off the animation; awaiting `outputs` resolves to the encoded video bytes. Note that `init_image` is the first frame and `strength` is how much it can change. The still's pixel dimensions must match `width` and `height` exactly: the I2V model rejects the call with `init_image dimensions WxH do not match video dimensions WxH` otherwise, so size the source image to your target resolution before passing it. Here's an example that reads a portrait PNG, animates it forward, and writes the first clip to disk:

```ts
const result = video({
  modelId: videoId,
  mode: "img2vid",
  prompt: "the subject slowly turns and smiles, soft natural lighting, cinematic",
  init_image: initImage,
  strength: 0.85,
  width: 480,
  height: 832,
  video_frames: 17,
  fps: 16,
});
const outputs = await result.outputs;
const firstClip = outputs[0];
if (!firstClip) throw new Error("No video returned from video()");
fs.writeFileSync("../../apps/desktop/output/video-gen/portrait.mp4", firstClip);
console.log(`Generated ${outputs.length} video`);
```

> Note: `clipVisionModelSrc` is the vision encoder that comes alongside the I2V model. Pairing it with the wrong model (or omitting it) means the first frame isn't in the input and the output has nothing to anchor on, so the result drifts frame-to-frame.

## Put it to the test

1. Call `loadModel` with `modelSrc: WAN2_1_I2V_14B_Q4_K_M`, `modelType: "sdcpp-generation"`, and the T5-XXL / VAE / CLIP vision sources in `modelConfig`.
2. Read a portrait PNG into a `Uint8Array` with `fs.readFileSync`.
3. Call `video({ modelId, mode: "img2vid", prompt, init_image, strength: 0.85, width, height, video_frames, fps })`, await the outputs, write `outputs[0]` to a `.mp4` file, and console.log the count.


# Video generation (/courses/qvac/en/video-generation)

Now that we can make images, we're going to make them move.

Video diffusion builds on the same engine as image diffusion, with a few extra files (the T5-XXL text encoder and a VAE tuned for frames) and a `mode: "video"` flag. The result is an AVI file of frames at the rate we asked for.

[Start Lesson 1 →](/courses/qvac/en/video-generation/load-video-model)

### All lessons in this chapter

1. [Load a video model](/courses/qvac/en/video-generation/load-video-model)
2. [Build a video from text](/courses/qvac/en/video-generation/txt2vid)
3. [Build a video from a still image](/courses/qvac/en/video-generation/img2vid)


# Load a video model (/courses/qvac/en/video-generation/load-video-model)

We're starting a new chapter on video generation, and we're going to load a model that turns prompts into short clips.

Text-to-image takes three files: the diffusion model, a text encoder, and a VAE. Video works the same way. We use the same files, but with `modelConfig.mode: "video"` so the engine knows to emit frames instead of pixels.

Wan 2.1 T2V (`WAN2_1_T2V_1_3B_FP16`) is the text-to-video model in the SDK. The text encoder is `UMT5_XXL_FP16`. The VAE is `WAN_2_1_COMFYUI_REPACKAGED_VAE`. We load all three in one call.

The first call takes the longest, since the three GGUF files together run hundreds of megabytes. Subsequent `video()` calls reuse the loaded model.

`mode: "video"` switches the diffusion engine from image generation to video generation (a sequence of frames). Note that without `mode: "video"`, the engine tries to render a single image instead of a clip. The `loadModel` call looks like:

```ts
const videoId = await loadModel({
  modelSrc: WAN2_1_T2V_1_3B_FP16,
  modelType: "sdcpp-generation",
  modelConfig: {
    mode: "video",
    t5XxlModelSrc: UMT5_XXL_FP16,
    vaeModelSrc: WAN_2_1_COMFYUI_REPACKAGED_VAE,
  },
});
console.log("videoId:", videoId);
```

The next lesson writes a real `video()` call against this `videoId`.

> Note: `mode: "video"` is required. Without it, the engine tries to render a single image instead of a clip.

## Put it to the test

1. Call `loadModel` with `modelType: "sdcpp-generation"`, `modelConfig.mode: "video"`, and the T5-XXL and VAE src fields.
2. Log the resulting `videoId`.


# Build a video from text (/courses/qvac/en/video-generation/txt2vid)

Now that we have a video model in memory, let's make our first clip.

`video()` returns the same return type as `diffusion()`: `outputs` is a `Promise<Uint8Array[]>`, and outputs is `AVI` bytes instead of `PNG`. We save it to disk with the same `fs.writeFileSync` pattern.

Frame count and framerate together determine video length. `video_frames: 17, fps: 16` is roughly one second of video. We bump `video_frames` and we get longer clips at proportionally higher inference cost.

Wan 2.1 T2V is three files: T5-XXL (text encoder), VAE (decoder), diffusion model. The `loadModel()` call mirrors I2V's, but T5-XXL replaces the vision tower:

```ts
const videoId = await loadModel({
  modelSrc: WAN2_1_T2V_1_3B_FP16,
  modelType: "sdcpp-generation",
  modelConfig: {
    mode: "video",
    t5XxlModelSrc: UMT5_XXL_FP16,
    vaeModelSrc: WAN_2_1_COMFYUI_REPACKAGED_VAE,
  },
});
```

The txt2vid `video({ ... })` matches img2vid's, with `mode: "txt2vid"` and no `init_image`:

```ts
const result = video({
  modelId: videoId,
  mode: "txt2vid",
  prompt: "a colorful bird flapping its wings",
  width: 480,
  height: 832,
  video_frames: 17,
  fps: 16,
});
```

Once `await result.outputs` resolves, write `outputs[0]` to a `.avi` file with `fs.writeFileSync`, the extension picks the container. Note that the `if (!firstClip)` guard throws if the result is empty as follows:

```ts
const outputs = await result.outputs;
const firstClip = outputs[0];
if (!firstClip) throw new Error("No video returned from video()");
fs.writeFileSync("../../apps/desktop/output/video-gen/bird.avi", firstClip);
console.log(`Generated ${outputs.length} video`);
```

Open the AVI in any player. That's our one-second text-prompted clip. The next lesson animates a still image instead of starting from text.

> Note: `width` and `height` on video calls are smaller than image calls. 480×832 keeps the per-frame memory budget reasonable on a 1.3B video model.

## Put it to the test

1. Call `loadModel` with `modelSrc: WAN2_1_T2V_1_3B_FP16`, `modelType: "sdcpp-generation"`, and `modelConfig.mode: "video"` (plus `t5XxlModelSrc` and `vaeModelSrc`).
2. Call `video({ modelId, mode: "txt2vid", prompt, width, height, video_frames: 17, fps: 16 })` and `await result.outputs`.
3. Write `outputs[0]` to a `.avi` file and console.log the count.


# VLA (/courses/qvac/en/vla)

VLA (vision-language-action) models are the model class for robot control.

VLA (vision-language-action) models take a camera frame and a task description and produce an action chunk. The chunk is the next `chunkSize` timesteps of an `actionDim`-dimensional action vector for the robot's actuators. The models are trained on a wide range of manipulation tasks. The SDK includes two families: SmolVLA (small, fast) and π₀.₅ (larger, more capable). Both share the same `vla()` call surface.

[Start Lesson 1 →](/courses/qvac/en/vla/vla-smolvla)

### All lessons in this chapter

1. [Run a SmolVLA action inference](/courses/qvac/en/vla/vla-smolvla)
2. [Run a pi05 action inference](/courses/qvac/en/vla/vla-pi05)


# Run a pi05 action inference (/courses/qvac/en/vla/vla-pi05)

SmolVLA was the previous lesson's model. This lesson runs π₀.₅ from Physical Intelligence. The API surface is the same `vla()` call, but two hparams drive a different input shape.

`loadModel` pulls π₀.₅ into memory, and `vlaHparams` reads the input dimensions (`visionImageSize`, `numCameras`, `tokenizerMaxLength`, `chunkSize`, `maxActionDim`) for the next step:

```ts
const modelId = await loadModel({
  modelSrc: PI05_BASE_Q_AGGRESSIVE,
  modelType: "ggml-vla",
  modelConfig: { backend: "cpu" },
});

const { hparams } = await vlaHparams({ modelId });
const size = hparams.visionImageSize;
const numCameras = hparams.numCameras ?? 3;
```

The four input buffers π₀.₅ expects: `numCameras` synthetic camera frames via `vlaPreprocessImage`, a BOS-only `tokens` and `mask` (the model starts decoding from a single BOS token), an empty state buffer (the state is tokenised into the prompt in this model, so the buffer is ignored), and a `chunkSize × maxActionDim` zero-filled noise buffer the diffusion step denoises:

```ts
const dummyPixels = new Uint8Array(size * size * 3).fill(128);
const images = Array.from({ length: numCameras }, () =>
  vlaPreprocessImage(dummyPixels, size, size, { size }),
);

const tokens = new Int32Array(hparams.tokenizerMaxLength);
const mask = new Uint8Array(hparams.tokenizerMaxLength);
tokens[0] = 1;
mask[0] = 1;

const state = new Float32Array(0);
const noise = new Float32Array(hparams.chunkSize * hparams.maxActionDim);
```

`vla()` takes the model id and prebuilt inputs and returns the same four fields the SmolVLA lesson destructured. `stats` uses the same snake\_case `*_ms` shape on both models, so the per-stage timings are interchangeable between lessons.

```ts
const { actions, actionDim, chunkSize, stats } = await vla({
  modelId,
  images,
  imgWidth: size,
  imgHeight: size,
  state,
  tokens,
  mask,
  noise,
});

console.log(`▸ Got ${chunkSize} action steps of dim ${actionDim}.`);
console.log(Array.from(actions.subarray(0, actionDim)));
if (stats) {
  console.log(
    `▸ Timing: vision=${stats.vision_ms?.toFixed(0)}ms ` +
      `prefill=${stats.prefill_total_ms?.toFixed(0)}ms ` +
      `ode=${stats.ode_ms?.toFixed(0)}ms ` +
      `total=${stats.total_ms?.toFixed(0)}ms`,
  );
}
```

The four `stats` fields are reported in pipeline order: vision encoder, language-model prefill, ODE solver, then wall-clock total. The optional chaining handles a stage that wasn't run. If a model short-circuits a phase, the field is `undefined` rather than missing from the object.

> Note: π₀.₅ is a larger model (\~3.9 GB) than SmolVLA (\~1.9 GB) and the inference time reflects that. On a desktop CPU, expect several seconds per call. The `backend: "cpu"` flag is the default and the only supported target in this SDK release.

## Put it to the test

1. Call `loadModel({ modelSrc: PI05_BASE_Q_AGGRESSIVE, modelType: "ggml-vla", modelConfig: { backend: "cpu" } })` and `vlaHparams({ modelId })`. Read `hparams.visionImageSize`, `hparams.numCameras`, `hparams.tokenizerMaxLength`, `hparams.chunkSize`, and `hparams.maxActionDim`.
2. Build `numCameras` `vlaPreprocessImage` frames, a BOS-only `tokens` and `mask`, an empty `new Float32Array(0)` state, and a zero-filled noise buffer of size `chunkSize * maxActionDim`.
3. Call `await vla({ modelId, images, imgWidth, imgHeight, state, tokens, mask, noise })` and console.log the action chunk and per-stage `stats`.


# Run a SmolVLA action inference (/courses/qvac/en/vla/vla-smolvla)

VLA (vision-language-action) models are the model class for robot control.

VLA (vision-language-action) models take a camera frame and a task description and produce an action chunk the robot should take. The training data covers real-world manipulation tasks: grasping, picking, placing. The SDK includes two families: SmolVLA (a small, fast model from Hugging Face) and π₀.₅ (a larger model from Physical Intelligence). They share the same API surface but the input shapes differ.

The first call loads SmolVLA into memory. SmolVLA is the smaller of the two VLAs in the SDK. The first load would look like the following:

```ts
const modelId = await loadModel({
  modelSrc: SMOLVLA_LIBERO_VISION_Q8,
  modelType: "ggml-vla",
  modelConfig: { backend: "cpu" },
});
```

Each VLA model has different input shapes; reading them up front is the canonical pre-call move:

```ts
const { hparams } = await vlaHparams({ modelId });
```

The hparams tell you the input shapes. For SmolVLA, you need two camera frames of `visionImageSize × visionImageSize × 3`, a state vector padded to `maxStateDim`, tokens of length `tokenizerMaxLength`, and a noise buffer of length `chunkSize × maxActionDim`.

SmolVLA wants state as a separate field, unlike π₀.₅ which tokenises it. The synthetic inputs sized to hparams would look like:

```ts
const size = hparams.visionImageSize;
const dummyPixels = new Uint8Array(size * size * 3).fill(128);
const front = vlaPreprocessImage(dummyPixels, size, size, { size });
const wrist = vlaPreprocessImage(dummyPixels, size, size, { size });

const tokens = new Int32Array(hparams.tokenizerMaxLength);
const mask = new Uint8Array(hparams.tokenizerMaxLength);
tokens[0] = 1;
mask[0] = 1;

const state = vlaPadState([0, 0, 0, 0, 0, 0], hparams.maxStateDim);
const noise = new Float32Array(hparams.chunkSize * hparams.maxActionDim);
```

`vla()` takes the model id and the prebuilt inputs, returns an action chunk plus per-stage timings:

```ts
const { actions, actionDim, chunkSize, stats } = await vla({
  modelId,
  images: [front, wrist],
  imgWidth: size,
  imgHeight: size,
  state,
  tokens,
  mask,
  noise,
});

console.log(`▸ Got ${chunkSize} action steps of dim ${actionDim}.`);
console.log(`▸ Timing: vision=${stats.vision_ms}ms prefill=${stats.prefill_total_ms}ms ode=${stats.ode_ms}ms total=${stats.total_ms}ms`);
```

The `stats` object has per-stage timings: vision encoder, language model, ODE solver, and the wall-clock total. They're useful for spotting which stage is the bottleneck on a given machine.

> Note: `vlaPadState` is the helper that pads a short state vector to the model's `maxStateDim`. For SmolVLA, the state is six floats (end-effector pose). For π₀.₅ the state is tokenised into the prompt instead, the state buffer is ignored.

## Put it to the test

1. Call `loadModel` with `modelSrc: SMOLVLA_LIBERO_VISION_Q8`, `modelType: "ggml-vla"`, and `modelConfig: { backend: "cpu" }`.
2. Call `vlaHparams({ modelId })` and read `hparams.visionImageSize`, `hparams.tokenizerMaxLength`, `hparams.chunkSize`, and `hparams.maxStateDim`.
3. Build two `vlaPreprocessImage` frames, a BOS-only `tokens` and `mask` array, a `vlaPadState([0,0,0,0,0,0], hparams.maxStateDim)` state, and a `chunkSize × maxActionDim` zero-filled action buffer.
4. Call `vla({ modelId, images: [front, wrist], imgWidth, imgHeight, state, tokens, mask, noise })` and log the action chunk and the per-stage timings.


# Voice assistant (/courses/qvac/en/voice-assistant)

Now that we've seen speech-to-text, text generation, and text-to-speech separately, we're going to put them together.

A voice assistant is a loop: listen, transcribe, answer, speak. Each piece lives in a chapter we've already done. This chapter pulls them together with a streaming pipeline that keeps latency low, and adds the VAD tuning that filters the assistant's own TTS output so it isn't transcribed as a new user turn.

[Start Lesson 1 →](/courses/qvac/en/voice-assistant/voice-assistant-loop)

### All lessons in this chapter

1. [Build a real-time voice assistant loop](/courses/qvac/en/voice-assistant/voice-assistant-loop)
2. [Stop the voice assistant from hearing itself](/courses/qvac/en/voice-assistant/voice-assistant-echo)


# Stop the voice assistant from hearing itself (/courses/qvac/en/voice-assistant/voice-assistant-echo)

The voice assistant from the previous lesson works for a turn or two, then the feedback loop takes over: the TTS output gets picked up by the mic, Whisper transcribes it as a new user turn, and the LLM answers it. Each turn triggers the next with no real user input.

Four things drive that loop:

* The default VAD commits segments too eagerly for long-running use.
* The mic keeps recording through TTS, so the TTS output comes back as a new user turn.
* VAD occasionally commits phantom transcripts from near-silence, single tokens like "you" or "Thanks." with no real speech behind them.
* The TTS audio rings through the speaker into the mic for a moment after the model finishes, getting transcribed as the tail of the user's next turn.

The defaults are tuned for one-shot dictation, not a loop, so the first fix is to override `vad_params` in the ASR `modelConfig`:

```ts
const vad_params = {
  threshold: 0.6,
  min_speech_duration_ms: 300,
  min_silence_duration_ms: 700,
  max_speech_duration_s: 15.0,
  speech_pad_ms: 200,
};
```

`min_silence_duration_ms: 700` is the value that matters. VAD uses it to decide when the user stopped talking, so a longer quiet window keeps the TTS tail ringing through the speaker from getting folded into the user's turn.

VAD handles the first issue. The other three happen in the loop body, so the fix is three helpers defined right before the main loop:

* `isSpeaking` flag: the loop checks it at the top of each iteration to skip frames while TTS plays
* `isMeaningfulTranscript`: filter that drops the empty and phantom transcripts before they reach the LLM
* `sleep(ms)` helper: the main loop calls it to wait for the post-playback cooldown.

Here's how that looks in code:

```ts
const POST_PLAYBACK_COOLDOWN_MS = 300;
const MIN_UTTERANCE_CHARS = 3;

function isMeaningfulTranscript(text: string): boolean {
  const trimmed = text.trim();
  if (trimmed.length === 0) return false;
  if (trimmed.includes("[No speech detected]")) return false;
  if (/^\[[^\]]+\]$/.test(trimmed)) return false;
  const letters = trimmed.replace(/[^\p{L}\p{N}]/gu, "");
  return letters.length >= MIN_UTTERANCE_CHARS;
}

function sleep(ms: number) {
  return new Promise((resolve) => setTimeout(resolve, ms));
}

let isSpeaking = false;
```

The mic side gets a new gate. The previous lesson's data handler pipes every frame into the session; this one drops frames while the assistant is talking so the speaker output never reaches Whisper:

```ts
ffmpeg.stdout.on("data", (chunk: Buffer) => {
  if (isSpeaking) return;
  session.write(chunk);
});
```

The session is the async iterable; `transcribeStream({ modelId })` returns a `Promise<session>`, so you `await` it before the `for await`. The loop checks `isSpeaking` and `isMeaningfulTranscript` at the top of each iteration to skip frames, and the `try/finally` around the LLM+TTS block flips `isSpeaking` back to false even if the LLM throws, so a crashed turn doesn't leave the mic muted:

```ts
const session = await transcribeStream({ modelId: asrModelId });
for await (const rawText of session) {
  if (isSpeaking) continue;
  if (!isMeaningfulTranscript(rawText)) continue;
  const userText = rawText.trim();

  history.push({ role: "user", content: userText });

  isSpeaking = true;
  try {
    const llmResult = completion({ modelId: llmModelId, history, stream: true });
    let assistantText = "";
    for await (const token of llmResult.tokenStream) {
      process.stdout.write(token);
      assistantText += token;
    }
    history.push({ role: "assistant", content: assistantText });

    const spoken = assistantText.trim();
    if (spoken.length > 0) {
      const ttsResult = textToSpeech({
        modelId: ttsModelId,
        text: spoken,
        inputType: "text",
        stream: false,
      });
      const samples = await ttsResult.buffer;
      if (samples.length > 0) {
        const wavBuffer = Buffer.concat([
          createWavHeader(samples.length * 2, TTS_SAMPLE_RATE),
          int16ArrayToBuffer(samples),
        ]);
        playAudio(wavBuffer);
      }
      await sleep(POST_PLAYBACK_COOLDOWN_MS);
    }
  } finally {
    isSpeaking = false;
  }
}
```

Two tuning knobs to revisit if the loop still misbehaves. If VAD commits segments while the user is still talking, raise `min_silence_duration_ms`. If VAD commits segments out of near-silence, raise `threshold` to 0.7.

> Note: the `isSpeaking` flag drops the transcripts after Whisper processes them, but the mic is still recording the whole time. Pausing the ffmpeg pipe would let the buffer pile up, so we keep the pipe draining and drop the transcripts in software. The trade-off is a little extra VAD work on audio we'll throw away, in exchange for never stalling on a full buffer.

The SDK installs its own `SIGINT` / `SIGTERM` handler that aborts in-flight RPC streams on shutdown. The abort rejects any pending `session.write()` or `unloadModel()` call with a `WorkerShutdownError`. The bare-rpc socket also emits an `RPCError` with `code: 'CHANNEL_CLOSED'` on the same teardown path. Both surface as unhandled stream errors. The `process.on("uncaughtException", ...)` filter ignores the shutdown noise and re-throws anything else:

```ts
process.on("uncaughtException", (err) => {
  if (err instanceof WorkerShutdownError) return;
  if (err?.code === "CHANNEL_CLOSED") return;
  throw err;
});
```

## Put it to the test

1. Add `vad_params` (threshold 0.6, min\_speech\_duration\_ms 300, min\_silence\_duration\_ms 700, max\_speech\_duration\_s 15, speech\_pad\_ms 200) to the ASR `modelConfig`.
2. Define `POST_PLAYBACK_COOLDOWN_MS = 300`, `MIN_UTTERANCE_CHARS = 3`, `isMeaningfulTranscript(text)`, `sleep(ms)`, and `let isSpeaking = false`.
3. Gate on `ffmpeg` and `ffplay` being on `PATH` at startup. Define the WAV helpers (`createWavHeader`, `int16ArrayToBuffer`, `playAudio`) and the mic helper (`startMicrophone`) above `main`.
4. Open a `const session = await transcribeStream({ modelId: asrModelId })` session, then `const ffmpeg = startMicrophone(...)`, then `ffmpeg.stdout.on("data", (chunk) => { if (isSpeaking) return; session.write(chunk) })`.
5. Iterate with `for await (const rawText of session)`. Skip transcripts while `isSpeaking`, skip non-meaningful ones, push user turn into `history`, wrap completion + textToSpeech in a `try/finally` that flips `isSpeaking = true` at the start and `false` at the end. After `await ttsResult.buffer`, wrap the samples in a WAV header, `playAudio(wavBuffer)`, then `await sleep(POST_PLAYBACK_COOLDOWN_MS)`. Wire the `WorkerShutdownError` filter to `uncaughtException` so the SDK's shutdown abort doesn't dump a stack trace.


# Build a real-time voice assistant loop (/courses/qvac/en/voice-assistant/voice-assistant-loop)

Now that we've seen speech-to-text, text generation, and text-to-speech separately, we're going to put them together.

A voice assistant is a loop: listen, transcribe, answer, speak. Each piece lives in a chapter we've already done. This lesson wires the three pieces into a streaming conversation.

Three models to load: ASR with Whisper + Silero VAD, LLM with Llama 3.2 1B, TTS with Supertonic English. You would load them in this sequence:

```ts
const asrModelId = await loadModel({
  modelSrc: WHISPER_TINY,
  modelConfig: {
    vadModelSrc: VAD_SILERO_5_1_2,
    audio_format: "f32le",
    language: "en",
  },
});
const llmModelId = await loadModel({
  modelSrc: LLAMA_3_2_1B_INST_Q4_0,
  modelConfig: { ctx_size: 4096 },
});
const ttsModelId = await loadModel({
  modelSrc: TTS_EN_SUPERTONIC_Q8_0,
  modelConfig: {
    ttsEngine: "supertonic",
    language: "en",
    voice: "F1",
    ttsSpeed: 1.05,
    ttsNumInferenceSteps: 5,
  },
});
```

The system prompt is what tells the model how to behave: short answers (TTS takes \~1s per sentence), no markdown (asterisks sound bad read aloud), no lists (hard to follow mid-task). The system prompt might look as follows:

```ts
const history: Array<{
  role: "system" | "user" | "assistant";
  content: string;
}> = [{ role: "system", content: SYSTEM_PROMPT }];
```

`transcribeStream({ modelId })` returns a `Promise<TranscribeStreamSession>`. `await` it to get the session, then iterate with `for await (const rawText of session)`. The session yields plain text strings.

Audio capture is a child ffmpeg process. The `startMicrophone()` helper spawns the system ffmpeg with the right `-i` args for the current platform (avfoundation on macOS, pulse on Linux, dshow on Windows) and pipes 16 kHz mono f32le PCM to stdout. Each chunk on stdout is a frame we feed into the session:

```ts
const ffmpeg = startMicrophone({ sampleRate: 16000, format: "f32le" });
const session = await transcribeStream({ modelId: asrModelId });

ffmpeg.stdout.on("data", (chunk: Buffer) => {
  session.write(chunk);
});
```

A startup check on `ffmpeg -version` and `ffplay -version` makes the failure mode obvious if either is missing. The `for (const tool of ['ffmpeg', 'ffplay'])` loop keeps it to a few lines.

The main loop is one async `for await` over the session. Each iteration produces a user turn, runs the LLM, then speaks the answer:

```ts
for await (const rawText of session) {
  const userText = rawText.trim();
  if (userText.length === 0) continue;

  history.push({ role: "user", content: userText });

  const llmResult = completion({
    modelId: llmModelId,
    history,
    stream: true,
  });
  let assistantText = "";
  for await (const token of llmResult.tokenStream) {
    process.stdout.write(token);
    assistantText += token;
  }
  history.push({ role: "assistant", content: assistantText });

  const ttsResult = textToSpeech({
    modelId: ttsModelId,
    text: assistantText.trim(),
    inputType: "text",
    stream: false,
  });
  const samples = await ttsResult.buffer;
  if (samples.length > 0) {
    const wavBuffer = Buffer.concat([
      createWavHeader(samples.length * 2, TTS_SAMPLE_RATE),
      int16ArrayToBuffer(samples),
    ]);
    playAudio(wavBuffer);
  }
}
```

`textToSpeech()` returns the audio as raw 16-bit signed PCM samples at 44.1 kHz mono. To play it we wrap the samples in a minimal WAV header and pipe the buffer into `ffplay`, which ships with ffmpeg.

A `cleanup()` handler kills the ffmpeg child, ends the session, and unloads all three models. Wire it to `SIGINT` and `SIGTERM` so Ctrl+C exits cleanly:

```ts
async function cleanup() {
  if (shuttingDown) return;
  shuttingDown = true;
  ffmpeg.kill();
  try { session.end(); } catch {}
  await unloadModel({ modelId: ttsModelId }).catch(() => {});
  await unloadModel({ modelId: llmModelId }).catch(() => {});
  await unloadModel({ modelId: asrModelId }).catch(() => {});
  process.exit(0);
}

process.on("SIGINT", () => void cleanup());
process.on("SIGTERM", () => void cleanup());
```

The SDK installs its own `SIGINT` / `SIGTERM` handler that aborts in-flight RPC streams on shutdown. The abort rejects any pending `session.write()` or `unloadModel()` call with a `WorkerShutdownError`. The bare-rpc socket also emits an `RPCError` with `code: 'CHANNEL_CLOSED'` on the same teardown path. Both surface as unhandled stream errors. The `process.on("uncaughtException", ...)` filter ignores the shutdown noise and re-throws anything else:

```ts
process.on("uncaughtException", (err) => {
  if (err instanceof WorkerShutdownError) return;
  if (err?.code === "CHANNEL_CLOSED") return;
  throw err;
});
```

> Note: the system prompt bans markdown and lists because the output is spoken aloud. A `### heading` or a `1. ` list reads as a stuttery mess through TTS.

## Put it to the test

1. Make three `loadModel()` calls for the ASR (Whisper + Silero VAD), LLM (Llama 3.2 1B), and TTS (Supertonic English) models.
2. Declare `history` with the system prompt as the first message.
3. Gate on `ffmpeg` and `ffplay` being on `PATH` at startup. Define the WAV helpers (`createWavHeader`, `int16ArrayToBuffer`, `playAudio`) and the mic helper (`startMicrophone`) above `main`.
4. Open a `const session = await transcribeStream({ modelId: asrModelId })` session, then `const ffmpeg = startMicrophone(...)`, then `ffmpeg.stdout.on("data", (chunk) => session.write(chunk))`.
5. Iterate with `for await (const rawText of session)`. Each iteration: trim the transcript, push it as a user turn into `history`, call `completion()` with `stream: true`, then push the assistant turn into `history`.
6. After `completion()` returns, call `textToSpeech()` and `await ttsResult.buffer`. Wrap the samples in a WAV header and `playAudio(wavBuffer)`.
7. After the loop, run the cleanup handler. Wire the `WorkerShutdownError` / `CHANNEL_CLOSED` filter to `uncaughtException` so the SDK's shutdown abort doesn't dump a stack.