Connect a delegated-inference consumer

The previous lesson booted a provider. This one consumes it.

The runner captures the provider's public key from the previous run's output and passes it to this snippet as process.argv[2]. You don't have to copy anything. If you want to connect to a different provider (a peer you don't own, a test instance on another machine), type the key into the Provider public key field above the editor. That value takes precedence over the captured one and persists across runs, so you only set it once.

The consumer side is the same loadModel() / completion() interface as every other lesson. The only new field is delegate, which routes the inference to a peer instead of running it locally.

The loadModel() call takes the same modelSrc and modelType parameters. It also takes delegate, which carries the routing fields (providerPublicKey, timeout, fallbackToLocal):

const modelId = await loadModel({
  modelSrc: LLAMA_3_2_1B_INST_Q4_0,
  delegate: {
    providerPublicKey,
    timeout: 60_000,
    fallbackToLocal: true,
  },
});

The completion() call uses the same modelId, history, and stream: true shape as the text-generation lessons. The modelId here is the delegate's handle, so the call routes through the peer over the DHT:

const response = completion({
  modelId,
  history: [{ role: "user", content: "Hello!" }],
  stream: true,
});

The loop iterates response.tokenStream and writes each token to stdout. The response.stats Promise resolves once the stream ends, so logging it after the loop gives the per-call metrics:

for await (const token of response.tokenStream) {
  process.stdout.write(token);
}
console.log("\n▸ Stats:", await response.stats);

The delegate block takes the provider's public key and a generous timeout. The first call on a cold DHT needs 15 to 45 seconds: bootstrapping hyperdht, looking up the provider's key, opening the connection. The SDK gives that headroom via timeout: 60_000. Once the DHT is warm, subsequent connections in the same process are sub-second.

fallbackToLocal: true is the safety net. If the provider is unreachable (restarting, offline, behind a firewall the consumer can't traverse), the consumer falls back to running the model on its own hardware. The call still succeeds, though with a different latency profile.

Note: the consumer's modelSrc has to match the provider's loaded model. If the provider has Llama 3.2 1B loaded, the consumer has to ask for the same constant, or the provider's router will reject the request with a model-mismatch error.

Put it to the test

1. Call loadModel with modelSrc: LLAMA_3_2_1B_INST_Q4_0 and a delegate: { providerPublicKey, timeout: 60_000, fallbackToLocal: true } block.
2. Call completion({ modelId, history: [{ role: "user", content: "Hello!" }], stream: true }).
3. Iterate response.tokenStream to stdout, then console.log response.stats when the stream ends.