Skip to content

Builds on 2. Durable host. Run two workers behind pinboard, send a message through one URL, then kill the worker streaming the reply. The surviving worker restores the thread from Postgres and finishes the echo.

  • Keep the durable tutorial's app.py, Postgres database, and smoke.mjs client.
  • Stop the local host from tutorial 2. Both fleet workers use that same database.
  • Use a fresh thread ID, fleet-t1, for the checks below.

1. Start pinboard

From the harness-sdk checkout:

pnpm install
pnpm --filter "@assistant-ui/pinboard..." build

Start pinboard in dev mode on port 8080. Its passport gives dev-token the worker and operator scopes, and admits anonymous clients:

pnpm --filter @assistant-ui/pinboard dev

Dev mode keeps pinboard's registry in this process. The workers store the transcript and run state in Postgres. Production configuration: run a fleet and authenticate.

2. Connect the host to pinboard

app.py stays as it is. PinnedWorker joins pinboard when these variables are set:

  • PINBOARD_URL: the registry this worker joins.
  • PINBOARD_TOKEN: the passport key carrying pinboard:worker.
  • ADVERTISE_URL: a URL that reaches this one worker process. Pinboard verifies it with a challenge on registration and every heartbeat.
  • DATABASE_URL: the same Postgres database for both workers, so either can restore a thread.

The same three variables drive a Node worker (@assistant-ui/pinned):

import { pinned } from "@assistant-ui/pinned";
import { serve } from "@assistant-ui/pinned/node";
import { Hono } from "hono";

const Counter = pinned(() => {
  let value = 0;
  const app = new Hono();
  app.post("/increment", (c) => c.json({ value: ++value }));
  return app;
});

await serve({ hosts: { "/counter": Counter } });

Keep the PostgresStateStore, app lifespan, and worker from tutorial 2.

3. Start two workers

In one terminal, start worker A:

PINBOARD_URL=http://127.0.0.1:8080 PINBOARD_TOKEN=dev-token ADVERTISE_URL=http://127.0.0.1:8000 \
DATABASE_URL=postgresql://postgres@localhost/chat uv run uvicorn app:app --port 8000

Wait for worker registered in pinboard's log, then start worker B in another terminal:

PINBOARD_URL=http://127.0.0.1:8080 PINBOARD_TOKEN=dev-token ADVERTISE_URL=http://127.0.0.1:8001 \
DATABASE_URL=postgresql://postgres@localhost/chat uv run uvicorn app:app --port 8001

Inside the repo, replace uv run uvicorn ... with uv run --directory python uvicorn --app-dir "$PWD" app:app --port 8000 (or 8001), keeping the same variables.

Confirm both registrations:

curl http://127.0.0.1:8080/registry/workers \
  -H 'Authorization: Bearer dev-token'

The response's workers array has two entries. Match each advertiseUrl to its terminal. Each entry also has a clientId for operator drains.

4. Send through pinboard

From the client directory, use tutorial 1's smoke.mjs:

URL=http://127.0.0.1:8080/threads/fleet-t1 MESSAGE="remember me" node smoke.mjs

On a fresh registry, the first placement opens a 15-second quarantine. The client retries the 503 registry cold start, retry responses automatically. Its last transcript line is:

user: remember me | assistant: echo: remember me
  • One worker logs POST /threads/fleet-t1/stream and POST /threads/fleet-t1/frames. That worker owns this placement.
  • A direct attach to a registered worker's URL returns 403. Clients use pinboard's URL.
  • A placement stays on its worker until released or the lease expires. Idle eviction releases the placement by default.
  • Multiple thread IDs from the same client IP can land on the same worker. Two workers provide capacity and a replacement owner; placement affinity determines the initial choice.

5. Kill the owner during a reply

Send a longer message through the same URL:

URL=http://127.0.0.1:8080/threads/fleet-t1 \
MESSAGE="$(node -e 'console.log("recover ".repeat(200))')" node smoke.mjs
  1. Wait until several words of the echo appear. Leave smoke.mjs running.
  2. Find the worker logging this turn's /frames request. Its Uvicorn startup log shows Started server process [PID].
  3. In another terminal, run kill -9 <PID> with that worker's PID. Keep pinboard, Postgres, and the other worker running.
  4. The client's stream reconnects. After the dead worker's lease expires, pinboard places the thread on the survivor. The lease TTL is 15 seconds by default; client retries add to the wait.
  5. The survivor logs the new /stream request. The earlier reply stays unchanged, and the interrupted echo restarts from its first word under the same assistant message ID, then completes. The thread still has two user messages and two assistant messages.

Send another turn:

URL=http://127.0.0.1:8080/threads/fleet-t1 MESSAGE="after failover" node smoke.mjs

It extends the recovered conversation through the same URL. GET /registry/workers now lists the surviving worker.

What you built

ComponentResponsibility
PinboardRegister workers, lease placements, and forward client requests to the owner
WorkersExecute the chat host and stream its state
Shared PostgresStore the transcript, command queues, and run state for recovery on either worker