Run a fleet

A fleet routes each (namespace, id) through pinboard to its owning worker. For a complete application, follow Run two workers. This guide covers registration, placement, and lease behavior. Durable state: durability. Drains, deploys, hostnames, and observability: operate a fleet.

PinboardCopy Anchor Link

Pinboard is the control plane: stateless nodes (Node, or a Durable Object on workerd) behind one URL, coordinating through Redis. They own the directory and proxy every data-plane request to the worker holding the id.

Build it, then start it against Redis with a passport:

pnpm --filter @assistant-ui/pinboard build
PINBOARD_TOKEN='...' \
PINBOARD_PASSPORT='{"providers":[{"type":"staticToken","id":"pinboard","keys":[{"name":"deployment","token":{"env":"PINBOARD_TOKEN"},"scopes":["pinboard:worker","pinboard:operator"],"claims":{"pinboard":{"namespaces":["*"],"allowedBackendHosts":["*"]}}}]}],"namespaces":{"*":{"allow":"anonymous"}}}' \
REDIS_URL=redis://localhost:6379 PORT=8080 pnpm --filter @assistant-ui/pinboard start

• Both commands run from the repository root. To launch the built server directly from there, replace the pnpm --filter @assistant-ui/pinboard start portion with node packages/pinned/pinboard/dist/node/main.js; keep the environment variables.
• PINBOARD_PASSPORT is required. REDIS_URL=memory: runs an in-process store for dev.
• "namespaces":{"*":{"allow":"anonymous"}} admits clients without a credential: dev only. Production policies: authenticate.
• Every PINBOARD_* variable: pinboard server. createPinboard.Options, servePinboard, and PinboardDO: pinboard reference.
• Scopes, claims, and attestation: Trust.
• Clients address instances against pinboard: GET http://pinboard/counters/alice/value.
• A namespace with no registered worker answers 404 unknown namespace.

Registering a workerCopy Anchor Link

Point PinnedWorker at pinboard and register the hosts on it:

worker = PinnedWorker(
    hosts={"/counters": Counter, "/tickers": Ticker},
    pinboard_url="http://pinboard.internal:8080",
    advertise_url="http://10.0.3.7:8000",  # must uniquely address THIS process
    token="the deployment token",          # passport key carrying pinboard:worker
    state_store=store,                     # PostgresStateStore: trays outlive this process
)
app.include_router(worker)                 # hosts, GET /_pinned/challenge and /_pinned/stats

• Omitted, pinboard_url, token, and advertise_url read PINBOARD_URL, PINBOARD_TOKEN, and ADVERTISE_URL. PINBOARD_URL without a token raises ValueError.

• One PinnedWorker per process: one client id, session, lease, and heartbeat shared by its hosts, one namespace each.

• app.include_router(worker) mounts every host under the worker's base: pinboard reaches <advertise_url>/_pinned/* and <advertise_url><namespace>/<id>/....

• Registration starts once every attached host's lifespan has started; a host cannot attach after that.

• Start order is free: a failed or 503-deferred registration retries with exponential backoff, capped at 30s. worker.ready stays False until registration succeeds. Fleet-mode instance routes require a valid proxy attestation, including before registration. A 401 logs check the deployment token and keeps retrying.

• Run each worker as its own process: uvicorn app:app --port 8001, --port 8002 with matching advertise_urls.

• On Kubernetes, address the pod, never a Service: advertise_url=f"http://{os.environ['POD_IP']}:8000" with POD_IP from the downward API (fieldRef: status.podIP). Pinboard reaches pods directly, so the pod network must be routable from it.

Exactly one process per advertise_url:

• Registration and every heartbeat carry a fresh challenge/challengeValue pair. Pinboard probes GET <advertise_url>/_pinned/challenge and requires the value only this process knows.
• Replicas behind a shared URL receive each other's traffic and answer 403.

The worker heartbeats every heartbeat_interval (3s) with headroom (available memory in MB, cgroup-aware):

• PinnedWorker(headroom=lambda: mb) replaces the probe.
• Wire format: pinboard registry.

Verify startupCopy Anchor Link

In another terminal, set PINBOARD_TOKEN to the deployment token used above. Its pinboard:operator scope permits the registry checks:

export PINBOARD_URL=http://localhost:8080
export PINBOARD_TOKEN='...'
curl --fail-with-body -sS -H "Authorization: Bearer $PINBOARD_TOKEN" "$PINBOARD_URL/registry/workers"

• The response is {"workers":[...]}. Each running worker should have its own clientId and advertiseUrl; namespaces should include /counters and /tickers for the mounts above.
• An empty array means the registry has no live workers visible to this credential. Check worker registration logs, token scopes, and pinboard's reachability to each advertised URL.
• Use clientId for operator drain and undrain. A process restart gets a new client ID.

Then call the Counter object's route through pinboard:

curl --fail-with-body -sS "$PINBOARD_URL/counters/alice/value"

The anonymous namespace policy above admits this request. A production namespace uses its configured client credentials.

PlacementCopy Anchor Link

• Placement is per (namespace, id): one atomic resolve claims the id on a live, undrained, non-degraded worker, biased toward headroom.
• The claim is sticky until released or the owner's lease lapses.
• Candidate filter, choice rule, and affinity: Placement.

First placements:

• They prefer the caller's own worker (cluster calls) or a stable per-client-IP pick, bounded by PINBOARD_AFFINITY_BALANCE_FACTOR. The client IP follows PINBOARD_TRUSTED_PROXIES.
• With equal reported headroom, IDs from one client IP prefer the same worker. New placements subtract one unit until the next heartbeat refresh.
• headroom is a constant 1024 without a cgroup limit or psutil (uv add "pinned[psutil]"). A local two-worker test then looks one-sided.

For service callers that create independent threads, set affinity.key in createPinboard or servePinboard:

const pinboard = servePinboard({
  redis,
  passport,
  affinity: { key: () => null },
});

• null uses power-of-two selection for new placements. Existing IDs keep their owners.
• A non-empty string selects a rendezvous preference. Use a stable per-thread key to spread independent threads; a shared service key groups them on one preferred worker.
• The synchronous callback receives the original Request (including basePath) and the transport's Client. Return the same key on retries; derive it from the URL or headers without consuming the body.
• Attested cluster calls retain their own-worker preference.
• The callback is a programmatic option. The standalone environment configuration uses client-IP affinity.

For CPU-bound workloads, supply PinnedWorker(headroom=probe) with a fast probe of remaining workload capacity:

• Report non-negative free slots in consistent units across workers; count all active work sharing the worker, including background runs.
• Express memory and event-loop limits in the same slot units before taking the minimum. Sample quickly enough for the workload.
• A lower balanceFactor closer to 1 tightens the preference guard. It still compares headroom, so large memory-MB values can preserve affinity through a large burst.
• Headroom is a selection weight. Zero headroom remains eligible; enforce hard concurrency limits with worker admission or a queue.
• Heartbeats replace the placement decrement. A slot probe must include work already admitted; placements awaiting activation can temporarily exceed the reported capacity.
• Retained idle placements are sticky and differ from active runs. New settings affect new placements and re-placement after release or failover.

Host(Api, release_placement_on_idle=False) keeps the claim across idle eviction:

• An idle id stays on its worker, and after a deploy it re-places as a dead-owner row.
• The row stays until the id is released or the worker's session ends.

Leases and heartbeatsCopy Anchor Link

Each heartbeat ack renews the worker record's lease (PINBOARD_LEASE_TTL_MS, 15s), surfaced in the ack as expiresInMs.

• A frozen process (VM pause, long GC) waking past its lease refuses requests, and pinboard nacks its next heartbeat.

When a worker diesCopy Anchor Link

• A plain shutdown releases the worker's placements and deregisters on lifespan exit. install_sigterm_drain() adds a drain window first (wind-down and drain).
• A lost lease leaves the ids re-placeable once expires_at passes. The next request places them on another worker.
• A worker refusing an activation in-band (lease unhealthy) is released and the id retried once elsewhere (in-band refusal).
• A 409 reregister nack loses the session. Instances stay. The worker re-registers at once, listing its live ids; the ids it no longer owns come back as duplicates and are quit (register).
• A 409 superseded nack aborts every instance and stops registering (epoch displacement).

Cold start, after a fresh memory: boot or a Redis wipe:

• The first placement attempt stamps the bucket, and placement refuses for one lease TTL from there. quarantineMs on GET /registry/health shows the remainder, 0 before that first attempt.
• The statewire client reconnects on its own; a curl reader retries after the Retry-After header.
• Workers re-register within a heartbeat interval. Ids re-place on demand.

Local modeCopy Anchor Link

A worker without pinboard_url or PINBOARD_URL is local mode: an in-memory directory in this process, same external API. One worker per state_store.

• Local mode is dev mode: attestation checks are off, and the worker logs a DEV MODE warning.
• Python and Node refuse it when NODE_ENV, ENVIRONMENT or APP_ENV is production (or prod), unless PINBOARD_ALLOW_DEV_IN_PRODUCTION=1.
• A token or advertise URL (PINBOARD_TOKEN, ADVERTISE_URL) without a pinboard URL throws, as does an empty PINBOARD_URL.

from fastapi import FastAPI
from pinned import PinnedWorker

app = FastAPI()
worker = PinnedWorker(hosts={"/counters": Counter})
app.include_router(worker)

• Counter is the class from authoring. Run it with uvicorn app:app --port 8000.
• Full lifecycle (activation, idle eviction, streams, create_task, drain) minus the fleet: instances are singletons per process, lease.healthy stays True, lease.expires_at stays None.
• Host knobs and their defaults: Host, PinnedHost.

NextCopy Anchor Link

• Operate a fleet: drains, rolling deploys, hostnames, observability.