Pinboard Registry

Pinboard is the control plane between clients and workers.

Unauthenticated   GET  /healthz            GET  /readyz
Worker            POST /registry/register  POST /registry/heartbeat  POST /registry/release  POST /registry/tag  POST /registry/deregister
Operator          GET  /registry/health    GET  /registry/workers    GET  /registry/overview GET  /registry/stats
                  POST /registry/wind_down POST /registry/drain      POST /registry/undrain  GET  /metrics (Node entry)
Pinboard → worker GET  {advertiseUrl}/_pinned/challenge?challenge=  GET {advertiseUrl}/_pinned/stats  POST {advertiseUrl}{ns}/{id}/_wake

• /registry/* and /metrics:
  • 401 {"error":"unauthorized"} without a valid Passport credential.
  • 403 {"error":"forbidden"} for a credential missing the route's scope (pinboard:worker for worker routes, pinboard:operator for the rest; principals) or carrying a malformed pinboard claim.
  • Unknown /registry/* paths authenticate as pinboard:worker and pass the serving check before answering 404 {"error":"not found"}.
• Every /registry/* route but /registry/health answers 503 {"error":"not the serving fleet"} + Retry-After: 1 while this node is not serving.
• Bodies are JSON. A body over 1 MiB (8 MiB for register) is 413 {"error":"body too large"}. A body failing the route's schema is 400 {"error":"expected {...}"}. Unparseable JSON is 400 {"error":"invalid json"}.
• Data plane: an id that is empty, over 256 characters, or outside [A-Za-z0-9._~-] is 400 text/plain. A namespace is / plus one path segment.
• POST {advertiseUrl}{ns}/{id}/_wake is the durability sweep's wake: 200 {"ok":true} once the instance is resident, 202 {"activating":true} while hydrating. Mounted only on a host with a durability store.

Leases and epochsCopy Anchor Link

• Placement row: hash {prefix}{{bucket}}:{env}:{ns}:placements (the bucket sits in literal hash-tag braces), field id, bucket = the first 8 hex digits of sha256(id) mod shards.
  • Fields: sessionId, epoch?, expiry?, hostname?.
    • epoch? is absent only on ownerless placeholder rows paged in by the sweep.
    • expiry? is the wind-down deadline. The row counts as free past it.
    • hostname? is the resurrection target on placeholder rows.
  • Placed = row present, owner live, wind-down unexpired.
• Worker record: a hash field carrying expiry (now + leaseTtl at register and each heartbeat, bounded by a timed drain).
• Dead once expiry ≤ now.
• Pruned once expiry + lease < now, together with bindings without a live worker: by the next register, by each mirror poll, and by the sweeper's GC.
• Redis TTLs: the control queue, the sweeper lease, the fleet license, and the drain key of a timed drain (a null cordon has none). Membership is one hash with a per-member expiry field.
• Sessions: registration mints a sessionId. Placements belong to the session, one session per placement. Placements pointing at a dead session are re-claimed on the next resolve.
• Placement epoch: a decimal string, monotonic per bucket (max(now-based, previous + 1, floor + 1)), minted at every claim.
  • It rides each forward as Pinboard-Epoch, quotes releases, and fences overwritten-marker deletes.
  • A worker answers 400 malformed placement epoch to a non-digit value.
• Binding epochs (hostname-aware): an equal or lower epoch conflicts 409 (register).
• Fleet license: {schema, shards, hostnameAware, epoch} in Redis, claimed with a TTL of one lease, overwritten by a higher epoch, and compared byte for byte.
  • Cluster members must match every live member's config unless that member's epoch is lower.

Worker protocolCopy Anchor Link

POST /registry/registerCopy Anchor Link

→ { clientId, secret, namespaces: string[], advertiseUrl, headroom: number, draining: boolean,
    challenge, challengeValue, hostname?, env?, envs?: {ns: env}, takeover?: "wind_down" | "deny",
    epoch?: integer ≥ 0, instances?: {ns: id[]}, prevSessionId? }
← 200 { sessionId, expiresInMs, draining, drainDeadline?, duplicates?: [{ns, id}] }

• Pinboard probes GET {advertiseUrl}/_pinned/challenge?challenge=<challenge> with Pinboard-Proxy-Secret: <secret> (timeout verifyTimeoutMs, default 2500 ms) and requires {"value": <challengeValue>}.
  • Credential scope (403 below) is checked before the probe.
  • Every registration probe failure is 400 {"error":"registration probe failed"}. Probe details remain in host logs. Worker retry backoff remains active.
• claims.pinboard.namespaces is required and nonempty for both registry roles; ["*"] grants all namespaces. Optional allowedPorts limits advertise URLs to integer ports 1–65535; omission permits every port within allowedBackendHosts.
• 403 {"error":"namespace not allowed: …" | "backend host not allowed: …" | "advertiseUrl port not allowed: …" | "hostname not allowed: …" | "env not allowed: …"} when the body falls outside the credential's pinboard claims.
• 409 {"error"} on a conflict:

• A higher epoch rewrites the binding and quarantines new placements in the displaced (hostname, ns) for one lease. Ids with a placement row bypass the quarantine.
• Field rules:
  • Single mode rejects hostname, env, envs, takeover, and a nonzero epoch (400).
  • Hostname-aware mode requires hostname; env defaults to hostname.
  • envs keys and instances keys must be registered namespaces.
  • advertiseUrl is an HTTP(S) URL with a hostname and without userinfo. Trailing slashes are stripped.
  • hostname and env are lowercased. *, whitespace, /, or : in either is 400 {"error":"invalid hostname: …" | "invalid env: …"}.
  • takeover defaults to wind_down.
• instances reclaims ids still held in memory: rows owned by prevSessionId re-bind to the new session keeping their epoch. Every other id returns in duplicates for the worker to evict. A reclaimed row carrying a wind-down expiry re-queues its wind_down to the new session.
• expiresInMs is the lease TTL (PINBOARD_LEASE_TTL_MS, default 15000). draining is the worker's own flag OR an operator drain. drainDeadline is the remaining seconds of a timed operator drain.

POST /registry/heartbeatCopy Anchor Link

→ { sessionId, headroom, draining, challenge, challengeValue, acks?: msgId[] }
← 200 { expiresInMs, draining, drainDeadline?, degraded?: true,
        messages?: [{ msgId, type: "wind_down", ns, id, deadline: seconds | null }] }

• challenge and challengeValue are both required. Pinboard re-probes the advertise URL each heartbeat.
• 409 {"error", "reregister": true}:
  • unknown session (unknown or expired)
  • registration voided: challenge value mismatch on the advertise URL (the probe answered another worker's value; records dropped)
• 409 {"error", "superseded": true}: registration superseded: {hostname}{ns} now bound to env {env} at epoch {n}; records dropped.
• The Python worker re-registers only from the heartbeat path, gives up on release and deregister after three tries, and retries a heartbeat 403 with the same session.
• 403 {"error":"forbidden"}: the credential's account differs from the registering one, or its claims no longer cover the record.
• A probe that is unreachable, non-2xx, or body-less still renews the lease and sets degradedSince on the record. A passing probe clears it. Only the heartbeat probe sets the flag.
  • While set: existing placements keep routing, new placements skip the worker, the ack carries degraded: true.
• messages re-deliver on every ack until acks names their msgIds. deadline is relative seconds at peek time, null revokes. The control queue expires two lease TTLs after its last write or peek.
• The record's headroom is replaced by the heartbeat's. Each new placement decrements the stored value until the next heartbeat.

POST /registry/releaseCopy Anchor Link

→ [{ sessionId, namespace, id, epoch?: "digits" }]   (non-empty)
← 200 { released: number }

• Compare-and-delete: a row is freed only while its sessionId matches and its epoch equals the quoted one ('' quotes an epoch-less row).
• 409 {"error":"unknown session","reregister":true} for a session with no record. 400 {"error":"invalid namespace" | "namespace not registered: …"}.
• 403 from another account or outside the credential's claims.
• Release skips the expiry check heartbeat and deregister apply.

POST /registry/tagCopy Anchor Link

→ { sessionId, namespace, id, epoch: "digits", tag }   (tag: 1–256 chars)
← 200 { tagged: boolean }

• Compare-and-set of the placement's owner tag: written only while the row's sessionId matches and its epoch equals the quoted one; tagged: false otherwise.
• The tag survives register-time reclaim and wind-down and is dropped with the row on release or re-placement.
• 409 {"error":"unknown session","reregister":true} for a session with no record. 400 {"error":"invalid namespace" | "namespace not registered: …"}.
• 403 from another account or outside the credential's claims.

POST /registry/deregisterCopy Anchor Link

→ { sessionId }, ← 200 {}.

• Removes the session from every namespace list and drops its record. Placements are left to dead-owner eviction.
• Unknown session: 409 {"error":"unknown session","reregister":true}. Another account's session: 403.

HealthCopy Anchor Link

• GET /healthz: 200 {"ok":true}.
• GET /readyz: empty body. 200 when not draining and the node holds the fleet license and passes cluster membership, else 503.
  • Cold-start quarantine shows only as quarantineMs on /registry/health.

Operator endpointsCopy Anchor Link

• GET /registry/health: { resolutions: {outcome: count}, connections, quarantineMs, uptimeMs, rssBytes? } for this node. It answers while booting or draining.
  • Outcomes: forward, unknown-ns, no-backends, quarantine, not-serving, no-hostname-workers, binding-quarantine, takeover-pending, env-conflict, nack.
  • quarantineMs: the largest remaining cold-start window across the buckets, rounded up.
• GET /registry/workers: { workers: [{ sessionId, clientId, advertiseUrl, namespaces, hostname, envs, takeover, headroom, draining, degradedSince?, expiry }] }.
  • Live records with secret and account stripped, namespaces filtered to the credential's claims, sorted by hostname then session.
• GET /registry/overview, scoped like /registry/workers:
  • { workers, bindings: { hostnames: {"{hostname}{ns}": env}, envs: {env: takeover} }, namespaces: [{ hostname, env, namespace, sessionIds, entries: [{ id, sessionId, expiry?, tag? }] }] }
  • Query filters, ANDed: ns (exact namespace, 400 {"error":"invalid namespace"} when malformed), prefix (id prefix), tag (exact owner tag; untagged rows are excluded, 400 {"error":"empty tag"} when blank). ns drops other namespaces from namespaces; prefix and tag filter entries. Live placements only.
  • GET /registry/overview?ns=/threads&prefix=H~&tag=ws:W lists one workspace's placements under one harness.
• GET /registry/stats: { workers: [{ sessionId, clientId, hostname, stats } | { …, error: "unreachable" }] }.
  • Each live worker's GET {advertiseUrl}/_pinned/stats, attested, 3 s timeout, 8 concurrent.
  • Namespaces and instances are filtered to registered ones.
• POST /registry/wind_down { namespace, id?, deadline: seconds | null, env? } → { queued: boolean | number }.
  • Stamps the placement's wind-down expiry at now + max(deadline * 1000, leaseTtlMs) (null clears) and queues a wind_down control message to the owner. The fence survives worker death and re-notifies a replacement owner.
  • Without id, every placement in the (env, namespace), queued the count.
  • 400 {"error":"env is required in hostname-aware mode" | "env requires hostname-aware mode"} when env is absent in hostname-aware mode or present otherwise.
  • Scope is judged on the owner record's hostname and envs[namespace], or on the row's hostname when the owner record is gone.
  • Out of scope: 403 {"error":"namespace not allowed: …" | "env not allowed: …" | "forbidden"}.
• POST /registry/drain { clientId, deadline: seconds | null } → { drained: true }.
  • Writes the client's drain config: null cordons, a number forces release at now + max(deadline * 1000, leaseTtlMs). A timed drain key lives 7 days past its deadline.
  • 403 {"error":"forbidden"} for an unknown or out-of-scope clientId.
• POST /registry/undrain { clientId } → { revoked: boolean }. Unknown or out-of-scope clientId: 403.
• Account ownership is checked by register, heartbeat, release, and deregister only. Wind-down, drain, undrain, and the read routes judge the credential's namespace, hostname, and env claims.
• Recipes: wind-down and drain, operator drain.
• GET /metrics: Prometheus text. Node entry only, outside basePath, present only with a metrics registry configured (otherwise 404 {"error":"unknown namespace"}).
  • 403 {"error":"forbidden"} unless the operator policy carries "*" in namespaces and unrestricted hostnames/envs.
  • 500 {"error":"metrics collection failed"} on a snapshot failure.
  • The fleet overview behind the collectors is memoized for 5 s.

PlacementCopy Anchor Link

The resolve script runs atomically per (env, namespace, id):

1. An existing row forwards to its owner when the owner is live (record present, expiry > now) and its wind-down has not passed.
   • A row whose owner's hostname differs from the request's applies the env's takeover: deny → takeover-denied; wind_down → the row is fenced at now + leaseTtl (takeover-pending until then).
2. No ns_workers list for {hostname}{ns} → unknown-ns (404 {"error":"unknown namespace"}).
   • In hostname-aware mode the binding lookup runs first and can refuse no-hostname-workers, binding-quarantine, or env-conflict before the script.
   • A script no-backends there reports as no-hostname-workers.
3. boot_time is SET NX to now, per bucket. While boot_time + leaseTtl > now the bucket's result is quarantine.
4. Candidates: sessions in the namespace list that are live, not draining, not degraded, not the excluded session, with expiry - now ≥ min(5000, leaseTtl / 3).
   • Outlier-ejected sessions are used only when no other candidate remains.
   • None → no-backends.
5. Affinity: a caller-attested request prefers its own session. Otherwise affinity.key(req, client), or the client-IP key per trustedProxies when unset, picks a stable candidate by rendezvous hash. A custom null key skips the preference.
   • The preference is honored while that worker's headroom ≥ mean / balanceFactor.
6. Otherwise power-of-two: two random candidates, the higher headroom wins.
7. A new epoch is minted. The old row (if any) is deleted and the id listed on the overwritten hash. The chosen worker's stored headroom is decremented. The row { sessionId, epoch, expiry? } is written with HSET.

Refusals are 503 {"error"} + Retry-After: max(1, ceil(remaining / 1000)) seconds:

• Namespace lists and worker records are read per resolve. Nothing is cached. Heartbeat re-adds the worker to every namespace list in its bucket, and a bucket missing a namespace list copies the previous bucket's.
• Route misses: 404 {"error":"unknown namespace"} for a request path, 404 {"error":"not found"} for an upgrade or a path outside basePath. A handler throw is 500 {"error":"internal error"}.
• Hostname-aware mode: the Host header (port stripped, lowercased) names the hostname (400 {"error":"missing Host header"} without one). The (hostname, ns) binding names the env, the env binding the takeover.

In-band refusal and re-placementCopy Anchor Link

• A worker refusing a new activation answers 503 with Retry-After: 1 and Pinned-Refusal: lease-unhealthy. Pinboard:
  • strips the header, releases the placement, counts nack;
  • re-resolves once excluding that session (upgrades and body-less requests only; a request with a body gets the 503 at once);
  • answers 503 {"error":"worker refused activation, retry"} when the retry also refuses or no other worker remains;
  • answers with that refusal's string on a refusal from the re-resolve, 404 {"error":"unknown namespace"} when the namespace vanished meanwhile.
• A resolution older than 5 s by the time the forward sends is re-resolved, twice at most; the worker's epoch guard catches a forward that is still stale.
• A fresh placement whose forward never reached TCP is released so the client's retry re-places. An unreachable forward never marks the record degraded.
• Forward failures: 502 bad gateway (text) for an unreachable worker, 504 upstream timeout after forwards.timeoutMs (default 30000), an empty 502 when the client aborted before the forward started.
• A 2xx from an HTTP forward (upgrades excluded) on an overwritten-listed id clears the marker at or below the served epoch.

Attestation and headersCopy Anchor Link

Every inbound Pinboard-* header is stripped before forwarding. X-Forwarded-For is replaced with the observed peer address. Hop-by-hop headers are dropped. Responses lose Pinboard-* and hop-by-hop headers and gain X-Accel-Buffering: no.

• Pinboard-Attestation: set on every forwarded request, _wake, and _pinned/stats call. Compact JWS, header {"alg":"HS256","typ":"pinboard+jwt"}, signed with the worker's registered secret.
  • Claims: aud = the worker's sessionId (in-process cluster calls use the worker's local audience), iat, exp = iat + 30, caller? ({ns}/{id} of an attested calling instance), depth?, acct?.
  • acct? = { sub, scopes?, claims } of the data-plane credential with pinboard:* scopes and the pinboard claim removed.
  • The worker verifies with 30 s clock skew and answers 403 {"detail":"invalid attestation"} or WebSocket close 1008. _pinned/challenge also accepts Pinboard-Proxy-Secret.
• Pinboard-Epoch: the placement epoch of the resolved row, on forwards and _wake.
• Pinboard-Proxy-Secret: the worker's secret, only on the _pinned/challenge probe.
  • The worker answers {"value": …}, 404 unknown challenge for a stale challenge, 403 invalid proxy secret without the secret or a valid attestation.
• Caller attestation, inbound from an instance: Pinboard-From: {ns}/{id}, Pinboard-Session, Pinboard-Session-Secret, Pinboard-Subrequest-Depth (≤ 9 digits).
  • Verified against the session's secret and the caller's placement row.
  • Failures are 403 {"error"}: caller attestation requires session credentials, unknown session, invalid session secret, malformed caller claim, caller placement mismatch, malformed subrequest depth.
  • A depth over 50 is 508 {"error":"subrequest depth exceeded"}.
• A verified caller bypasses passport; the target must lie inside the caller's registration (its hostname and envs[ns]), else 403 {"error":"target outside caller scope"}.
• Data-plane passport runs only when namespaces rules are configured.
  • Failures: 401 {"error":"invalid credential" | "authentication required"}, 403 {"error":"namespace not allowed" | "insufficient scope"}.