Signatures quoted from source. Usage: Author a PinnedAPI, Run a fleet. Wire: pinboard registry. Related: pinboard, trays and durability.
Entry points
| Import from | Exports |
|---|---|
pinned | PinnedAPI, PinnedContext, AuthRejected, auth_result, PinnedWorker, Host, Lease, InstanceLease, attestation, Attestation, Domain, RealDomain, PinnedHost, Hostable, HostableContext, route, plus trays and durability, observable, Observable |
pinned.metrics | pinned_* collectors |
pinned.testing | instance, pinned_host, pinned_app, client_for, pinned_client, wake_until_ok, ASGIStream, BarrierStateStore |
pinned.pinboard.wire | invalid_id_reason |
pinned.hosting, pinned.pinboard, pinned.trays | LocalDirectory (the PinnedHost.directory type), routes_of (RouteSpec from pinned.hosting.routing); Cluster (the self.cluster transport), RegistrationClient (worker.registration; default_headroom from pinned.pinboard.registration); ABSENT, Admission, activate_trays, close_trays |
pinned.api
PinnedAPI
class PinnedAPI:
def __init__(self, ctx: HostableContext) -> None
async def authorize(self, conn: HTTPConnection) -> Any; async def lifespan(self) -> AsyncIterator[None]; def stats(self) -> dict[str, Any] | None
id: str; lease: InstanceLease; domain: Domain; cluster: httpx.AsyncClient; frontier: Frontier; placement_epoch: int # properties
def abort(self) -> None; def create_task(self, coro: Coroutine[Any, Any, Any]) -> asyncio.Task[Any]Instance base class: override authorize and lifespan, add @route.* methods.
PinnedAPI.* | Shape |
|---|---|
authorize(conn) | gates every @route.* handler; raise AuthRejected to deny; a returned Starlette Response is the reply on an HTTP route (TypeError on a WebSocket route) and discards a cold instance like a denial; any other return value is the caller identity; runs per request, before lifespan on a cold instance |
lifespan(), stats() | setup before the single yield, teardown after, zero or two yields raising RuntimeError; per-instance row merged into GET /_pinned/stats; return a dict: the base None breaks that route once an instance is live |
cluster | httpx.AsyncClient addressing other instances by /<ns>/<id>/... with this instance attested as caller; RuntimeError outside a PinnedHost, on an unplaced target without a pinboard, without a live pinboard session, or past a call depth of 50 |
frontier | the instance effect log; await self.frontier.wait() is the durable receipt; RuntimeError without a state store |
placement_epoch | the pinboard-epoch header's value, 0 without a pinboard; follows an in-place epoch advance; RuntimeError without a state store |
create_task(coro) | asyncio.Task bound to the instance's lifetime; blocks idle eviction while it runs |
PinnedContext
class PinnedContext:
def __init__(self, id: str, *, lease: InstanceLease, abort: Callable[[], None], create_task: Callable[[Coroutine[Any, Any, Any]], "asyncio.Task[Any]"], durability: InstanceDurability | None = None, cluster: httpx.AsyncClient | None = None, domain: Domain) -> NoneThe concrete HostableContext; with a durability it sets durability.frontier = Frontier(durability, self.abort, self.create_task).
AuthRejected
class AuthRejected(Exception):
def __init__(self, message: str, *, status_code: Literal[401, 403] = 401, payload: Any = None, code: str | None = None)
message: str; status_code: int; payload: Any; code: str | NoneDeliberate route denial: HTTP answers status_code with {"message": ...} or the exact payload. A WebSocket sends the payload as one JSON text frame (accepted sockets only) and closes 1008 with message. ValueError when status_code is not exactly 401 or 403. TypeError when message or code is not a string.
auth_result
async def auth_result(conn: HTTPConnection) -> AnyThe current route resolver's return value, for Depends(auth_result). RuntimeError outside a pinned route.
pinned.pinboard
PinnedWorker
class PinnedWorker(APIRouter):
def __init__(self, *, hosts: Mapping[str, type[PinnedAPI] | Host] | None = None, ..., **kwargs: Any) -> None # keyword parameters below; kwargs reach APIRouter
def host(self, namespace: str, api: type[PinnedAPI] | Host, **options: Any) -> PinnedHost
pinboard_url; advertise_url; hostname; env; takeover; epoch; state_store; domain: Domain
dispatch_slots: asyncio.BoundedSemaphore; lease: Lease; registration: RegistrationClient | None; secret: str # the registration's secret, or a fresh 32-byte hex in local mode
ready: bool; superseded: bool; draining: bool; drained: bool # properties: registered (control plane mode) and inside the lease; displaced by a higher-epoch worker
def abort_all(self) -> None
async def drain(self, deadline: float | None) -> None
def install_sigterm_drain(self, deadline: float = 30.0, sig: int = signal.SIGTERM) -> None # needs a running loop; drains, then hands sig to a handler installed with loop.add_signal_handlerOne pinboard registration (one client id, lease, heartbeat) shared by every attached PinnedHost. The only router an app mounts: app.include_router(worker) serves every host and merges each host's lifespan. Mounts GET /_pinned/stats and, in control plane mode, GET /_pinned/challenge, both behind the proxy secret or a valid attestation.
| Name | Type | Default | Meaning |
|---|---|---|---|
hosts | Mapping[str, type[PinnedAPI] | Host] | None | None | namespace to class (or Host); each entry is a PinnedHost served by this router |
pinboard_url | str | None | PINBOARD_URL | control plane URL; unset is local mode; requires token and advertise_url |
token | str | None | PINBOARD_TOKEN | registry bearer token |
advertise_url | str | None | ADVERTISE_URL | URL pinboard forwards to; <advertise_url>/_pinned/* reaches this router |
hostname | str | None | PINNED_HOSTNAME | stable identity for hostname-aware fleets |
env | str | None | None | the worker's environment; requires hostname |
takeover | Literal["wind_down", "deny"] | None | None | what a request from another hostname does to an id placed here; None leaves pinboard's "wind_down"; requires hostname |
epoch | int | 0 | deploy epoch, >= 0; a higher epoch supersedes a lower one |
state_store | StateStore | None | None | durable KV backing tray storage, shared by every attached host |
max_concurrent_dispatches | int | 256 | tray dispatches running at once across every host, >= 1 (dispatch_slots) |
heartbeat_interval | float | 3.0 | seconds between heartbeats |
lease_buffer | float | 5.0 | safety margin before the re-place deadline (lease.configure(buffer=)) |
headroom | Callable[[], int] | default_headroom | probe behind the headroom figure on every register and heartbeat; default_headroom is cgroup limit minus usage, capped by psutil available memory (1024 MB without either) |
domain | Domain | None | None | clock/task substrate; RealDomain() when omitted |
drain(deadline) and install_sigterm_drain(deadline): Wind-down and drain. worker.registration.client_id is clientId in GET /registry/workers and the target of operator drain.
pinboard_url,token,advertise_url,hostname: an argument overrides its environment variable; an empty variable counts as unset, except an emptyPINBOARD_URL, which raisesValueError.tokenoradvertise_urlwithout a pinboard URL raisesValueError.- No pinboard URL is local mode, which is dev mode: it logs a
DEV MODEwarning and raisesRuntimeErrorwhenNODE_ENV,ENVIRONMENTorAPP_ENVisproduction/prod, unlessPINBOARD_ALLOW_DEV_IN_PRODUCTION=1. worker.host(namespace, api, **options)adds a host and returns itsPinnedHost;optionsoverride aHost's. Call it beforeapp.include_router(worker): inclusion copies the routes.ValueErroron the argument rules above. Attaching a host raisesRuntimeErrorafter registration andValueErroron a duplicate namespace.
Host
class Host:
def __init__(self, cls: type[PinnedAPI], **options: Any) -> None
cls; optionsA PinnedAPI subclass with per-namespace PinnedHost keywords (env, idle_timeout, max_instances, ...): PinnedWorker(hosts={"/ns": Host(Api, idle_timeout=60)}).
Lease
class Lease:
def __init__(self, *, domain: Domain) -> None
healthy: bool; expires_at: datetime | None; expires_at_monotonic: float | None; expired: bool; buffer: float # properties; buffer default 5.0
ttl: float | None; flaps: int
def cap(self, deadline: float | None) -> None # monotonic ceiling renewals may not pass; None lifts it
def renewed(self, ttl: float, *, anchor: float | None = None) -> None # ack: runs until anchor (default now) + ttl
def configure(self, *, buffer: float | None = None, on_unhealthy: Callable[[], None] | None = None, on_restored: Callable[[], None] | None = None) -> None
async def close(self) -> NoneWorker lease health anchored to heartbeat acks: healthy until expires_at - buffer passes without an ack (then on_unhealthy, flaps += 1). The host refuses new activations while unhealthy and every request past expires_at. A fresh ack restores. Without a control plane expires_at stays None and healthy True. configure(buffer=) raises ValueError when not positive.
InstanceLease
class InstanceLease:
def __init__(self, worker: Lease, eviction_deadline_monotonic: Callable[[], float | None], *, winding_down: Callable[[], bool] = lambda: False, wind_down: Callable[[float | None], None] | None = None) -> None
healthy: bool; expires_at: datetime | None; expired: bool; buffer: float; eviction_deadline: datetime | None; winding_down: bool # properties
def wind_down(self, deadline: float | None = None) -> None
on_wind_down: Callable[[datetime | None], None] | None; on_idle: Callable[[], None] | None
def configure(self, *, buffer=None, on_unhealthy=None, on_restored=None, on_wind_down: Callable[[datetime | None], None] | None = None, on_idle: Callable[[], None] | None = None) -> Noneself.lease on an instance: the worker lease plus winding_down and eviction_deadline, the earliest local instance, operator instance, local pod, or operator worker deadline.
wind_down()requests cooperative eviction;deadlinesets a local execution deadline in seconds. Repeated calls preserve or tighten the deadline.ValueErrorfor a deadline other thanNoneor a finite, nonnegative number; booleans are rejected.await self.eviction_checkpoint()raisesEvictionRequested, anasyncio.CancelledErrorsubclass, whilewinding_downis true.- Eviction finishes task cleanup and the durability frontier before releasing placement.
attestation
def attestation(conn: HTTPConnection) -> Attestation.Instance | Attestation.Account | NoneThe verified caller of this request; None for anonymous traffic.
Attestation.Instance, Attestation.Account
@dataclass(frozen=True, match_args=False)
class Attestation.Instance:
ns: str; id: str; depth: int = 0
__match_args__ = ("ns", "id")
@dataclass(frozen=True, match_args=False)
class Attestation.Account:
sub: str; claims: Mapping[str, Any] = field(default_factory=dict); scopes: frozenset[str] = frozenset()
__match_args__ = ("sub",)Attested calling instance (matchable as Attestation.Instance(ns, id)) and passport-verified principal (Attestation.Account(sub)).
Ids
def pinned.pinboard.wire.invalid_id_reason(id: str) -> str | NoneWhy an id is rejected, None when valid: non-empty, at most 256 characters, [A-Za-z0-9._~-] only. A rejected id is a 400 (WebSocket close 1008).
pinned.domain
Domain, RealDomain
@runtime_checkable
class Domain(Protocol):
def now(self) -> float; def monotonic(self) -> float; def token_hex(self, nbytes: int) -> str; def rng(self, label: str) -> random.Random
async def sleep(self, seconds: float) -> None; async def wait(self, event: asyncio.Event, timeout: float) -> bool
def spawn(self, coro: Coroutine[Any, Any, Any]) -> asyncio.Task[Any]
class RealDomain:
def __init__(self, *, seed: int = 0) -> NoneAmbient primitives every clock, sleep, task spawn and random draw goes through. RealDomain is the process (system clocks, running loop, one RNG substream per label off seed). TypeError on a non-int seed. ValueError on an empty rng label.
pinned.hosting
PinnedHost
class PinnedHost(APIRouter):
def __init__(self, cls: type[PinnedAPI], *, worker: PinnedWorker, namespace: str, ...) -> None # keyword parameters below
cls; worker; namespace; env; domain: Domain; directory: LocalDirectory
lease: Lease; registration: RegistrationClient | None; ready: bool; draining: bool; drained: bool # properties; the worker's, drained = draining with no live instances
def wind_down(self, id: str, deadline: float | None) -> None
async def lifespan(self, app: Any = None) -> AsyncIterator[None] # async context manager: starts the sweep; closes instances, the cluster client, then the workerWhat each PinnedWorker(hosts=...) entry and worker.host(...) builds; construct it directly only for advanced use and tests. One live cls instance per id at <namespace>/{id}/.... With a worker state_store, POST <namespace>/{id}/_wake answers 200 {"ok": true} once resident and 202 {"activating": true} while hydrating.
| Name | Type | Default | Meaning |
|---|---|---|---|
cls | type[PinnedAPI] | required | instance class |
worker | PinnedWorker | required | registration, lease, and state store shared with the other hosts |
namespace | str | required | '/' plus one path segment: route prefix and durability/attestation namespace |
env | str | None | None | data identity overriding the worker's; requires PinnedWorker(hostname=...) |
idle_timeout | float | 300.0 | seconds without in-flight work before eviction |
release_placement_on_idle | bool | True | False keeps the placement row and epoch fence across an idle eviction: the next request re-activates the id on this worker at the same epoch, and a request racing the eviction waits for the close; abort, wind-down, drain, and shutdown still release, and a waiting request answers 503 when one lands mid-close |
detached_timeout | float | 86_400.0 | seconds without an open stream or WebSocket before a deadline=0 wind-down; positive |
sweep_interval | float | 5.0 | seconds between idle sweeps |
max_instances | int | 10_000 | live-instance cap; >= 1 |
max_concurrent_activations | int | 2 | lifespan setups running at once, the rest queue; >= 1 |
activation_queue_deadline | float | 1.0 | seconds an attach waits on the activation queue before the retryable 503 (Retry-After: 1); positive finite |
wind_down(id, deadline) serves until the effective deadline, evicts on idle, aborts at the deadline. 0 evicts now, None revokes a pending wind-down. ValueError on a non-finite deadline.
TypeError when cls is not a PinnedAPI subclass, worker not a PinnedWorker, release_placement_on_idle not a bool, or activation_queue_deadline not numeric. ValueError on a malformed namespace, env without a worker hostname, or an out-of-range max_instances, max_concurrent_activations, activation_queue_deadline, or detached_timeout.
Hostable, HostableContext
@runtime_checkable
class HostableContext(Protocol):
id: str; lease: InstanceLease; durability: InstanceDurability | None; cluster: httpx.AsyncClient | None; domain: Domain # properties
def abort(self) -> None; def create_task(self, coro: Coroutine[Any, Any, Any]) -> asyncio.Task[Any]
@runtime_checkable
class Hostable(Protocol):
def lifespan(self) -> AsyncIterator[None]The host/instance seam: the host builds an instance from a HostableContext, drives its lifespan (yield exactly once), and mounts its class's @route.* methods. PinnedContext and PinnedAPI are the concrete pair.
route
class route:
get / post / put / delete / patch / websocket = (path: str, *, auth: AuthResolver | None | object = DEFAULT_AUTH) -> Callable[[F], F]Marks an async def instance method as a handler at <namespace>/{id}<path>. Stacked decorators register every spec.
route.* | Shape |
|---|---|
path | starts with /; ValueError otherwise |
auth | omitted calls authorize(conn); an async def instance method (first parameter self) calls that instead; None makes the route public; anything else is a TypeError |
| handler | async def with self first; a parameter named id or pinned_record is a TypeError |
pinned.metrics
Metrics
Registered on the prometheus_client default registry when the package is importable; no-ops without it. One worker process per series; the app serves them (Observability).
| Metric | Labels |
|---|---|
pinned_instance_activations_total, pinned_instance_evictions_total, pinned_instance_aborts_total (counters) | none |
pinned_active_instances, pinned_lease_healthy, pinned_registered (gauges) | none |
pinned_registration_failures_total, pinned_duplicate_quits_total (counters) | none |
pinned_request_failures_total | phase, transport |
pinned_callback_failures_total | hook |
pinned_shutdown_failures_total | phase |
pinned_control_plane_failures_total | operation, reason |
pinned_request_duration_seconds (histogram) | namespace |
pinned.testing
instance, pinned_host, pinned_app, client_for, pinned_client
async with instance(cls: type[T], id: str = "test", *, abort: Callable[[], None] = lambda: None, state_store: StateStore | None = None) as api: ...
def pinned_host(cls: type[T], *, namespace: str = "/threads", **kwargs: Any) -> PinnedHost
async with pinned_app(router: PinnedHost) as app: ... # FastAPI
def client_for(app: FastAPI, headers: dict | None = None) -> httpx.AsyncClient
async with pinned_client(router: PinnedHost, *, attested: bool = True) as client: ...
async def wake_until_ok(client: httpx.AsyncClient, url: str, *, timeout: float = 5.0) -> Noneinstance: one live instance outside a host overstate_store(a freshMemoryStateStorewhen omitted). Lifespan entered and trays activated on the way in, both closed on the way out, exceptions included.TypeErrorfor a non-PinnedAPIclass,ValueErrorfor an invalid id.pinned_host: a host on a freshPinnedWorker()unlessworker=is passed; the rest ofkwargsreachesPinnedHost.pinned_app: the host and its worker mounted on a freshFastAPIwith the lifespan running.pinned_client:pinned_app+client_for, sending a valid attestation when the host expects one.wake_until_ok: polls_wakelike the sweeper until200 {"ok": true}.
ASGIStream, BarrierStateStore
async with ASGIStream(app, path: str, method: str = "GET") as stream:
stream.status: int | None
await stream.next_chunk(timeout: float = 5) -> bytes | None # None ends the body
await stream.drain(timeout: float = 5) -> None
class BarrierStateStore(MemoryStateStore):
parked: asyncio.Event
def hold(self) -> None; def release(self) -> NoneIncremental ASGI response reader (httpx.ASGITransport buffers full bodies); a MemoryStateStore whose hold() parks every mutation before it lands (parked fires while one waits) until release(). Also re-exported from pinned.sim.store: SimStateStore, SimOp, SimOutcome, SIM_OUTCOMES, StoreFailed.