Skip to content

Signatures quoted from source. Usage: Author a PinnedAPI, Run a fleet. Wire: pinboard registry. Related: pinboard, trays and durability.

Entry points

Import fromExports
pinnedPinnedAPI, PinnedContext, AuthRejected, auth_result, PinnedWorker, Host, Lease, InstanceLease, attestation, Attestation, Domain, RealDomain, PinnedHost, Hostable, HostableContext, route, plus trays and durability, observable, Observable
pinned.metricspinned_* collectors
pinned.testinginstance, pinned_host, pinned_app, client_for, pinned_client, wake_until_ok, ASGIStream, BarrierStateStore
pinned.pinboard.wireinvalid_id_reason
pinned.hosting, pinned.pinboard, pinned.traysLocalDirectory (the PinnedHost.directory type), routes_of (RouteSpec from pinned.hosting.routing); Cluster (the self.cluster transport), RegistrationClient (worker.registration; default_headroom from pinned.pinboard.registration); ABSENT, Admission, activate_trays, close_trays

pinned.api

PinnedAPI

class PinnedAPI:
    def __init__(self, ctx: HostableContext) -> None
    async def authorize(self, conn: HTTPConnection) -> Any;  async def lifespan(self) -> AsyncIterator[None];  def stats(self) -> dict[str, Any] | None
    id: str; lease: InstanceLease; domain: Domain; cluster: httpx.AsyncClient; frontier: Frontier; placement_epoch: int   # properties
    def abort(self) -> None;  def create_task(self, coro: Coroutine[Any, Any, Any]) -> asyncio.Task[Any]

Instance base class: override authorize and lifespan, add @route.* methods.

PinnedAPI.*Shape
authorize(conn)gates every @route.* handler; raise AuthRejected to deny; a returned Starlette Response is the reply on an HTTP route (TypeError on a WebSocket route) and discards a cold instance like a denial; any other return value is the caller identity; runs per request, before lifespan on a cold instance
lifespan(), stats()setup before the single yield, teardown after, zero or two yields raising RuntimeError; per-instance row merged into GET /_pinned/stats; return a dict: the base None breaks that route once an instance is live
clusterhttpx.AsyncClient addressing other instances by /<ns>/<id>/... with this instance attested as caller; RuntimeError outside a PinnedHost, on an unplaced target without a pinboard, without a live pinboard session, or past a call depth of 50
frontierthe instance effect log; await self.frontier.wait() is the durable receipt; RuntimeError without a state store
placement_epochthe pinboard-epoch header's value, 0 without a pinboard; follows an in-place epoch advance; RuntimeError without a state store
create_task(coro)asyncio.Task bound to the instance's lifetime; blocks idle eviction while it runs

PinnedContext

class PinnedContext:
    def __init__(self, id: str, *, lease: InstanceLease, abort: Callable[[], None], create_task: Callable[[Coroutine[Any, Any, Any]], "asyncio.Task[Any]"], durability: InstanceDurability | None = None, cluster: httpx.AsyncClient | None = None, domain: Domain) -> None

The concrete HostableContext; with a durability it sets durability.frontier = Frontier(durability, self.abort, self.create_task).

AuthRejected

class AuthRejected(Exception):
    def __init__(self, message: str, *, status_code: Literal[401, 403] = 401, payload: Any = None, code: str | None = None)
    message: str; status_code: int; payload: Any; code: str | None

Deliberate route denial: HTTP answers status_code with {"message": ...} or the exact payload. A WebSocket sends the payload as one JSON text frame (accepted sockets only) and closes 1008 with message. ValueError when status_code is not exactly 401 or 403. TypeError when message or code is not a string.

auth_result

async def auth_result(conn: HTTPConnection) -> Any

The current route resolver's return value, for Depends(auth_result). RuntimeError outside a pinned route.

pinned.pinboard

PinnedWorker

class PinnedWorker(APIRouter):
    def __init__(self, *, hosts: Mapping[str, type[PinnedAPI] | Host] | None = None, ..., **kwargs: Any) -> None   # keyword parameters below; kwargs reach APIRouter
    def host(self, namespace: str, api: type[PinnedAPI] | Host, **options: Any) -> PinnedHost
    pinboard_url; advertise_url; hostname; env; takeover; epoch; state_store; domain: Domain
    dispatch_slots: asyncio.BoundedSemaphore; lease: Lease; registration: RegistrationClient | None; secret: str   # the registration's secret, or a fresh 32-byte hex in local mode
    ready: bool; superseded: bool; draining: bool; drained: bool   # properties: registered (control plane mode) and inside the lease; displaced by a higher-epoch worker
    def abort_all(self) -> None
    async def drain(self, deadline: float | None) -> None
    def install_sigterm_drain(self, deadline: float = 30.0, sig: int = signal.SIGTERM) -> None   # needs a running loop; drains, then hands sig to a handler installed with loop.add_signal_handler

One pinboard registration (one client id, lease, heartbeat) shared by every attached PinnedHost. The only router an app mounts: app.include_router(worker) serves every host and merges each host's lifespan. Mounts GET /_pinned/stats and, in control plane mode, GET /_pinned/challenge, both behind the proxy secret or a valid attestation.

NameTypeDefaultMeaning
hostsMapping[str, type[PinnedAPI] | Host] | NoneNonenamespace to class (or Host); each entry is a PinnedHost served by this router
pinboard_urlstr | NonePINBOARD_URLcontrol plane URL; unset is local mode; requires token and advertise_url
tokenstr | NonePINBOARD_TOKENregistry bearer token
advertise_urlstr | NoneADVERTISE_URLURL pinboard forwards to; <advertise_url>/_pinned/* reaches this router
hostnamestr | NonePINNED_HOSTNAMEstable identity for hostname-aware fleets
envstr | NoneNonethe worker's environment; requires hostname
takeoverLiteral["wind_down", "deny"] | NoneNonewhat a request from another hostname does to an id placed here; None leaves pinboard's "wind_down"; requires hostname
epochint0deploy epoch, >= 0; a higher epoch supersedes a lower one
state_storeStateStore | NoneNonedurable KV backing tray storage, shared by every attached host
max_concurrent_dispatchesint256tray dispatches running at once across every host, >= 1 (dispatch_slots)
heartbeat_intervalfloat3.0seconds between heartbeats
lease_bufferfloat5.0safety margin before the re-place deadline (lease.configure(buffer=))
headroomCallable[[], int]default_headroomprobe behind the headroom figure on every register and heartbeat; default_headroom is cgroup limit minus usage, capped by psutil available memory (1024 MB without either)
domainDomain | NoneNoneclock/task substrate; RealDomain() when omitted

drain(deadline) and install_sigterm_drain(deadline): Wind-down and drain. worker.registration.client_id is clientId in GET /registry/workers and the target of operator drain.

  • pinboard_url, token, advertise_url, hostname: an argument overrides its environment variable; an empty variable counts as unset, except an empty PINBOARD_URL, which raises ValueError.
  • token or advertise_url without a pinboard URL raises ValueError.
  • No pinboard URL is local mode, which is dev mode: it logs a DEV MODE warning and raises RuntimeError when NODE_ENV, ENVIRONMENT or APP_ENV is production/prod, unless PINBOARD_ALLOW_DEV_IN_PRODUCTION=1.
  • worker.host(namespace, api, **options) adds a host and returns its PinnedHost; options override a Host's. Call it before app.include_router(worker): inclusion copies the routes.
  • ValueError on the argument rules above. Attaching a host raises RuntimeError after registration and ValueError on a duplicate namespace.

Host

class Host:
    def __init__(self, cls: type[PinnedAPI], **options: Any) -> None
    cls; options

A PinnedAPI subclass with per-namespace PinnedHost keywords (env, idle_timeout, max_instances, ...): PinnedWorker(hosts={"/ns": Host(Api, idle_timeout=60)}).

Lease

class Lease:
    def __init__(self, *, domain: Domain) -> None
    healthy: bool; expires_at: datetime | None; expires_at_monotonic: float | None; expired: bool; buffer: float   # properties; buffer default 5.0
    ttl: float | None; flaps: int
    def cap(self, deadline: float | None) -> None                          # monotonic ceiling renewals may not pass; None lifts it
    def renewed(self, ttl: float, *, anchor: float | None = None) -> None   # ack: runs until anchor (default now) + ttl
    def configure(self, *, buffer: float | None = None, on_unhealthy: Callable[[], None] | None = None, on_restored: Callable[[], None] | None = None) -> None
    async def close(self) -> None

Worker lease health anchored to heartbeat acks: healthy until expires_at - buffer passes without an ack (then on_unhealthy, flaps += 1). The host refuses new activations while unhealthy and every request past expires_at. A fresh ack restores. Without a control plane expires_at stays None and healthy True. configure(buffer=) raises ValueError when not positive.

InstanceLease

class InstanceLease:
    def __init__(self, worker: Lease, eviction_deadline_monotonic: Callable[[], float | None], *, winding_down: Callable[[], bool] = lambda: False, wind_down: Callable[[float | None], None] | None = None) -> None
    healthy: bool; expires_at: datetime | None; expired: bool; buffer: float; eviction_deadline: datetime | None; winding_down: bool   # properties
    def wind_down(self, deadline: float | None = None) -> None
    on_wind_down: Callable[[datetime | None], None] | None; on_idle: Callable[[], None] | None
    def configure(self, *, buffer=None, on_unhealthy=None, on_restored=None, on_wind_down: Callable[[datetime | None], None] | None = None, on_idle: Callable[[], None] | None = None) -> None

self.lease on an instance: the worker lease plus winding_down and eviction_deadline, the earliest local instance, operator instance, local pod, or operator worker deadline.

  • wind_down() requests cooperative eviction; deadline sets a local execution deadline in seconds. Repeated calls preserve or tighten the deadline.
  • ValueError for a deadline other than None or a finite, nonnegative number; booleans are rejected.
  • await self.eviction_checkpoint() raises EvictionRequested, an asyncio.CancelledError subclass, while winding_down is true.
  • Eviction finishes task cleanup and the durability frontier before releasing placement.

attestation

def attestation(conn: HTTPConnection) -> Attestation.Instance | Attestation.Account | None

The verified caller of this request; None for anonymous traffic.

Attestation.Instance, Attestation.Account

@dataclass(frozen=True, match_args=False)
class Attestation.Instance:
    ns: str; id: str; depth: int = 0
    __match_args__ = ("ns", "id")

@dataclass(frozen=True, match_args=False)
class Attestation.Account:
    sub: str; claims: Mapping[str, Any] = field(default_factory=dict); scopes: frozenset[str] = frozenset()
    __match_args__ = ("sub",)

Attested calling instance (matchable as Attestation.Instance(ns, id)) and passport-verified principal (Attestation.Account(sub)).

Ids

def pinned.pinboard.wire.invalid_id_reason(id: str) -> str | None

Why an id is rejected, None when valid: non-empty, at most 256 characters, [A-Za-z0-9._~-] only. A rejected id is a 400 (WebSocket close 1008).

pinned.domain

Domain, RealDomain

@runtime_checkable
class Domain(Protocol):
    def now(self) -> float;  def monotonic(self) -> float;  def token_hex(self, nbytes: int) -> str;  def rng(self, label: str) -> random.Random
    async def sleep(self, seconds: float) -> None;  async def wait(self, event: asyncio.Event, timeout: float) -> bool
    def spawn(self, coro: Coroutine[Any, Any, Any]) -> asyncio.Task[Any]

class RealDomain:
    def __init__(self, *, seed: int = 0) -> None

Ambient primitives every clock, sleep, task spawn and random draw goes through. RealDomain is the process (system clocks, running loop, one RNG substream per label off seed). TypeError on a non-int seed. ValueError on an empty rng label.

pinned.hosting

PinnedHost

class PinnedHost(APIRouter):
    def __init__(self, cls: type[PinnedAPI], *, worker: PinnedWorker, namespace: str, ...) -> None   # keyword parameters below
    cls; worker; namespace; env; domain: Domain; directory: LocalDirectory
    lease: Lease; registration: RegistrationClient | None; ready: bool; draining: bool; drained: bool   # properties; the worker's, drained = draining with no live instances
    def wind_down(self, id: str, deadline: float | None) -> None
    async def lifespan(self, app: Any = None) -> AsyncIterator[None]   # async context manager: starts the sweep; closes instances, the cluster client, then the worker

What each PinnedWorker(hosts=...) entry and worker.host(...) builds; construct it directly only for advanced use and tests. One live cls instance per id at <namespace>/{id}/.... With a worker state_store, POST <namespace>/{id}/_wake answers 200 {"ok": true} once resident and 202 {"activating": true} while hydrating.

NameTypeDefaultMeaning
clstype[PinnedAPI]requiredinstance class
workerPinnedWorkerrequiredregistration, lease, and state store shared with the other hosts
namespacestrrequired'/' plus one path segment: route prefix and durability/attestation namespace
envstr | NoneNonedata identity overriding the worker's; requires PinnedWorker(hostname=...)
idle_timeoutfloat300.0seconds without in-flight work before eviction
release_placement_on_idleboolTrueFalse keeps the placement row and epoch fence across an idle eviction: the next request re-activates the id on this worker at the same epoch, and a request racing the eviction waits for the close; abort, wind-down, drain, and shutdown still release, and a waiting request answers 503 when one lands mid-close
detached_timeoutfloat86_400.0seconds without an open stream or WebSocket before a deadline=0 wind-down; positive
sweep_intervalfloat5.0seconds between idle sweeps
max_instancesint10_000live-instance cap; >= 1
max_concurrent_activationsint2lifespan setups running at once, the rest queue; >= 1
activation_queue_deadlinefloat1.0seconds an attach waits on the activation queue before the retryable 503 (Retry-After: 1); positive finite

wind_down(id, deadline) serves until the effective deadline, evicts on idle, aborts at the deadline. 0 evicts now, None revokes a pending wind-down. ValueError on a non-finite deadline.

TypeError when cls is not a PinnedAPI subclass, worker not a PinnedWorker, release_placement_on_idle not a bool, or activation_queue_deadline not numeric. ValueError on a malformed namespace, env without a worker hostname, or an out-of-range max_instances, max_concurrent_activations, activation_queue_deadline, or detached_timeout.

Hostable, HostableContext

@runtime_checkable
class HostableContext(Protocol):
    id: str; lease: InstanceLease; durability: InstanceDurability | None; cluster: httpx.AsyncClient | None; domain: Domain   # properties
    def abort(self) -> None;  def create_task(self, coro: Coroutine[Any, Any, Any]) -> asyncio.Task[Any]

@runtime_checkable
class Hostable(Protocol):
    def lifespan(self) -> AsyncIterator[None]

The host/instance seam: the host builds an instance from a HostableContext, drives its lifespan (yield exactly once), and mounts its class's @route.* methods. PinnedContext and PinnedAPI are the concrete pair.

route

class route:
    get / post / put / delete / patch / websocket = (path: str, *, auth: AuthResolver | None | object = DEFAULT_AUTH) -> Callable[[F], F]

Marks an async def instance method as a handler at <namespace>/{id}<path>. Stacked decorators register every spec.

route.*Shape
pathstarts with /; ValueError otherwise
authomitted calls authorize(conn); an async def instance method (first parameter self) calls that instead; None makes the route public; anything else is a TypeError
handlerasync def with self first; a parameter named id or pinned_record is a TypeError

pinned.metrics

Metrics

Registered on the prometheus_client default registry when the package is importable; no-ops without it. One worker process per series; the app serves them (Observability).

MetricLabels
pinned_instance_activations_total, pinned_instance_evictions_total, pinned_instance_aborts_total (counters)none
pinned_active_instances, pinned_lease_healthy, pinned_registered (gauges)none
pinned_registration_failures_total, pinned_duplicate_quits_total (counters)none
pinned_request_failures_totalphase, transport
pinned_callback_failures_totalhook
pinned_shutdown_failures_totalphase
pinned_control_plane_failures_totaloperation, reason
pinned_request_duration_seconds (histogram)namespace

pinned.testing

instance, pinned_host, pinned_app, client_for, pinned_client

async with instance(cls: type[T], id: str = "test", *, abort: Callable[[], None] = lambda: None, state_store: StateStore | None = None) as api: ...
def pinned_host(cls: type[T], *, namespace: str = "/threads", **kwargs: Any) -> PinnedHost
async with pinned_app(router: PinnedHost) as app: ...            # FastAPI
def client_for(app: FastAPI, headers: dict | None = None) -> httpx.AsyncClient
async with pinned_client(router: PinnedHost, *, attested: bool = True) as client: ...
async def wake_until_ok(client: httpx.AsyncClient, url: str, *, timeout: float = 5.0) -> None
  • instance: one live instance outside a host over state_store (a fresh MemoryStateStore when omitted). Lifespan entered and trays activated on the way in, both closed on the way out, exceptions included. TypeError for a non-PinnedAPI class, ValueError for an invalid id.
  • pinned_host: a host on a fresh PinnedWorker() unless worker= is passed; the rest of kwargs reaches PinnedHost.
  • pinned_app: the host and its worker mounted on a fresh FastAPI with the lifespan running.
  • pinned_client: pinned_app + client_for, sending a valid attestation when the host expects one.
  • wake_until_ok: polls _wake like the sweeper until 200 {"ok": true}.

ASGIStream, BarrierStateStore

async with ASGIStream(app, path: str, method: str = "GET") as stream:
    stream.status: int | None
    await stream.next_chunk(timeout: float = 5) -> bytes | None   # None ends the body
    await stream.drain(timeout: float = 5) -> None

class BarrierStateStore(MemoryStateStore):
    parked: asyncio.Event
    def hold(self) -> None;  def release(self) -> None

Incremental ASGI response reader (httpx.ASGITransport buffers full bodies); a MemoryStateStore whose hold() parks every mutation before it lands (parked fires while one waits) until release(). Also re-exported from pinned.sim.store: SimStateStore, SimOp, SimOutcome, SIM_OUTCOMES, StoreFailed.