Persist across restarts

Pick a state store, set it up, and read what survives a crash, a restart, a re-placement

With a persistent store, tray state outlives the process that wrote it. Placement across workers: run a fleet.

Pick a store

Trays (authoring) persist through a state store shared by every worker: PinnedWorker(state_store=...).

Store	Import	Rows live
`MemoryStateStore()`	`pinned.durability`	in the process; empties with it
`PostgresStateStore(dsn=...)`	`pinned.durability.postgres` (`postgres` extra)	in Postgres, read by every worker
your own	any object matching the `StateStore` protocol	wherever you put them: SQLite, files

Signatures: StateStore, PostgresStateStore.

Set up Postgres

await store.setup() once at boot. Two CREATE TABLE IF NOT EXISTS statements; every worker can run it, concurrently.
The tables are pinned_epoch (ns, id, epoch) and pinned_tray_index (ns, id, tray_ns, tray_id, marked, doc, env, hostname).
A tray is one row (ns, id, tray_ns, tray_id) -> (marked, doc): marked is true while work is in flight, doc is the whole durable payload.
Stores write single rows in order. apply_batch is the optional coalescing fast path.
An embedded pinboard reads the same pinned_tray_index table through new PostgresDurabilityStore(db), or MemoryDurabilityStore in tests, both from @assistant-ui/pinboard.

Connections:

Process	Pool	Statements
Worker	`pool_size` per process, default 8, dialed lazily	DDL for `setup()`, then reads and writes on both tables
Pinboard node	one `pg.Pool` per node carrying `PINBOARD_DURABILITY_POSTGRES_URL`	one `SELECT` paging `marked` rows of `pinned_tray_index`, from the `{0}:sweeper` lease holder only

What survives what

The table assumes a persistent store shared by the workers. A MemoryStateStore loses its rows when its process exits.

Event	Instance memory	Tray state	Where the id reactivates
Idle eviction	lost	kept	anywhere, on next request; `release_placement_on_idle=False`: this worker, at the kept epoch
Deploy / drain	lost	kept	the new deployment
Worker crash	lost	kept	another live worker, after the lease lapses; until then requests for ids placed on the dead worker answer 502 `bad gateway` (504 `upstream timeout`), text/plain
Lease expiry with the process alive	retained until the instance closes	kept	the worker refuses requests until renewal; pinboard may assign a new owner
Redis wipe	resident instances can remain	kept	workers re-register their live ids; duplicates close; placement and wake resume under the cold-start rules

On reactivation, an unfinished run is redelivered from its stored dispatch. ctx.resume is true when its input batch was durably confirmed; an unconfirmed batch is presented as input again. The executor continues from its checkpoint in resume mode (runs).

Lease and Redis recovery: when a worker dies.

Env and fleet isolation: durable objects.

Epochs and fencing

Four counters share the name:

Epoch	Scope	Minted	Fences
Placement	one `(env, namespace, id)` row	at every claim, monotonic per bucket; rides forwards as `Pinboard-Epoch`	store writes (`FenceLost`), releases, overwritten-marker deletes
Binding	`(hostname, namespace) → env`, `env → takeover`	`PinnedWorker(epoch=...)` at registration	equal or lower disagreeing registrations (`409`); displaced holders get `superseded`; displacement
Fleet license	the fleet config `{schema, shards, hostnameAware, epoch}`	operator config, overwritten by a higher epoch	a node whose config differs from the held license at an equal or lower epoch, or from any live member, fails `readyz`
Run	one run entry (Python machine)	bumped at each dispatch	`run/stop` with a differing `epoch` rejects `wrong-state`; runs spec

Every store op carries the caller's epoch: at or above the stored instance epoch it advances that epoch and executes, below it raises FenceLost and the stale holder aborts.
An instance with no stored epoch is at 0, as is the placement epoch in local mode.
Releases carry the epoch: a release for an older epoch is a no-op against a newer placement.

Sweep and wake

The pinboard sweeper (one node, every sweepIntervalMs) wakes marked trays whose worker died: POST <advertiseUrl><ns>/<id>/_wake with an attestation. 200 means resident; 202 means still hydrating and the sweeper re-checks after wakeTimeoutMs.
Pinboard reads the marked rows from PINBOARD_DURABILITY_POSTGRES_URL, the same database as the workers' store. Without it, a marked tray whose instance is gone waits for the next request for its id.
Failed wakes back off and park; knobs: DurabilityOptions.
Alarms (@assistant-ui/pinned): the sweeper pages the store's due ("pinned", "alarm") rows and fires each through POST <advertiseUrl><ns>/<id>/_alarm at the placement epoch, placing evicted ids on a live worker within one sweepIntervalMs. Resident instances fire on time from their owner's timer. A failed or interrupted run leaves the row due, so it fires again: ctx.alarm.
Local mode persists the same way through the worker's state_store; a tray resumes on the next activation of its id. Local mode has no placement, so run one worker per store: two workers share epoch 0 and both write the same rows.

Edit on GitHubllms.txt llms-full.txt

Persist across restarts

Pick a state store, set it up, and read what survives a crash, a restart, a re-placement

With a persistent store, tray state outlives the process that wrote it. Placement across workers: run a fleet.

Pick a store

Trays (authoring) persist through a state store shared by every worker: PinnedWorker(state_store=...).

Store	Import	Rows live
`MemoryStateStore()`	`pinned.durability`	in the process; empties with it
`PostgresStateStore(dsn=...)`	`pinned.durability.postgres` (`postgres` extra)	in Postgres, read by every worker
your own	any object matching the `StateStore` protocol	wherever you put them: SQLite, files

Signatures: StateStore, PostgresStateStore.

Set up Postgres

await store.setup() once at boot. Two CREATE TABLE IF NOT EXISTS statements; every worker can run it, concurrently.
The tables are pinned_epoch (ns, id, epoch) and pinned_tray_index (ns, id, tray_ns, tray_id, marked, doc, env, hostname).
A tray is one row (ns, id, tray_ns, tray_id) -> (marked, doc): marked is true while work is in flight, doc is the whole durable payload.
Stores write single rows in order. apply_batch is the optional coalescing fast path.
An embedded pinboard reads the same pinned_tray_index table through new PostgresDurabilityStore(db), or MemoryDurabilityStore in tests, both from @assistant-ui/pinboard.

Connections:

Process	Pool	Statements
Worker	`pool_size` per process, default 8, dialed lazily	DDL for `setup()`, then reads and writes on both tables
Pinboard node	one `pg.Pool` per node carrying `PINBOARD_DURABILITY_POSTGRES_URL`	one `SELECT` paging `marked` rows of `pinned_tray_index`, from the `{0}:sweeper` lease holder only

What survives what

The table assumes a persistent store shared by the workers. A MemoryStateStore loses its rows when its process exits.

Event	Instance memory	Tray state	Where the id reactivates
Idle eviction	lost	kept	anywhere, on next request; `release_placement_on_idle=False`: this worker, at the kept epoch
Deploy / drain	lost	kept	the new deployment
Worker crash	lost	kept	another live worker, after the lease lapses; until then requests for ids placed on the dead worker answer 502 `bad gateway` (504 `upstream timeout`), text/plain
Lease expiry with the process alive	retained until the instance closes	kept	the worker refuses requests until renewal; pinboard may assign a new owner
Redis wipe	resident instances can remain	kept	workers re-register their live ids; duplicates close; placement and wake resume under the cold-start rules

Lease and Redis recovery: when a worker dies.

Env and fleet isolation: durable objects.

Epochs and fencing

Four counters share the name:

Epoch	Scope	Minted	Fences
Placement	one `(env, namespace, id)` row	at every claim, monotonic per bucket; rides forwards as `Pinboard-Epoch`	store writes (`FenceLost`), releases, overwritten-marker deletes
Binding	`(hostname, namespace) → env`, `env → takeover`	`PinnedWorker(epoch=...)` at registration	equal or lower disagreeing registrations (`409`); displaced holders get `superseded`; displacement
Fleet license	the fleet config `{schema, shards, hostnameAware, epoch}`	operator config, overwritten by a higher epoch	a node whose config differs from the held license at an equal or lower epoch, or from any live member, fails `readyz`
Run	one run entry (Python machine)	bumped at each dispatch	`run/stop` with a differing `epoch` rejects `wrong-state`; runs spec

Every store op carries the caller's epoch: at or above the stored instance epoch it advances that epoch and executes, below it raises FenceLost and the stale holder aborts.
An instance with no stored epoch is at 0, as is the placement epoch in local mode.
Releases carry the epoch: a release for an older epoch is a no-op against a newer placement.

Sweep and wake

The pinboard sweeper (one node, every sweepIntervalMs) wakes marked trays whose worker died: POST <advertiseUrl><ns>/<id>/_wake with an attestation. 200 means resident; 202 means still hydrating and the sweeper re-checks after wakeTimeoutMs.
Pinboard reads the marked rows from PINBOARD_DURABILITY_POSTGRES_URL, the same database as the workers' store. Without it, a marked tray whose instance is gone waits for the next request for its id.
Failed wakes back off and park; knobs: DurabilityOptions.
Alarms (@assistant-ui/pinned): the sweeper pages the store's due ("pinned", "alarm") rows and fires each through POST <advertiseUrl><ns>/<id>/_alarm at the placement epoch, placing evicted ids on a live worker within one sweepIntervalMs. Resident instances fire on time from their owner's timer. A failed or interrupted run leaves the row due, so it fires again: ctx.alarm.
Local mode persists the same way through the worker's state_store; a tray resumes on the next activation of its id. Local mode has no placement, so run one worker per store: two workers share epoch 0 and both write the same rows.

Edit on GitHubllms.txt llms-full.txt

Persist across restarts

Pick a storeCopy Anchor Link

Set up PostgresCopy Anchor Link

What survives whatCopy Anchor Link

Epochs and fencingCopy Anchor Link

Sweep and wakeCopy Anchor Link

Pick a storeCopy Anchor Link

Set up PostgresCopy Anchor Link

What survives whatCopy Anchor Link

Epochs and fencingCopy Anchor Link

Sweep and wakeCopy Anchor Link

Persist across restarts

Pick a storeCopy Anchor Link

Set up PostgresCopy Anchor Link

What survives whatCopy Anchor Link

Epochs and fencingCopy Anchor Link

Sweep and wakeCopy Anchor Link

Pick a store

Set up Postgres

What survives what

Epochs and fencing

Sweep and wake

Pick a store

Set up Postgres

What survives what

Epochs and fencing

Sweep and wake

Pick a store

Set up Postgres

What survives what

Epochs and fencing

Sweep and wake