Durability and leases

This is the documentation for unreleased changes on main. For the released library, read v0.1.

Covers DESIGN.md §7 (persistence), §8 (worker, attempt, lease, deadline) and the parts of §9 that decide what happens to a tool call nobody finished. Read it before touching psych_runtime/runtime/worker.py or a Store adapter.

Supervision and execution are separate loops

A Worker does two jobs. It executes Runs, and it enforces the timeouts that stop a Run running forever. Those must not share a call stack.

The failure mode is specific. If the loop that executes an Attempt is also the loop that checks deadlines, then an Attempt that hangs on an unbounded await blocks the only code that could have noticed and killed it. A model stream that returned 200 and then went quiet does this. So does a sandbox call against a frozen container. The deadline becomes unreachable exactly when it is needed.

So the supervisor pass is bounded and touches storage only. It scans for expired leases, fires deadlines and force-settles. It starts Attempts detached and returns without awaiting any of them. A pass that throws must not end supervision, which means it arms its own successor before doing anything that can fail. On asyncio this mostly falls out of using separate tasks: a periodic supervisor task and per-Attempt tasks cannot block each other, because the event loop schedules them independently.

Claiming a Run

Claiming is one conditional write, not a read followed by a write. It checks that the Run is queued and is the runnable head of its queue key, meaning no earlier unsettled Run in the same family. It stamps the attempt id, owner and lease expiry, increments the attempt count, initialises the deadline when it is still unset, and returns the claimed row. One statement, one round trip, and two concurrent claims can never both win.

Two details are load-bearing:

Return the row from the same statement. The caller learns both whether it won and what it won, without a second read that could observe a different state.

Enforce ordering inside the conditional write. A "no earlier unsettled Run in this queue key" clause means a queued Run cannot be claimed out of order, and no separate serialisation mechanism is needed to get that.

On Postgres and MySQL this is UPDATE ... WHERE <state and ownership conditions> RETURNING. On DynamoDB it is a conditional UpdateItem over the same fields. The Store contract suite asserts the behaviour, not the SQL.

The lease and the deadline answer different questions

This is the point people get wrong, so it is stated plainly.

A lease answers "does some process still claim to own this Run?" It is renewed on a heartbeat while a Worker holds the Attempt, and its expiry is what lets a different Worker reclaim the Run. Renewal follows process liveness, because that is the question a lease asks. A hung Attempt in a live process therefore keeps renewing its lease, and that is correct: the process really is alive and really does still own the Run.

A deadline answers "has this Run run too long?" It is an absolute wall-clock value stamped once when the Run is claimed and never re-anchored, so a Run cannot dodge it by crashing repeatedly. The process holding the Attempt enforces it, by firing an abort signal and, after a grace window, force-settling regardless of what the lease says.

Deriving one from the other breaks both. Make renewal conditional on observable progress and a Run waiting on a legitimately slow model gets its lease expired and reclaimed out from under it. Drop the deadline and rely on the lease alone and a hung Attempt is never caught, because the expired-lease branch is unreachable while a live process keeps renewing.

Psych ships both, independently. psych_runtime/runtime/worker.py holds the mechanism and the reasoning at the place it happens.

The shipped numbers: a 30 second lease renewed every 10 seconds, so three heartbeats fit inside one lease and a single missed tick is survivable. A 60 second grace between the abort signal and force settlement. Ten attempts per Run before it is failed as unrecoverable.

Renewal is gated on the owner id. A lease already reclaimed by someone else silently no-ops the renewal rather than erroring, so a zombie process cannot clobber the new owner's lease.

Reclaim fences the old owner

Reclaiming an expired lease is a second conditional write, gated on the old attempt id. That does two things at once. It installs the new attempt id and lease atomically, and it fences out the old owner, whose later writes now target a stale attempt id and lose every subsequent conditional write. There is no separate fence token to manage.

Grace is anchored to the signal, not to the deadline

The grace window starts when the abort was first signalled, not when the deadline passed and not at "now" on each scan. If the supervisor pass was itself delayed past the deadline, a naive now >= deadline + grace check would abort and force-settle in the same breath, giving the model stream or the tool call no chance to unwind. Anchoring to the first signal guarantees a full grace window after the abort actually fires, whatever the scheduling jitter before it.

Force settlement writes through the ordinary path

Force settlement does not write a bespoke "killed" record. It re-enters the same settlement path a crash recovery uses. The only difference is that the hung Attempt's task is still technically alive in this process, so once the settlement write wins, the Worker orphans it: it drops the bookkeeping entry and rotates any cached writer handle, so a late write from the zombie fails only its own stale handle and never a successor's.

Orphan safety

What makes it safe to walk away from a hung Attempt is that every tool call is a pair of Records, never one combined write. A request Record before the call runs, a result Record after. An orphaned Attempt therefore leaves calls that are visibly incomplete rather than ambiguous.

The settling Worker walks the trailing tool batch and writes an explicit unknown-outcome result for every call still missing one. It never silently drops a call, never guesses a result, and never re-executes a tool on recovery. The distinction the settlement carries matters to the model:

• The call was announced but no request Record exists. Nothing ran. Say so, and say it is safe to retry.
• A request Record exists but no result. The call may have run and may have had effects. Say so, and say to retry only if the operation is read-only or idempotent, otherwise to verify external state first.

A settlement is an ordinary Record appended through the ordinary conditional-write path by the Worker that reclaimed the lease. It is never a reader-side patch. A read-only projection of "what this Run would look like once settled" is non-authoritative and must never stand in for the appended Record.

Runs that were never claimed

A queued Run can need settling too: its Version was removed, or admission keeps failing. That deadline anchors at admission time rather than at claim time, is computed from columns already on the row so it costs no extra read, and settles directly, because no Attempt was ever created and there is no lease to go through.

Stream idle timeout

A provider that returns 200 and then never sends another byte holds a turn open forever. The guard is a cap on the gap between chunks, and it has three properties worth stating because each is easy to get wrong.

The timer runs only while a read is outstanding. A slow consumer must not look like a silent source. In asyncio that means wrapping the per-chunk await (asyncio.wait_for around __anext__), not running an ambient connection-level timer between chunks.

The default is generous, five minutes. Reasoning models go quiet for minutes when neither keepalives nor reasoning deltas are streamed, and a false trip burns a turn retry.

Zero disables it by skipping the wrapper, rather than installing it with an effectively infinite timeout, so the disabled case costs nothing at run time.

On timeout, close the underlying response rather than only raising. Otherwise the socket lingers and the provider keeps streaming into nothing.

Two retry budgets, at two scopes

They are different knobs and collapsing them into one breaks whichever loses.

In-turn model retries are fast, in-process and small: three tries with exponential backoff and jitter. They cover a transient provider failure inside one turn.

The per-Run attempt budget is slow and cross-process: ten. It counts crash and reclaim cycles, incremented once per claim or replace-attempt, and is checked before a replacement Attempt is handed out at all.

Share one budget and either crash recovery starves because transient model retries ate it, or a genuinely stuck Run gets reclaimed forever because nothing counts against a shared cap.

The in-turn count is derived from the log, not held in memory. Walk the durable history backwards and count consecutive trailing assistant messages classified as retryable errors, stopping at the first user message, which marks a new operation boundary. This is the only crash-consistent version: the budget a restart computes is identical to the one the live process had, and isolated transient errors separated by successful turns do not share a budget. The reducer purity rule already requires this shape.

Classifying a failure as transient

Two tiers, and the order matters.

A structured marker stamped by the call site that knows for certain it hit a transient condition, such as the stream idle timeout above. Checked first and authoritative.

A heuristic over the error text for everything else: 5xx, 429, 408, connection reset or refused, socket errors, timeouts. Explicitly second-class and lossy, because it is reading messages Psych did not author. 400, 401, 403 and schema errors are terminal.

Keep the two tiers visibly separate. When the classifier gets it wrong, you need to know whether a marker was missing or a regex was too loose.