Metering and telemetry
Why this area is shaped the way it is, and what breaks under the alternative.
Covers DESIGN.md §13 in full: usage accounting, pricing, latency, the report and
telemetry. Read it before touching psych_runtime/model/usage.py,
psych_runtime/model/pricing.py, psych_runtime/report/ or psych_runtime/telemetry/.
The usage shape
Token counters are recorded per model call and never aggregated at write time. A
single input_tokens field makes correct cost impossible, so the shape is fixed:
| Field | Meaning |
|---|---|
input | uncached input tokens |
output | output tokens |
cache_read | tokens read from the prompt cache |
cache_write | tokens written to the prompt cache |
cache_write_1h | the subset of cache_write written at one-hour retention |
reasoning | reasoning tokens, when the provider reports them |
total | the provider's own total |
Two of those are subsets, not additions, and treating them as additive double-counts:
cache_write_1h is part of cache_write, not on top of it. It says how many of
those writes used the longer retention. Only some providers report the split.
reasoning is part of output, not on top of it. Its optionality carries meaning:
absent means the provider reported no breakdown, and zero means it reported one and
there were none. Do not collapse absent and zero when deciding whether to show a
reasoning breakdown.
The cost formula
Three details are easy to get wrong and each produces a wrong number that no test catches unless the test pins the exact relationship.
Tiering is by total input-side tokens. A tier threshold like "above 200K input
tokens" is checked against input + cache_read + cache_write, the whole request's
input-side count, not against input alone. Tiers are best-match: pick the highest
threshold that is still exceeded, not the first that matches.
One tier prices the whole request. Input, output, cache reads and cache writes are all priced at the matched tier's rates. A request that crosses the input tier threshold pays the higher output rate too.
Long-retention cache writes are priced at twice the input rate, not at twice the cache-write rate and not from a separate rate field. Cache writes bill like input tokens, and the longer retention costs double that. Short writes are priced at the ordinary cache-write rate. Getting the base wrong shifts every long-cache-write model's cost by a provider-specific margin that looks plausible.
Compute cost once and construct the Usage model with it. Do not mutate a pre-zeroed cost object in place; frozen models make the whole aliasing question go away.
No known price means cost=None, never zero
A silent zero makes metering look correct and be wrong. This is the invariant, and the way to hold it is structural.
Do the price lookup before the arithmetic. On a miss, record cost=None and skip
the formula entirely. Do not try to make the formula itself null-aware; every branch
inside it assumes rates exist, and a nullable rate threaded through means every future
edit has to remember the null case.
rates = price_resolver.get(model_id)
if rates is None: cost = None
else: cost = <the formula above>A consumer's own price table is authoritative, which is why PriceResolver is a port.
The bundled table goes stale between regenerations and says so.
Currency is part of the price, not an assumption. Two calls in one Run priced in different currencies cannot be summed into one total, and the reducer refuses that log rather than stating a number that is not a total.
Latency is durable, not a telemetry side channel
Every Record carries its timing. Queue wait, time to first token, stream duration and tool duration are fields on the log, not attributes on a span.
The distinction matters because telemetry is best-effort and drops silently under a
no-op adapter, while psych_runtime.report() must account for a Run's wall-clock time after a
process restart, reading nothing but the log. A latency breakdown that exists only in a
trace is a latency breakdown that vanishes exactly when someone needs it.
Where the two overlap, use the same words for the same thing, so a consumer correlating a report against traces is not translating vocabulary. That is a naming convention, not a licence to treat one as a substitute for the other.
The telemetry port is a context manager, not a start/end pair
A span exists for the duration of a block and settles on the way out, on return or on exception:
async with telemetry.start_span(...) as span:
...An imperative span.end() puts the burden on every call site to be correct on every
exception path. The block form makes "settle exactly once, even on a throw" automatic.
Nesting is calling start_span again from inside, so a span is itself a context.
Automatic status is one behaviour implemented once in the adapter, uniformly for every span: error on a throw, unless an explicit status was already set.
The span schema is data
Span names, their legal parents and their attributes are declared in a schema, and conformance tests check the running code against it. The declaration carries, per span:
- a description
- a parentage constraint: any parent, root or external only, or a named set of legal parent spans
- start attributes and end attributes, each with a type, an optional closed set of values, and metadata for whether the value is sensitive and whether its cardinality is high
- named sub-events with their own attributes
- a default status plus a human-readable description of when the status is an error
That last field documents intent for a reader. It is not consulted at run time; the automatic-status behaviour lives in the adapter.
Start attributes may be required; end attributes are always optional, which matches the reality that some things are only known once the span closes.
This is where Python needs more work than a statically typed host. A schema like this can be enforced entirely at compile time in a language with structural types and literal inference: a misspelled attribute or a span nested under an illegal parent is a type error, and no runtime check is needed. Python has no equivalent, so the same guarantees have to be bought with a runtime validator plus a conformance group. Budget for it as real work rather than a mechanical translation.
Span vocabulary
The orchestration spans map onto Psych's own model rather than borrowing a foreign
prefix. Attribute keys use Psych's vocabulary: run_id, not someone else's namespaced
equivalent.
| Span | Parent | Carries |
|---|---|---|
| run | root or external | one admitted Run, with a boolean saying whether this is a fresh start or a crash-recovery resume |
| checkpoint | run | settling dangling work on reclaim |
| turn | run | one assistant response plus its tool batch |
| step | turn or checkpoint | one durable attempt: kind, attempt number, and an outcome from a closed set (succeeded, retry, failed, aborted, deferred, overflow) |
| tool | turn or run | one tool execution: name, call id, whether it is safe to replay, whether this execution is settling a dangling call, and whether the result was an error |
| model request | step | one provider call: provider, model, whether streaming, then response id, stop reason, HTTP status, the full usage breakdown, cost, chunk count and time to first chunk |
| sleep | step or run | one retry delay, with an outcome of elapsed or aborted |
| store write | any | one committed append, with the assigned sequence number once known |
The recovery boolean on the run span and the recovery boolean on the tool span are the two attributes most worth keeping. Without them a trace cannot distinguish a Run that worked from a Run that worked on its third try.
Run and Scope identifiers are high-cardinality by declaration, so nobody uses them as a metrics dimension by accident.
What the conformance suite tests
The suite tests the adapter contract, so it applies unchanged to the no-op adapter, the OTel adapter and any in-memory test double. Five groups:
Callback lifecycle. The callback runs exactly once, its side effects are visible before the returned awaitable resolves, and its return value or raised exception passes through unmodified, including values that are not exceptions and objects that throw on every attribute access. An unreadable error must still propagate as the same object.
Status. An explicit status always wins over the automatic one, in every combination. Set ok and then throw and the span stays ok. Set an explicit error and then reject and the explicit message survives.
Recording. Attributes merge across calls rather than replacing, and an undefined value means "ignore this key". A failed attribute call is atomic: if reading the payload throws, the whole call is discarded rather than half applied. Events keep their order. Every method on a settled span is an inert no-op that neither throws nor mutates the recorded span, and starting a child span after the parent settled still runs its callback while recording nothing.
Parentage. Nested calls link correctly. Concurrent children of one parent both attach to it. End order is captured by a monotonic counter reflecting actual settlement order, not creation order.
Passivity. Telemetry must never break the calling code, however hostile its own inputs. An unreadable options object still lets the wrapped callback run and its result pass through, degrading to "no span recorded" instead of propagating its own failure.
A sixth group is specific to Python and covers what a type checker would otherwise have caught: given the declared schema, assert that every span the running code emits matches its declared attributes, required and optional and enumerated, and that its parent is legal. Without it, a renamed attribute or a span moved under the wrong parent stops being caught at all.