0078: Derived recovery terminals for orphaned spans

Status: accepted (2026-08-17) · Scope: product tier (core/events/v2, core/agents, adapters)

Context

Adapter hooks do not deliver every completion. A hook process can lose the append lease, a post can arrive without its pre under concurrent delivery, a CLI process can exit before its completion emits, and a Stop signal can go missing entirely. The intake spool and diagnostics work (multi-signal finalization, ADR 0077, and the spool-first recorder) stopped the silent destruction of delivered signals, but the ledger still cannot say anything about a span whose completion never arrived. One production fleet measured roughly six hundred evidence-loss events per day before the spool landed, and an explicit session-end request sat wedged for hours behind two spans that nothing could ever close.

Every open span today is either closed by a native completion or left open forever. There is no honest third state, so open spans accumulate, explicit ends block, and the read side approaches its span cap with no relief valve.

Decision

The recovery payload block

tool.requested, tool.completed, and command.completed gain one optional payload field:

recovery: {
  reason:
    | "request_not_observed"
    | "completion_not_observed_before_turn_end"
    | "completion_not_observed_before_next_turn"
    | "span_cap_pressure"
    | "explicit_end_salvage"
    | "command_completion_not_observed";
  requested_event_id?: EventId;
  elapsed_upper_bound_ms?: Observation<number>;
}

The block is the marker of a recovery event: an event the recovery machinery minted because the native signal never arrived. requested_event_id links a derived terminal back to the span’s original request event when the producer state still holds it.

Recovery records what Harnery observed. It never claims the adapter supplied a missing result: a recovered terminal always carries outcome: "unknown", an unknown duration, no exit code, and no content. Unknown stays unknown.

elapsed_upper_bound_ms is the request-to-sweep interval, not tool execution time. The recorder emits an exact derived bound only when the opening and closing samples share one boot-relative monotonic clock. Cross-boot, missing, invalid, and regressing samples remain unknown with a specific clock-proof reason. Consumers may use the field to bound unknown tool time, but never as an observed duration.

Validator rules

The validators enforce the contract bidirectionally where history allows it:

1. recovery present requires provenance.attestation: "derived".
2. recovery present on tool.completed or command.completed requires outcome: "unknown".
3. A tool.requested or tool.completed with attestation: "derived" must carry recovery, and a native one must not. Command events are exempt from the forward direction because CLI-teed command telemetry has always been attested derived; for commands, the recovery block alone separates recovered terminals from ordinary derived telemetry.
4. Reason codes bind to event types: request_not_observed appears only on tool.requested (and such an event never carries requested_event_id, because it is the request); command_completion_not_observed appears only on command.completed; the remaining four appear only on tool.completed.
5. Recovered completion durations remain unknown and match the recovery reason. An observed elapsed_upper_bound_ms requires exact derived attestation and is forbidden on tool.requested and command.completed.

Reason codes and pinned confidence

Each reason carries a pinned provenance.confidence, chosen by the strength of the evidence behind the derived event:

Reason	Emitted on	Confidence
request_not_observed	derived tool.requested minted for a post that matched no open span	high when the post payload carried the tool input, else medium
completion_not_observed_before_turn_end	turn-boundary sweep of spans stamped with the ending turn’s native id	medium
completion_not_observed_before_next_turn	sweep at the next turn.started after a lost Stop signal	low
span_cap_pressure	write-side watermark relief for spans from already-ended turns	low
explicit_end_salvage	explicit-end salvage over exactly the approved open-span set	medium
command_completion_not_observed	session-end closer for non-terminal command spans	medium

A post recorded as a native completion on a recovery-minted span keeps native attestation and its real outcome; only the machinery-minted side of the pair carries recovery.

Recovery policy constants

The recorder and finalizer policies that emit these events pin their constants here so they are decided once: closed-span memory retains the current plus two most recent turns; the write-side span watermark is 128 and the read-side hard cap of 256 becomes a loud health condition instead of a failure; a pending explicit end expires by cancellation after 24 hours unless it is salvage-eligible, and salvage evaluation always precedes expiry. New native work cancels a pending explicit end; derived recovery events never do.

Turn attribution for recovery comes from the native turn_id in the signal payload on adapters that supply one. Producer-state turn stamps are a fallback for adapters without native turn ids, and events attributed that way are downgraded one confidence step. Spans with no trustworthy turn attribution are left open and fail closed; only explicit-end salvage may reach them.

Shipping as a ledger epoch advance

Payloads are strict objects, so adding the field regenerates the event schema digest, and the digest gates the genesis manifest, the reader’s authority checks, and spool-row validation. The change therefore ships as a ledger epoch advance rather than an in-place edit:

1. Quiesce: drain the ready spool and every session’s intake spool on the old build, and refuse the advance while undrained rows exist.
2. Archive: the live epoch root moves, whole and read-only, under .harnery/ledgers/v2-candidates/<genesis-id> with a tree manifest and a path fence, exactly as the existing epoch-archive flow does.
3. Reinstall: a fresh candidate genesis carrying the new digest is installed and activated through the existing candidate and activation gates. The new genesis anchors its terminal-segment digest to the archived epoch’s active ledger file, preserving the provenance chain.
4. Carry: undrained intake rows are raw adapter signals with no schema digest, so any that appear during the window are drained into the new epoch, where an unmatched post becomes a recovery pair under the rules above.

Recovery capability gating is keyed off the adapter id in code, outside the digested capability-profile structure, so enabling or tuning recovery per adapter never changes a capability digest and never trips producer-state authority checks. At launch, turn-boundary recovery is enabled for adapters with native turn ids and reliable boundary signals; others stay observation-only until evidence justifies enabling them.

Disposition of pre-advance epochs

Archived epochs validate under their own contract generation and are never repaired in place. Generations that ended the old epoch with open spans or pending finalization requests are administratively closed by the advance itself: their requests are superseded, their sessions are classified audit-only by the collector (tool_pairing_incomplete, command_pairing_incomplete), and every original byte is preserved read-only.

Evidence eligibility

Recovered sessions are audit-visible and excluded from healthy corpus quotas. A clean native session remains eligible; a session containing any recovery event or command-pairing gap does not, unless a later, separately approved policy admits a specific recovery class. Recovery exists for coordination honesty, not to launder incomplete telemetry into training evidence.

Latency receipt bounds

The latency projection unions recovery bounds separately from observed span intervals. Unknown tool metrics may expose upper_bound_ms, while known_ms remains the union of observed terminals. The turn receipt reports the bound’s share of turn wall and marks the tool channel ineligible for ranking until its duration is observed. A bound crossing the turn wall is clipped for the percentage calculation and emits recovery_bound_exceeds_turn_wall.

Consequences

Every span now reaches one of exactly three states: natively closed, recovered with an explicit machine-attested reason, or deliberately left open awaiting salvage. Explicit ends stop wedging behind unclosable spans. The span cap stops being a slow-motion brick. Corpus consumers can distinguish native completion from recovery at a glance, per event, with a validator behind the distinction.

An earlier implementation treated the same-clock request-to-sweep interval as an exact recovered duration. Batch 07 showed boundary bursts containing spans orphaned for up to 30 minutes, so the clock relation proved only when Harnery swept the span. This revision keeps duration unknown and labels that interval as an upper bound.

The cost is an epoch boundary: pre-advance events live in an archived ledger the new build cannot append to, and every machine with an active ledger must run the advance once after updating. The advance is a single idempotent command; machines still routed to the V1 ledger are unaffected.

Rejected alternatives

A tool.abandoned event type. A second terminal vocabulary forces every consumer to join two shapes forever. One terminal event with a recovery marker keeps span algebra single-shaped.

Letting lifecycle events close spans implicitly. A turn.completed that silently closes spans destroys the distinction between observed and inferred, and makes replay order-sensitive.

An in-place schema advance via ledger.schema_advanced. The event exists in the contract but no reader implements segment-effective digest switching; building that machinery for one additive field is more risk than an epoch boundary, and the epoch archive flow already exists and is proven.

Inventing outcomes for recovered spans. Marking swept spans failed or cancelled would poison outcome statistics with guesses. Unknown is the only honest label (see also ADR 0077 on evidence eligibility).

Keying the rule on attestation alone. Command telemetry is legitimately derived in normal operation, so attestation cannot distinguish recovery there; the explicit block can, everywhere, forever.