ADR 0137: bounded event-ledger epochs

Date: 2026-08-30 Status: Accepted

Context

The V3 event ledger is an append-only authority: every canonical read re-validates the complete epoch, including canonical serialization, digests, producer boot sequences, and causal links (ADR 0080). ADR 0082 made repeated reads inside one process cheap, but hook producers are one-shot processes. Each hook invocation therefore pays a full cold read of all recorded history, twice per tool call per agent, and nothing bounds the active segment’s growth.

On one busy multi-agent host the active segment reached 176 MB and 95,000 events within three days of its genesis. A single hook process then peaked above 1 GiB of resident memory and spent about seven seconds validating history before recording one event. With several agents running concurrently, the hook fan-out alone exhausted the machine’s memory budget. Measurements attributed nearly all of the cost to per-event canonicalization, hashing, and schema validation of historical frames, not to runtime startup.

Alternatives

Persist the incremental read cache across processes. Rejected. The cached snapshot includes every parsed event, so a durable cache is the ledger again in a second format, with its own integrity problem. It would speed up reads without bounding memory.

Seal segments under the existing catalog without replacing the epoch. Rejected. The canonical reader verifies every sealed segment’s bytes and digest on each read, so sealing changes the file layout but not the cost curve. Cold reads would still scale with all history.

A resident daemon that serves projections to hooks. Deferred. It could amortize the read once per machine, but it adds a availability and trust boundary that hooks currently do not have, and it still needs a growth bound underneath it.

Reduce event volume. Insufficient alone. Smaller payloads slow the growth rate; they do not bound it, and validation cost scales with event count as well as bytes.

Decision

Two mechanisms, shipped together because the second makes the first safe.

Automatic size rotation. The shared route boundary (resolveLiveEventLedgerRouteV3) rotates a valid, active epoch once active.ndjson reaches a threshold: default 32 MiB, configurable via events.rotate_active_bytes in .harnery/config.jsonc or the HARNERY_EVENT_V3_ROTATE_ACTIVE_BYTES environment variable, disabled by a non-positive value. Rotation runs under the bootstrap lease, re-checks size and control state inside the lock, drains the durable ready spool into the epoch that produced it, then archives the entire ledgers/v3 directory (events, spool, producer states, manifests) unmodified into .harnery/ledgers/v3-archives/ and activates a fresh genesis, the same mechanics the runtime-incompatibility refresh already used. Only a currently valid ledger rotates; integrity failures stay closed for the explicit recovery command. The atomic archive rename is also the live-session snapshot. Before rotation returns, every non-terminal archived producer is re-anchored with a fresh epoch-local generation, attestation, boot sequence, and session span. A turn that was open at the boundary is reopened from the owner-only producer state; an idle session stays between turns. Late old-epoch producers cannot overwrite the replacement state because private-state publication carries the same genesis fence as event publication. Rotation failure never blocks recording; the oversized epoch keeps serving.

Epoch fence on writes. An event’s authority is epoch-local: producer boot sequences must start at 1 and every causal link must resolve inside its own epoch, so an event built against one epoch and committed into its successor poisons the successor’s integrity. Producers now stamp the genesis id they were built against. The writer refuses a fenced write once the live genesis differs (epoch_replaced), tags spooled ready rows with their epoch, and the drain quarantines any tagged row whose epoch was replaced underneath it (preserved beside the spool as .epoch-replaced, never appended). Producer state carries the same stamp (epoch_genesis_id), and no consumer adopts a state from another epoch; a state that predates the field is treated the same way. LiveEventLedgerRouteV3 exposes genesis_id so derived writers (image artifacts, finalization observations) fence against the epoch their source event was recorded in, and the authority outbox accepts an explicit fence for transaction publication and reconciliation.

Result

A rotation on the 176 MB production epoch archived it intact and brought the next hook’s cold read back to a fresh epoch of a few kilobytes. Regression tests cover: rotation archives byte-identical history and leaves a complete successor; the spool drains into the epoch that produced it; a fenced write for a replaced epoch is refused with nothing spooled; a tagged ready row landing in the successor’s spool is quarantined, not committed; route resolution rotates at the configured threshold and honors the disable value; and a live session that re-publishes stale producer state after rotation is re-onboarded into a fresh generation while the successor ledger stays complete.

A later rotation showed the remaining pre-hook gap: for 45 seconds, 1,485 command observations could not find a current producer generation. Each was dropped and wrote its own command_emit_unjoinable diagnostic. Re-anchoring now happens inside rotation, before control returns to any producer. A regression records a command immediately after rotation, without another hook, and proves it joins the reopened turn. A second regression pauses an old-epoch hook after its ready row is published, rotates underneath it, and proves its late state publication neither overwrites the re-anchored generation nor breaks intake or lease cleanup.

A rotation that fires while a session is mid-turn exposed a second gap, this one in re-onboarding rather than in the archive. The session’s next hook after rotation is usually a tool signal, not a prompt. Mid-flight onboarding opened a generation for it but no turn, because only a prompt opened turns, so command telemetry refused every join for the rest of the turn and the end-of-turn gate could not close. Onboarding also inherited the tool hook’s deferred drain, so the derived session.started waited in the spool where the coordination view could not see it, and every authority command reported no live generation until some other writer drained. Recovery therefore depended on the next human prompt. Onboarding now commits its own events through the append path and, when the triggering signal is one an adapter can only deliver inside a turn, opens a derived turn that keeps the payload’s native turn id. A regression drives a live session across a rotation on a deferred tool hook and proves the successor ledger holds the derived session and turn, the generation is authority eligible, a command joins the turn, and the next native prompt closes the derived turn before opening its own.

Two storage follow-ups remain. Ready rows spooled with deferred drain in the final moments before rotation are archived undrained; their bytes are preserved in the archived spool, and rotation drains the spool first to keep that window small. ADR 0150 closed the archive follow-up with a guarded byte-and-age policy that keeps a configurable newest-epoch floor. The active epoch remains outside that policy.

Rotation also exposed a writer race on a multi-agent host. It came from re-onboarding, not the archive. Rotation re-anchors every live generation at once, so it mints fresh generations and attestations in one burst while unrelated authority writes continue. That is the condition required by a known append-ordering defect, in which an event carrying an attestation can be committed before the event that mints it, because a drain orders frames by their declared causes and an attestation dependency is not one of them. The reader validates attestations in file order, so such a pair reports unresolved_attestation and the control boundary fails closed for every session until the ordering resolves. Rotation neither introduces that defect nor changes drain ordering: the epoch tag is appended after the row digest in the ready filename, so the spool sort prefix is unchanged, and the replaced-epoch quarantine preserves list order. What rotation changes is how often the triggering condition arises, from rare epoch replacement to every threshold crossing.

The writer now closes that race under the append lease. It reads the validated checkpoint for the active authority and drains a ready row only after every causal parent and required attestation is committed or ordered earlier in the same batch. A dependent row that arrives before its attestation declaration stays in the WAL. Once the declaration appears, one drain appends the declaration first and the dependent second. A torn-tail repair discards the old checkpoint before ordering, so the writer cannot treat a truncated declaration as committed. The regression enters through the same filesystem boundary: a coordination event arrives while the attestation row is flushed but not yet renamed. It stays held, then drains behind the declaration. The final ledger is complete and has no diagnostics.

Two consequences are worth knowing. Diagnosis is unreliable from a single sample, because the torn-tail repair path can truncate and re-commit a frame, so the control boundary can read valid and invalid in consecutive reads while coordination keeps working; sample repeatedly before concluding either way. And while an authority reads invalid, rotation correctly refuses to fire, since it replaces only a valid active epoch, so a persistent integrity failure also suspends the growth bound until the authority is recovered. The repair belongs to the writer-side ordering invariant rather than rotation. Raising the threshold only reduces exposure, and ADR 0138 removes most of the per-read cost that motivated a low threshold in the first place.