0232: Daemons restart when the installed code changes

Status: accepted (2026-11-02) · Scope: product tier (core/code-identity, core/presence/relay-client, core/semantic/service, core/supervisor/service)

Context

Harnery’s long-lived background processes are detached daemons: the presence relay client, the semantic reader, and the local supervisor. Each one keeps running the modules it loaded when it started. A fix that reaches the checkout through a pull, or the installation through an upgrade, does not reach a daemon that is already running. The daemon keeps the old behavior, including the bug the fix was for, until someone finds the process and kills it by hand. Nothing in the daemon notices that the code underneath it moved.

The supervisor and the governor service already record a running_build and report restart_needed, but only as status for a person to read. That build is read with a git subprocess, which is too expensive to run on every tick, and an unknown build counts as “restart needed”, which would loop if acted on.

Decision

A cheap code identity. core/code-identity computes <version> from the package’s package.json, plus @<commit> when the package root is a Git checkout. The commit comes from plain file reads: .git as a directory, or a .git file naming a gitdir (submodules and linked worktrees, with commondir for shared refs), then HEAD, then the loose ref or packed-refs. No subprocess runs. The result is cached against the stat of every file consulted, so a repeat read costs a few stat calls. An identity that cannot be read is null.
null never means changed. A daemon whose starting identity is null never restarts for code reasons, and a null read during the loop is ignored. A daemon that cannot tell must not restart in a loop.
Confirmed before acting. Each daemon checks at most every 30 seconds and acts only when two consecutive reads agree on the same new identity, so it does not hand over to an update that is still being written.
Each daemon records what it loaded. The relay’s pid file, the semantic status record, and the supervisor status record carry code: { id, root }, where root is the package root the identity describes.
Exit and hand over. On a confirmed change the daemon writes one lifecycle record (“code changed (old -> new); exiting for restart”) and leaves through its normal path. The semantic reader stops at the same point an operator stop uses, after the current model call. The relay closes its socket cleanly. Then each starts its replacement from the new code: the relay through its own launcher, the semantic reader and the supervisor through their start functions after releasing their leases. The relaunch happens only in the detached daemon entrypoints, so a library caller that runs the loop in process only sees it return with exit_reason: "code_changed".
Launchers replace a stale daemon. ensureRelayDaemon (run by the session-start and stop hooks), ensureSemanticServiceRunning, and ensureSupervisorRunning (run by web up and by workflow admission) compare a live local daemon’s recorded identity with the current one and replace it when they differ. A record with no identity predates this change: that daemon is never replaced while it runs. It is left alone until it exits on its own (idle exit, a stop command), and the next launcher call then starts a current one normally. A record from another package root is left alone too, so two installations that share a project do not take turns replacing each other’s daemon.
A replacement is a request of its own, never a signal. For the semantic reader and the supervisor, the launcher writes a replace.json naming the daemon generation (its status nonce) it targets. The daemon finishes its current model call or cycle, exits, and relaunches itself with the settings it was started with; the launcher only waits up to ten seconds for the successor. A signal is never used, because on Windows it terminates the process outright, mid-call. The operator’s stop request is never used either, because for the semantic reader it also means “paused” and for the supervisor an operator stop issued during the handover could not be told apart from the launcher’s.
The launcher never touches a stop or pause. It does not write, rewrite, or remove the operator’s stop request. A stale daemon with a stop or pause already pending is left to stop. If the daemon exits without a successor appearing, the launcher checks for a stop or pause again after its last status read and starts nothing when one exists; otherwise it starts the successor with the old daemon’s recorded settings (calls_per_hour, or interval_ms, idle_exit_ms and keep_alive).
Automatic starts never clear a stop. Every automatic start passes --keep-pause (semantic) or --honor-stop (supervisor): a replacement, a relaunch, and an ordinary start of a stopped daemon by web up or workflow admission alike. A daemon started that way never deletes a stop or pause. It checks before taking its lease and again after, and exits (releasing the lease) when one exists; one that lands later stops its loop as usual. A launcher that finds a stopped daemon with a stop request pending reports it as stopped and starts nothing. Only the explicit start command clears a stop or pause, so a daemon an operator stopped stays stopped.
One relay at a time, held by the operating system. A relay daemon must hold the project’s relay instance lock before it opens the relay socket or starts watching the ledger, and it holds the lock for its whole life. On Windows the lock is a named pipe (\\.\pipe\harnery-relay-<hash of the project root>). On every POSIX system it is a TCP listener bound to 127.0.0.1 only, at a port derived from a hash of the lock name and the resolved project root, in 20000 to 32767, below the Linux (32768 and up) and macOS (49152 and up) ephemeral ranges. Binding is atomic and the kernel releases it when the holder exits or crashes: there is no file, nothing to unlink, no stale state to recover, and every runtime on the machine computes the same endpoint. Another port is never tried, because a second port would be a second lock.
Who holds the port. Each holder answers every connection with a banner carrying only the lock’s id (a hash of its name and project root) and a random per-process nonce. A starter whose bind fails reads it: the same lock id means a live peer holds the lock, the normal case. Anything else (another program on the port, or another project’s harnery lock whose port collided) makes the lock unavailable. The daemon then does not connect, logs the reason and port once, records them for presence relay-status (lock: { state: "unavailable", reason, port }), and exits; git-ref presence carries on, and the next launcher call tries again. With an exclusive bind there is nothing for a holder to be displaced by, so there is no periodic re-verification.
Replacement waits for the lock. A successor that finds the lock held by a live daemon on older code marks the pid file handover, which that daemon sees as superseded within five seconds; it closes its socket and exits, releasing the lock. The successor polls for the lock for up to 75 seconds and connects only once it holds it. If time runs out it hands the pid file back, logs once, and exits without connecting; the next launcher call tries again. A daemon that sees its own code change launches its successor and exits, so the same handover applies. A daemon that finds the lock held in any other case is a duplicate launch: it logs one line and exits.
The reservation and pid file are bookkeeping. A launch reservation (relay-launch.json, expiring after 20 seconds and overwritten once expired) saves duplicate spawns when hooks race, and the pid file feeds status and tells a stale daemon it was superseded. Neither decides that a daemon may connect.
The guarantee covers daemons that take the lock. Relay daemons from before the lock hold none. They are never replaced while they run (no code identity), and a successor that holds the lock still waits for a live pid-file owner to exit before connecting. It records that owner as original in its handover pid record, so a later successor that finds it dead and the original still alive waits for the original too, and records it in turn. At most one connected daemon per project is guaranteed only once every daemon on the machine runs this code.
The lock outlives the socket. Every exit (superseded, code change, or idle) closes the relay socket and waits up to five seconds for its close event before the process exits. The lock is never released by hand; process exit releases it, so no successor can connect while the old socket is still open.
Operator stops win. A daemon stopped by its stop command never relaunches itself, whatever the code did.

Alternatives

A git rev-parse HEAD subprocess per check. Correct, but it starts a process every tick in every daemon. On Windows each spawn is costly and is one more chance to open a console window.
Watch the source files. A recursive watch over the package is expensive, platform-dependent, and fires on every edit in a development checkout, where restarting on each save is not wanted.
Compare file modification times of the loaded modules. It catches uncommitted edits too, but it needs the module list, costs a stat per file, and restarts on edits nobody meant to ship. The version and commit are what a pull or an upgrade changes.
Only report it. Status already said “restart needed” for two of these services, and the problem still needed a person to notice and act.
A Unix socket file with self-verification. The holder bound a socket file, answered with a nonce banner, waited a short settle, and probed its own path to confirm it had not been displaced, re-verifying every few seconds. Rejected: a socket file left by a crash can only be recovered by probing it and then unlinking it, and that gap cannot be closed. Starter B probes the stale file and pauses; starter A recovers it, passes its settle check, and connects; B resumes, unlinks A’s live socket, binds, passes its own check, and connects too, so two relay connections overlap until the next verification. A loopback TCP port has no file to unlink.
A file-based exclusive reservation as the correctness guarantee. An atomically created reservation file, reclaimed by rename when expired, with a handover record naming the predecessor, was built and reviewed through several rounds. Every repair opened another window: a reclaimer could displace a fresh reservation, a crashed successor could lose its predecessor, and a stale snapshot could launch a second successor. A file cannot be released by the operating system when its holder dies, so ownership always rested on reads that could be out of date. The reservation stays only as spawn dedupe; an operating-system-held lock carries the guarantee.
Imitate an operator stop for pre-change daemons. A daemon from before this change understands only the operator’s stop request, so the launcher could write one, wait, and withdraw it. Rejected: the launcher cannot tell its own request from an operator’s stop or pause written over it or already pending (on Windows a stop sends no signal, so the daemon can still be running with one in place), and withdrawing it at the end of a bounded wait leaves a daemon that is mid-call never seeing it at all. Each attempt to repair that lost or overrode an operator’s decision. Such daemons are instead left running until they exit on their own.
Exit and rely on the launchers alone. That works for the relay, whose launcher runs at every agent turn. The semantic reader and the supervisor are started when the dashboard starts, so they would stay down until it was restarted. They relaunch themselves instead.

Consequences

A pull or an upgrade reaches the running relay, semantic reader, and supervisor within about a minute, or at the next launcher call.
Uncommitted edits in a development checkout do not restart daemons. Restart them with their stop and start commands while iterating.
The governor service is not covered. It is started only by an explicit governor service start and has no automatic restart path, so exiting on a code change would leave its goals unattended. Its status keeps reporting restart_needed for the operator.
The relaunch runs the old process’s start function, which builds the new daemon’s command line. A future change to an internal daemon entrypoint’s arguments must stay readable by the previous release’s launcher, or the first relaunch after that upgrade fails. The failure is logged, and the next launcher call starts the daemon.
A semantic reader inside a long model call can take longer than the launcher’s ten-second wait. The launcher then reports it as still running, and the reader relaunches itself when the call ends.
A daemon started before this change keeps running its old code until it exits on its own or someone stops it. Restart it once with its stop and start commands to pick up the new code sooner.
Signalling a stale daemon to stop it at once was rejected. It would be faster, but on Windows the signal kills the process mid-call and skips its lease release and shutdown record.