harn qa-run

harn qa-run [options] <target>

Executes a complete page-QA matrix against one target (URL, local file path, or framework route) in a single invocation. Without it, a signoff means a long serial sequence of harn browse commands, one per viewport, theme, state, interaction, and critique pass. qa-run runs that whole sequence itself: the QA planner first, then the deterministic gates for every required context through a bounded child-process pool, then interaction assertions serially, then a capture of every context into a page review pack through the same pool (each browser closes as soon as its tiles are on disk), then the critique: one in-process pool of vision calls over every tile of every context, with no browser open, and finally the QA snapshot. It writes page-qa-result.json with a terminal verdict of passed, failed, or incomplete.

Every child is a real browse invocation of the same host CLI (argv arrays only, never a shell string), so profile, cookie, and capture semantics are byte-identical to running the commands by hand. The vision calls run in the runner’s own process through the host’s injected critique provider, the same provider browse --check-critique uses.

Options

Flag	Description
`--job <file>`	QA job document (JSON, see below). Validated before any browser starts; all validation errors print at once. Absent: a minimal implicit job for `<target>`.
`--mode <mode>`	`signoff` persists a QA snapshot on a passing run; `review` never does. Default `signoff`. An explicit `--mode` overrides the job file’s mode.
`--concurrency <n>`	Concurrent deterministic gates and pack captures (1–8; default 2). Interactions always run serially.
`--pool <n>`	Vision calls in flight during the critique stage, across every context (1–16; default: the provider’s own concurrency). Sets `policy.critique_pool`.
`--tile-budget <n>`	Shared native tile budget across the whole run (8–400; default 96). Sets `policy.critique_tile_budget`.
`--accept-dispositions`	Permit exact matches to complete saved reviewer decisions. Off by default. Sets `policy.accept_dispositions`.
`--allow-metered`	Permit the critique provider’s metered-API fallback. Default: subscription-backed headless harnesses only.
`--retain <minutes>`	Minutes the run’s page review pack lives after the judge finishes before the whole pack is deleted (default 90). Sets `policy.review_pack_retention_minutes`.
`--out-dir <dir>`	Parent output directory. By default, each invocation creates a managed workspace under `.harnery/artifacts/`. An explicit parent is reused as given. Every invocation writes into its own `run-<run_id>/` beneath the parent and updates `latest.json`; prior runs are never overwritten and never pass as current.
`--json`	Print the full result JSON to stdout (the file is written either way).
`--detach`	Launch the run as a detached background process whose authoritative state lives on disk. Prints the run directory and returns immediately; reconnect with `harn qa-status`. See “Detached runs” below.
`--no-queue`	Skip machine-wide admission control. Default: the run waits for a `browser-qa` slot so simultaneous heavy runs take turns instead of thrashing the host.
`--queue-timeout <minutes>`	Maximum admission wait (1 to 240; default 20) before the run finalizes incomplete with an `admission` blocker.
`--queue-capacity <n>`	Concurrent `browser-qa` runs this machine should allow (1 to 8; default `HARNERY_QA_ADMISSION_CAPACITY` or 2). Advisory: the smallest capacity among concurrent waiters effectively governs.

Exit codes

Code	Meaning
0	Verdict `passed`.
1	Usage error or job validation failure (nothing was executed).
2	Verdict `failed`: a completed check found a defect.
4	Verdict `incomplete`: something could not be established (blockers name it).

Execution order

Plan. browse --qa-plan classifies the render against the persisted QA baseline and emits the coverage manifest. A missing, unparseable, or incomplete manifest is a plan blocker: the run stops before any further browser work and the verdict is incomplete.
Deterministic gates. Coverage is the union of the manifest’s contexts and the job’s extra contexts; a job can widen the matrix but never narrow it below the manifest. Each context (viewport × theme × state) gets one child browse invocation carrying all of its applicable job checks, with a unique --out prefix under the run directory. Dark contexts render through browse --color-scheme dark. Exit 0 is a pass; exit 2 is a failed gate (failure details are parsed from the JSON artifact); anything else (a timeout, a spawn error, a missing artifact) is an unknown outcome plus a named blocker. When the run goes on to write its pack, every rectangle-bearing finding in a gate’s JSON artifact (runts, contrast, truncation, clipping, overflow, collisions, target size) is recorded as a gate hit and mapped to the tiles that show it in the pack’s inspection plan and review.md.
Interactions. Each interaction_states entry runs serially: its setup flags, then every assertion as --assert with --assert-fail. A manifest-required interaction state the job does not declare is a blocker: coverage may never be narrower than the manifest.
Capture. A visual manifest requires a preliminary capture plan from every gate render, including failed gates. The runner allocates one shared tile budget, then reserves each context’s assigned tile count. Each capture child repeats its context setup and all applicable gates, plans the current page, and allocates within that reservation. Full source identity must remain unchanged during gates and native capture. One source change permits one complete gate and allocation retry on the same page; a second change or insufficient native coverage makes the run incomplete. Preliminary outcomes remain in preliminary-gates.json; final outcomes reference the fresh capture envelope and its source identity. Failed gates still produce diagnostic tiles and retain failed command outcomes. Partial writes remain outside the final context until the complete context is published atomically. Browser concurrency stays within the existing command pool, and every browser closes before vision starts.

Each capture child writes native tile PNGs, a half-size full-page overview, a contact sheet, DOM, and signature, then closes its browser. Native scrolled probes check full-page capture fidelity; disagreement triggers scrolled tile capture. Missing probes or an incomplete fallback cannot establish a pass. Coverage lists selected rectangles, reviewed and uncovered intervals, and omitted gate hits or scopes. There is no extra tile path outside the allocation. A child that dies or writes no readable context creates a capture blocker; other allocated contexts can still finish.
Critique. One pool of vision calls over every tile of every captured context, read from disk, in the runner’s own process. No browser is open. Pool size is policy.critique_pool (default: the provider’s own concurrency). Unless the job permits metered critique, the runner sets HARNERY_CRITIQUE_HEADLESS_ONLY=1 on its own environment before loading the provider; an exhausted headless list becomes an incomplete blocker naming the policy, never a metered call. When a persisted baseline exists, band-diff reuse skips tiles whose pixels provably did not change and carried no baseline finding. In signoff mode a capped context is a critique blocker and the run is incomplete; in review mode the row keeps its outcome and carries the flag. Tiles the pool never reached before the run deadline leave their context unknown with a blocker, never a partial pass. The pack’s review.md, inspection plan, and evidence/critique.json are refreshed with the machine findings.
Snapshot. In signoff mode the runner saves the signature, DOM, and native tile evidence from the pack. Critique is attached only when every tile was judged or safely reused, coverage was complete, and the review was unscoped. A reused clean second run can therefore support reuse on the third. A manifest that requires no visual pass still gets a dedicated browse --qa-snapshot pass. Signoff can pass only when every required snapshot was saved. Snapshot writes preserve the independent reviewer-decision slot.
Result. page-qa-result.json (schema version 4) in the run’s own directory, in manifest order regardless of completion order: the run identity block, host samples, last_completed_stage, the manifest, merged contexts, every command’s argv/exit/outcome/failures/artifacts/wall-time, critique outcomes per context (findings cite <context-id>/<tile-id>, plus the page coverage the tiles reached), the critique_pool block (concurrency, tiles total/reviewed/reused, wall time, provider, and the backends’ p50/p95 latency_ms over the whole pool), the review_pack location with its expires_at and size_bytes (the pack is deleted at expiry; the result keeps every finding inline), the snapshot record, per-stage wall times (capture is browser time, critique is judge time), blockers, and the verdict.

Queueing: machine-wide admission

A full QA matrix is heavy: a browser per context during gates and capture, then a pool of vision calls. Two matrices started at once on the same machine slow each other down more than they would running back to back. By default, every qa-run therefore queues on the machine-wide browser-qa admission resource before any browser work starts. At most two runs hold slots at a time; the rest wait their turn in FIFO order.

Capacity is 2 by default, overridable per machine with the HARNERY_QA_ADMISSION_CAPACITY environment variable or per invocation with --queue-capacity (1 to 8). Capacity is advisory: the smallest capacity among concurrent waiters effectively governs.
The wait is recorded, not hidden. Time spent queued lands in wall_time_ms.queue in the result. It is never counted in wall_time_ms.total, which stays pure runner time, so queue contention can never make a run look slower than it was.
A full queue is evidence, not an exception. When no slot frees within the timeout (default 20 minutes, --queue-timeout), the run finalizes a normal incomplete result with an admission blocker naming the current holders. The result document, latest.json pointer, and exit code 4 all behave as for any other incomplete run, so the evidence trail survives.
Opting out: --no-queue skips admission entirely for a run that must not wait.

While queued, the live status document (below) reports state: "queued" with the resource and waiting_since; inspect the whole queue with harn qa-status --queue or harn admission status. Admission is a single-machine, advisory protocol built on plain files (see harn admission for the mechanics); a crashed run’s slot is reclaimed automatically by dead-PID pruning.

Detached runs

--detach launches the matrix as a detached background process and returns immediately, printing the run ID, PID, run directory, and log path. The authoritative state lives entirely on disk in the run directory: job.json, run-status.json, the runner’s combined output in runner.log, and eventually page-qa-result.json. A terminal or client disconnect kills nothing and loses nothing, and because the job keeps running, there is never a reason to start a duplicate.

Reconnect with:

harn qa-status <run-dir> --wait

qa-status --wait blocks until the run completes and exits with the runner’s own exit code, so a detached invocation composes into scripts exactly like a foreground one. Detached runs queue for admission like any other (the status document shows queued while waiting); combine with --no-queue to skip that.

Run identity and output layout

Completed managed runs include after_review guidance in JSON output and a matching message in text output. After reviewing disposable or superseded evidence, use harn artifacts discard to shorten its retention. A passing verdict does not shorten retention automatically. Keep files needed for pending review, handoff, debugging, or final evidence.

Every invocation mints a fresh run and isolates its output. Without --out-dir, the parent is a manifest-backed workspace subject to the artifact store’s retention and size policies. An explicit --out-dir names a caller-owned parent directory instead. The run itself writes into run-<run_id>/ beneath the parent:

.harnery/artifacts/<date>_qa-run_<artifact-id>/
  .harnery-artifact.json     # retention and cleanup metadata
  latest.json                 # pointer to the newest completed run
  run-<run_id>/
    page-qa-result.json       # the result document (schema version 4)
    job.json                  # the effective validated job
    run-status.json           # live state + stage + heartbeat
    runner.log                # combined runner output (--detach runs)
    <context artifacts...>    # per-context png/html/json trios (gates, captures)
    pack/                     # the page review pack (see harn review-pack)
      review.md               # agent entry point: protocol, contexts, findings, plan
      manifest.json  findings.json  findings.schema.json
      evidence/               # inspection-plan, coverage, critique, index, files
      contexts/<id>/          # full-page.png, contacts.png, tiles/T001.png…, dom.html.gz, signature.json

latest.json is written atomically (temp file plus rename) after each run and carries {schema_version, run_id, dir, result, completed_at, verdict}. Consumers resolve the current result through this pointer instead of guessing at loose files; each run’s own directory remains the authoritative record, so a reused parent can never present an older run’s result as the current one.

Two companion documents sit beside the result in every run directory:

job.json is the effective validated job the run executed (after the CLI merged its authoritative target and mode). A reconnecting client can re-derive the job digest from it, which makes qa-verify --job <run-dir>/job.json possible without keeping the original job file around.
run-status.json is the live status document: {schema_version, run_id, pid, state, stage, started_at, updated_at}, plus queue while queued and verdict once completed. It is written atomically at start, at every stage boundary, and on a 15-second heartbeat timer, so a client that lost its terminal can mechanically distinguish a running job from a dead one (see harn qa-status). The status document is advisory; the result document stays authoritative once the run completes.

The result’s run block is what makes it verifiable evidence rather than a loose file:

Field	Meaning
`run_id`	Minted per invocation (a UUID). Also names the run directory.
`started_at`, `completed_at`	ISO-8601 UTC bounds of the invocation.
`tested_revision`	Git SHA or content identifier of what was tested, when resolvable. Source order: the job’s pinned `tested_revision`, then a one-shot git probe of the working directory.
`revision_source`	Where the revision came from: `job`, `git`, or `unknown` (no revision resolvable).
`worktree_dirty`	The git probe saw uncommitted changes; a revision alone does not prove content. Absent when no probe ran.
`job_digest`	SHA-256 over the effective validated job. Execution limits such as concurrency and timeout are excluded. Enabling disposition acceptance is included because it changes how findings affect the verdict.
`out_dir`	Absolute run directory the result was written into. A result found elsewhere has been moved or copied.

Two more fields support triage. host carries a load sample (1-minute load average, free and total memory, CPU count) at start and finish, so an incomplete run records the pressure that produced it. last_completed_stage names the last stage (plan, gates, interactions, capture, critique, snapshot) that finished without contributing a blocker, or null when none did.

One note on reading wall_time_ms: it measures one runner invocation, browser-process time included, not the duration of whatever larger task the invocation was part of. An agent loop that plans, edits, and re-runs QA several times accumulates its own wall clock; only the slice inside each qa-run shows up here. Admission queue wait is reported separately as wall_time_ms.queue (absent when the run did not queue) and is deliberately excluded from wall_time_ms.total, which stays pure runner time.

Use harn qa-verify to check that a result is fresh evidence for a specific invocation (run ID, job digest, revision, age, and recorded output directory).

The job document

A job (--job) widens or parameterizes the run. It is validated before anything executes; structural errors and secret-shaped fields are all reported at once.

{
  "schema_version": 1,
  "target": "http://localhost:3000/pricing",
  "mode": "signoff",
  "contexts": [
    { "id": "hd-dark-default", "viewport": "hd", "theme": "dark", "state": "default" }
  ],
  "checks": [
    { "id": "overflow", "args": ["--check-overflow", "--check-overflow-fail"] },
    { "id": "hero-contrast", "args": ["--check-contrast", ".hero", "--check-contrast-fail"], "contexts": ["desktop-light-default"] }
  ],
  "interaction_states": [
    { "name": "menu-open", "setup": ["--click", "#menu"], "assertions": ["exists .menu-panel"] }
  ],
  "qa_hints": { "scopes": ["#main"], "states": ["menu-open"] },
  "policy": { "command_concurrency": 2, "command_timeout_ms": 120000, "run_deadline_ms": 900000 }
}

The command’s <target> argument is authoritative; a job file may omit target.
contexts append to the planner’s matrix (union, never narrowing). Derived context IDs are <viewport>-<theme>-<state>.
checks[].args are argument arrays appended to the context’s capture command, never shell strings. A check without contexts applies to every context. The runner always adds the planner’s deterministic checks. Job checks can add coverage but cannot omit the manifest floor.
Each interaction needs at least one outcome assertion: a click without a proven outcome is not a gate.
policy.critique_tile_budget (default 96, integer 8 to 400), also --tile-budget, limits the total selected tiles across the run. A budget too small to include every context’s page edges refuses capture. Signoff stays incomplete when required coverage remains outside the budget.
policy.critique_max_tiles (integer 1 to 400) optionally limits each context within the shared budget. When omitted, a context can use any remaining shared slots. The gate and capture use the same geometry settings. These execution limits stay out of the job digest; coverage records the actual selection.
policy.accept_dispositions, also --accept-dispositions, defaults to false. When enabled, a saved artifact or not-a-defect decision can dismiss only the identical current finding on an identical native tile with matching context, capture recipe, rubric, and critique version. New or reworded findings remain open. Machine findings stay in the result, and dismissed lists the decisions applied. Enabling this policy changes the job digest. Gate failures, provider errors, and incomplete capture or judging cannot be dismissed.
policy.critique_pool (1 to 16; default the provider’s own concurrency, 4 for headless harnesses) is how many vision calls run at once during the critique stage, across every context. Every browser has closed by then, so a wider pool costs model-call parallelism, not browser memory. Also settable with --pool.
policy.run_deadline_ms (default 900000, minimum 10000) is the overall runner deadline. A run that exceeds it skips its remaining commands, records a deadline blocker, finalizes as incomplete, and releases its admission slot — the deadline clock starts after admission, so queue wait never counts against it. The per-command cap still bounds each child: SIGTERM goes to the child’s whole process group at command_timeout_ms, SIGKILL follows five seconds later, and a command that ran past its cap is reported as an error even if it then exited 0.
Jobs must not carry credentials. Secret-bearing field names (token, password, cookie, …) and credential-shaped values (Bearer …) are refused; authentication is referenced through existing browser profiles and cookie stores.

Verdict rules (fail-closed)

passed requires an intact planner manifest, every gate and interaction passed, every context captured, every required critique tile with a known outcome, and, in signoff mode, the snapshot persisted. A defect found by a completed check is failed. Anything the runner could not establish (a blocker, an unknown outcome, a dead child, a missing artifact) is incomplete; an incomplete run is never a pass.

Examples

# Review a local route with the implicit minimal job
harn qa-run http://localhost:3000/pricing --mode review

# Full signoff from a job file, results as JSON
harn qa-run http://localhost:3000/pricing --job pricing-qa.json --json

# Wider capture pool and a wider vision pool, metered critique fallback permitted
harn qa-run https://example.com/landing --concurrency 4 --pool 8 --allow-metered

# Detach: launch in the background, reconnect later for the verdict
harn qa-run http://localhost:3000/pricing --detach
harn qa-status <printed-run-dir> --wait

# Skip the machine-wide queue and cap this run's admission wait for peers
harn qa-run http://localhost:3000/pricing --no-queue
harn qa-run http://localhost:3000/pricing --queue-timeout 5 --queue-capacity 1