For Stakeholders

Scope: internal engineering tooling. No customer-facing change, no revenue impact, no compliance surface. This page is about delivery velocity, quality assurance, and the risk of letting an AI touch a code branch.

Business view

What was bought, in one line

A checkpoint. The AI QA agent used to run start-to-finish unsupervised; now a human reads and corrects its reasoning at two points before it tests or commits anything.

Why that matters more than it sounds

The agent's pipeline has four stages, and each one consumes the previous one's output. Under the old design, an error in stage one propagated through all four with no opportunity to intervene:

The AI misreads what an insurance-claim change was supposed to do → it writes a test plan for the wrong behavior → the tests "pass" → in full-auto, it even commits fixes reinforcing the misunderstanding → and the PR gets a green QA comment that means nothing.

The failure mode was not "the AI is wrong sometimes". It was "when the AI is wrong, the output looks exactly like when it is right." Confident wrong answers are worse than no answer, because they get trusted.

Two edit gates change the economics. A wrong review costs one cheap review run plus a few minutes of editing, instead of a five-hour pipeline producing authoritative-looking garbage.

The measurable claims

What it does not do

Setting expectations, honestly:

• It does not make the AI more accurate. It makes the AI's errors cheaper to

catch. The model is unchanged.

• It does not reduce total run time. Four supervised dispatches with human

reading between them is slower wall-clock than one unattended full-auto run. The trade is latency for trustworthiness.

• It does not remove the human. The value depends entirely on someone

actually reading and editing the comments. An unread review comment is worth nothing, and the tool cannot tell the difference.

That last point is the main adoption risk. If the team drifts back to dispatching full-auto because it needs no attention, the PR's benefit evaporates — and full-auto explicitly ignores edited comments (.github/workflows/sandbox-qa.yml:61).

Risk register

Risk 1 — An AI can commit to feature branches

fix and full-auto push to the PR branch; the workflow grants contents: write (.github/workflows/sandbox-qa.yml:51).

Severity: medium. Mitigations in place: only repo collaborators can dispatch a workflow; the default action never commits; commits are marked [skip ci] and are described as bounded. Residual: nothing in this repository constrains what a fix may change — the bound is in a host-side script. AI commits need normal code review. This risk predates the PR and is unchanged by it.

Risk 2 — Half the behavior is unreviewable

The workflow is a wrapper around /opt/homelab/sandbox/bin/sandbox-qa, a script on the homelab host that is not in this repository (.github/workflows/sandbox-qa.yml:93). Auto-planning, failed-only scoping, fix bounds, [skip ci], self-redeploy, full-auto chaining, and the live log streaming all live there.

Severity: medium, and structural. The two halves can drift with no signal — if the script's output keys change, runs degrade silently rather than breaking loudly. Mitigation to consider: version the script into a repository, or add a contract test that fails the workflow when an expected output is missing.

Risk 3 — Silent degradation

Several paths warn instead of failing: unparsable agent output produces a green run reporting zero stories (:156); a deleted plan comment causes silent regeneration (:80); an agent that dies early uploads no artifact at all, including the log the error message points you to (:242).

Severity: low-to-medium. A green QA run that tested nothing is exactly the false-confidence failure this PR was meant to reduce, reintroduced one layer down. Mitigation: the summary comment's tally is the honest signal; treat it, not the checkmark, as the gate.

Risk 4 — Issue-noise growth

test files a complete new set of issues on every run and only fix deduplicates (:161). The composable design invites iteration, so issue volume under a PR label grows with each cycle.

Severity: low, operational. Mitigation: bulk-close by label; prefer fix for iteration.

Risk 5 — Capacity contention

Single self-hosted runner slot, 300-minute timeout, non-cancelling concurrency (:46-48, :57-58). A wedged QA run stalls sandbox deploys repo-wide.

Severity: low, pre-existing. Mitigation: cancel manually. Worth watching if QA dispatch frequency rises — which is exactly what this PR encourages.

Risk 6 — Breaking interface change

mode → action, with all values renamed and no alias (:26-36). Any script or bot invoking this workflow programmatically breaks.

Severity: low. Mitigation: confirm nothing outside the Actions UI dispatches it; see Migration.

Decision points

Three things worth an explicit call before or shortly after merge:

1. Should full-auto stay? It is the shape this PR was written to replace,

it ignores human edits, and it is one of two actions that commits. Keeping it is defensible for unattended overnight runs; keeping it and letting it be the habitual choice defeats the change.

1. Should the host script be versioned into a repository? Risk 2 is the only

structural one here, and it is fixable.

1. Should silent degradation be made loud? Changing the core.warning at

:156 to a failure, and emitting QA_STATE_DIR early so artifacts always upload, are both small changes that convert the worst failure modes from invisible to obvious.

Status

Unreleased. The PR is open against main; this supersedes PR #162, which was merged before its final rewrite landed. The PR description reports the host-side script is already deployed and smoke-tested with a real review run on another branch — that deployment sits outside this repository and is not evidenced by the diff.