For Quality
You are both the primary user of this feature and the team who has to verify the feature itself. This page covers both.
Business view
What changes for you day to day
You get four buttons instead of one, and two places to intervene before the robot does anything expensive.
The workflow you should adopt as default:
review— cheap, no browser, no commits. Read it. This is where you find
out whether the AI understood the change. If it did not, everything after would have been wasted.
- Edit the review comment. Delete wrong findings. Correct the premise.
plan— it builds stories on your text.- Edit the plan comment. This is your leverage point: the plan is the test
suite. Add the edge cases you know from the field, delete the flaky ones, fix the roles.
test— executes exactly what the comment says.- Triage the open issues. Dispatch
fixfor the ones you want the AI to
attempt; handle the rest yourself.
full-auto is for when you don't intend to supervise. Note that it ignores plan and review comments you have already edited — it regenerates both. If you have curated a plan, use test.
Your new failure modes
| Symptom | Cause | What to do |
|---|---|---|
| Duplicate issues everywhere | test run twice; no dedup | Bulk-close the pr-N label before re-testing |
| "0 stories" on a green run | stories.json was unparsable — a warning, not a failure | Download the artifact, inspect stories.json |
| Plan you edited was ignored | You dispatched full-auto, or you deleted the marker | Use test; keep the <!-- sandbox-qa-plan --> line |
| No artifact to download | Agent died before emitting its state dir | Read the job log; the artifact guard never fired |
| Story count looks healthy but coverage is thin | SKIPPED stories close their issues | Read the ⏭️ count in the summary, not the open-issue count |
The number to trust
The summary comment's tally:
12 stories: 9 ✅ · 2 ❌ · 0 🔧 · 1 ⏭️
Not the green checkmark, and not the open-issue count. The tally is the only place skips are visible.
Technical view: testing this PR
The change is one workflow file, so "testing" means dispatching it and watching what lands on GitHub. There is no unit-testable surface.
Priority 1 — the composability contract
The core claim. Each of these is a distinct dispatch on a real PR.
| # | Steps | Expected |
|---|---|---|
| Q1 | Dispatch review on a branch with an open PR | One comment appears with header ### 🔍 QA code review (AI — edit freely; …) and a hidden <!-- sandbox-qa-review --> marker (:127, :132) |
| Q2 | Edit that comment, insert a distinctive sentinel string, dispatch plan | The resulting plan reflects the sentinel — proving :76-82 fed the edited body forward |
| Q3 | Edit the plan comment to delete all but one story, dispatch test | Exactly one story issue is filed |
| Q4 | Dispatch review again | The same comment is updated, not a second one (upsertComment, :114-122) |
| Q5 | Dispatch test, then fix | Issues are updated in place with a **Re-test:** comment (:179-182), not duplicated |
Q2 and Q3 are the whole feature. If they pass, the PR does what it says.
Priority 2 — the artifact-name fix
This is the regression this PR repairs, and the branch it ships on is itself the test case.
| # | Steps | Expected |
|---|---|---|
| Q6 | Run any reporting action on a branch whose name contains / (e.g. ci/qa-composable) | The Upload run artifacts step succeeds; artifact is named sandbox-qa-<slug>-<run_id> with no / (:247) |
| Q7 | Download that artifact | It contains agent.log and the stage outputs — the files the error messages at :128 and :227 point to |
Q6 against a slashless branch proves nothing. Test on a slashed branch.
Priority 3 — degradation paths
Where the code chooses to warn rather than fail. These are the ones that will silently mislead someone in three months.
| # | Steps | Expected per code | Watch for |
|---|---|---|---|
| Q8 | Delete the plan comment, dispatch test | if (c) at :80 writes no file; run proceeds and auto-plans (:10) | Is there any signal in the summary that the plan was regenerated? |
| Q9 | Dispatch any action on a branch with no open PR | core.info("no open PR — using stored state only") at :72; run continues | Comments are skipped (:115); issues still get a branch-<name> label (:143) |
| Q10 | Force a malformed stories.json | core.warning at :156, run stays green, summary says 0 stories | Confirm this is acceptable, or argue for setFailed |
| Q11 | Make the agent exit before emitting QA_STATE_DIR | Upload step is skipped (:242) — no artifact | The failure message tells you to read a log that was never uploaded |
| Q12 | Produce a review over 60 000 characters | Truncated with a "…(truncated…)" footer (:130) | The truncated text is what plan receives, since :76-82 reads the comment |
Priority 4 — inputs and labels
| # | Steps | Expected |
|---|---|---|
| Q13 | gh workflow run sandbox-qa.yml -f mode=test-only | Fails — input renamed to action (:26). Confirm no scripts or bots use the old name |
| Q14 | Dispatch review and observe the form | The scope dropdown is shown even though it is test-only (:37-44) — cosmetic, confirm it is harmless |
| Q15 | Have the agent emit a story with priority: "critical" | Issue gets a critical label (:199). If that label does not exist in the repo, issues.create throws and the publish step fails partway through — the creation loop at :144-150 only creates ai-automated and the PR label |
| Q16 | Dispatch full-auto after editing the plan comment | Edits are ignored — the fetch step's if at :61 excludes full-auto. Confirm this is intended and documented |
Q15 is the most likely hard failure in the file. Verify the label exists before the first critical story appears.
Priority 5 — safety
| # | Steps | Expected |
|---|---|---|
| Q17 | Run test (all scopes) | Zero commits on the branch. :9-10 claims it, and the publish step contains no git operations |
| Q18 | Run fix | Commits appear, carry [skip ci], and do not retrigger sandbox.yml |
| Q19 | Start a QA run, then push a commit to the branch | sandbox.yml's deploy queues behind the QA job — single runner slot. Confirm the wait is tolerable |
| Q20 | Cancel a running QA job mid-flight | State directory is left consistent enough for the next dispatch to proceed |
What you cannot test from this repository
Roughly half the advertised behavior lives in /opt/homelab/sandbox/bin/sandbox-qa, which is not in this codebase: the auto-plan fallback, failed-only honouring QA_SCOPE, the fix bound, [skip ci] tagging, self-redeploy, full-auto chaining, and the live log streaming. The full list is in Operating notes.
Treat those as integration tests against the host, not as code review of this PR. Q1–Q20 above are all black-box tests through GitHub, which is the only honest way to exercise them.