On this page
What actually changed (scope of test)Test checklist — the workflow itselfEdge cases & failure modesThe big coverage boundary — say it out loudGaps between description and code (verify, don't assume)Suggested Stories to hand back to the epicSee alsoFor Quality
What to test in this PR, and where the edges are. Meta-note: this is a change to the QA tooling itself, so "quality" here means verifying the plumbing, not clinic features.
What actually changed (scope of test)
One file: .github/workflows/sandbox-qa.yml, +62 / −189. Everything to verify is in that file plus the contract it has with the off-repo sandbox-qa binary. You cannot test the binary from this repo — flag that as a coverage boundary.
Test checklist — the workflow itself
Dispatch & inputs
- [ ] Actions → Sandbox QA offers exactly
prepare, test, retest, fix; default isprepare(.github/workflows/sandbox-qa.yml:37-42). - [ ] No
scopeinput remains (removed). - [ ] Dispatch is branch-scoped (
$GITHUB_REF_NAME), not PR-event-driven (:64).
Agent step
- [ ] Only
GITHUB_TOKENis passed as env (:60-61) —QA_SCOPE/QA_REVIEW_FILE/QA_PLAN_FILEare gone. - [ ]
set -epresent; the phase name is the third CLI arg (:63-64). - [ ] Non-zero agent exit fails the job — but post-steps still run (see below).
Summary comment (the meat of the diff)
- [ ] Runs on
always() && steps.qa.outputs.QA_STATE_DIR(:67) — verify it posts even when the agent step "fails," as long as a state dir exists. - [ ] No open PR → no comment, no error (
:82-86). Test: dispatch on a branch with no PR; expect the QA work to run and the comment step to log "no open PR" and return cleanly. - [ ] Single upsert by marker
<!-- sandbox-qa-report -->— running two phases produces one comment, rewritten, not two (:104-114). - [ ] Jira/Confluence links appear only when
JIRA_EPIC_URL/CONFLUENCE_URLare non-empty (:95-96). Test the link-less path (agent emits neither) — comment should still be valid. - [ ]
preparegets the "edit stories, then dispatch test" CTA; every other phase gets the "results are on the tickets" CTA (:98-100). - [ ]
QA_SUMMARYmissing/empty → comment still renders (summary is"",:78-80).
Artifact
- [ ] Artifact name is
sandbox-qa-<QA_SLUG>-<run_id>and uploads theQA_STATE_DIR(:116-121). - [ ]
retention-days: 14,if-no-files-found: warn(:122-123). - [ ] Branch with a
/in its name still yields a legal artifact name (that's whyQA_SLUGexists).
Concurrency & safety
- [ ] Two dispatches on the same branch don't run concurrently, and the first is not cancelled (
:44-46). - [ ]
fixcommits carry[skip ci]sosandbox.ymldoesn't redeploy (verify no deploy-loop). Header contract:19-20,:22-26.
Edge cases & failure modes
| Edge | Expected | Risk if wrong |
|---|---|---|
Agent dies before emitting QA_STATE_DIR | Both post-steps skip; nothing posted/uploaded | User sees a bare failed job, no clue — check homelab logs |
| Branch has no open PR | QA runs, comment step returns quietly | Looks like a no-op to the user |
Agent emits QA_STATE_DIR but no links/summary | Minimal comment with heading + timestamp only | Acceptable degrade |
Two markers from old PRs (review/plan) linger | Ignored — nothing reads them now | Cosmetic only |
fix on an unconfirmed failure | Should not happen — gate is human; but nothing in the workflow enforces "confirmed" | Trust boundary — relies on the agent + human |
The big coverage boundary — say it out loud
- The Jira/Confluence behavior is untestable from this repo. Story creation, PASS→Done transitions, screenshot attachments, Bug tickets, the Confluence results table — all live in
qa-agent/atlassian-qa.py/sandbox-qa, which are not in/work/repo. To QA those, you need access to the homelab and a real Atlassian instance. Treat the PR body's claims about them as unverified from here. - The "reads human comments as instructions" logic (header
:17-18) is likewise off-repo. You can only verify it end-to-end on a live run, not by reading this diff.
Gaps between description and code (verify, don't assume)
- PR body: "screenshots attached per ticket," "Bug tickets," "cross-linked with PR/sandbox/docs." The diff contains none of this — only
JIRA_EPIC_URL/CONFLUENCE_URLlinks. Verify against a live run. - PR body: "GitHub story-issues are retired." Verifiable and true — the entire issue-creation block is deleted.
- PR body: "Foreground with live agent streaming." Consistent with the single non-detached
run:step (contrastsandbox.yml:66, which detaches the docs agent).
Suggested Stories to hand back to the epic
- As a QA engineer, dispatching
prepareon a branch with an open PR posts one🧪 Quality Reviewcomment with a Jira link. - Dispatching any phase on a branch with no open PR completes green and posts nothing.
- Re-dispatching a phase rewrites the same comment rather than adding a second.
- A
fixrun pushes a[skip ci]commit and does not trigger a sandbox redeploy viasandbox.yml. - An agent step that exits non-zero after emitting
QA_STATE_DIRstill uploads the artifact.
See also
- Workflow Internals — line-by-line reference for the checks above.
- What Changed — the deletions to confirm.