Issues & reporting

What lands in the repository after a test, fix, or full-auto run.

Business view

One issue per user story

When the agent tests, every story it walks becomes a GitHub issue. The issue title carries the story ID and a status emoji; the body carries the story text and what actually happened.

[QA-7] ❌ Receptionist can move an appointment to a fully booked slot

Issues that passed are created and then immediately closed. Issues that failed stay open. So the open-issue list under the PR's label is exactly "what still needs a human", and the closed ones remain browsable as a record of what was checked.

The four possible results:

That last row deserves a flag. A skipped story closes its issue, which means a story the agent could not attempt looks, at a glance, like a story that succeeded. If you are using the open-issue count as your quality signal, skips are invisible in it. Read the summary comment's tally, which breaks skips out separately.

Labels

Every issue gets two labels, created on demand:

• ai-automated (purple) — filed by the QA agent, not a human
• pr-241 (green) — this PR's issues, so you can filter to one change

The second label is how the summary comment builds its "see all results" link.

The summary comment

One comment per PR, overwritten each run:

### 🧪 QA review 2026-07-20 16:40 UTC · run · mode test · 12 stories: 9 ✅ · 2 ❌ · 0 🔧 · 1 ⏭️ Full story-by-story results: ai-automated + pr-241 issues (open = needs attention) …the agent's prose summary… <details><summary>Story issues</summary> ❌ #310 QA-7 · ✅ #311 QA-8 · … </details>

What fix does differently

fix does not file new issues. It finds the ones from the previous run and updates them in place: a comment with the re-test outcome, a retitled issue with the new emoji, and closure if it now passes. So an issue's comment thread becomes the history of that story across attempts.

Technical view

Label creation

const prLabel = pr ? `pr-${pr.number}` : `branch-${branch}`.slice(0, 50);
for (const l of [
  { name: "ai-automated", color: "8B5CF6", description: "Created by the Sandbox QA agent" },
  { name: prLabel, color: "0E8A16", description: `Sandbox QA stories for ${pr ? "PR #" + pr.number : branch}` },
]) {
  try { await github.rest.issues.createLabel({ owner: context.repo.owner, repo: context.repo.repo, ...l }); }
  catch (e) { if (e.status !== 422) throw e; }
}

— .github/workflows/sandbox-qa.yml:143-150

Create-and-swallow-422 is idempotent label creation; 422 is GitHub's "already exists". Any other status rethrows.

The no-PR fallback slices to 50 characters after prefixing, so branch-feature/very-long-name… is truncated — two long branches sharing a 50-character prefix would collide on one label. Branch names with / are legal in GitHub label names, so no sanitization is needed here (unlike artifact names — see Artifacts & outputs).

Reading the stories

const RESULT_EMOJI = { PASS: "✅", FAIL: "❌", FIXED: "🔧", SKIPPED: "⏭️" };
const storiesFile = "${{ steps.qa.outputs.QA_STORIES }}";
let stories = [];
if (storiesFile && fs.existsSync(storiesFile)) {
  try { stories = JSON.parse(fs.readFileSync(storiesFile, "utf8")); } catch (e) { core.warning("stories.json unparsable: " + e); }
}

— .github/workflows/sandbox-qa.yml:152-157

An unparsable stories.json produces a warning, not a failure. The run stays green and posts a summary saying "0 stories". Combined with the malformed-JSON path being the most likely way an agent run goes wrong, this is the single loudest silent-failure mode in the file. Check the tally, not the checkmark.

The story shape the code depends on: id (required — entries without it are skipped at :174), title, story, result, resultDetail, role, priority.

The fix path: matching existing issues

let existingByPrefix = new Map();
if (action === "fix") {
  const all = await github.paginate(github.rest.issues.listForRepo, {
    owner: context.repo.owner, repo: context.repo.repo,
    labels: prLabel, state: "all", per_page: 100,
  });
  for (const i of all) {
    const m = i.title.match(/^\[(QA-\d+)\]/);
    if (m && !i.pull_request) existingByPrefix.set(m[1], i);
  }
}

— .github/workflows/sandbox-qa.yml:158-170

Notes:

• state: "all" — closed issues are matched too, so a story that passed,

regressed, and is now being fixed reopens the original rather than forking a new one.

• !i.pull_request filters out PRs, which the issues API also returns.
• github.paginate — no 100-issue ceiling.
• The comment above this block is stale. Lines 158-159 still say

fix-issues updates the EXISTING story issues and every other mode creates them, using the pre-PR mode name. The code below it correctly tests action === "fix". Cosmetic, but it is the kind of drift that misleads the next reader.

The consequence nobody flags: test duplicates

existingByPrefix is populated only for fix. test and full-auto fall through to the create path unconditionally (.github/workflows/sandbox-qa.yml:191-200).

So running test twice files two complete sets of story issues. A third run files a third. There is no dedup, no "already exists" check, and no cleanup. scope: failed-only reduces the number of stories run but does not change this — the failures it re-runs still get brand-new issues.

This is a real operational cost of the composable design: the old test-only mode was typically run once, whereas the new workflow invites iteration. Anyone who runs test → edit plan → test on a 20-story plan ends up with 40 issues under the PR label. Practical mitigation: bulk-close by label between runs, or prefer fix when you want in-place updates.

Updating vs creating

const prior = action === "fix" ? existingByPrefix.get(s.id) : null;
if (prior) {
  await github.rest.issues.createComment({ ..., issue_number: prior.number,
    body: [`**Re-test: ${emoji} ${s.result}** · ${now} · [run](${runUrl})`, "", s.resultDetail || ""].join("\n") });
  await github.rest.issues.update({ ..., issue_number: prior.number,
    title: `[${s.id}] ${emoji} ${s.title}`, state: done ? "closed" : "open" });
  storyLinks.push(`${emoji} #${prior.number} ${s.id}`);
  continue;
}

— .github/workflows/sandbox-qa.yml:177-190

Note state: done ? "closed" : "open" — fix can also reopen an issue that had been closed if the re-test regresses.

The create path:

const { data: issue } = await github.rest.issues.create({
  title: `[${s.id}] ${emoji} ${s.title}`,
  body: [
    `**Result: ${emoji} ${s.result || "UNKNOWN"}** · role \`${s.role || "?"}\` · priority ${s.priority || "normal"}`,
    pr ? `PR: #${pr.number} · tested ${now} · [run](${runUrl})` : `branch \`${branch}\` · tested ${now}`,
    "", "## User story", s.story || "", "", "## Test result", s.resultDetail || "",
  ].join("\n"),
  labels: ["ai-automated", prLabel, ...(s.priority === "critical" ? ["critical"] : [])],
});
if (done) { await github.rest.issues.update({ ..., issue_number: issue.number, state: "closed" }); }

— .github/workflows/sandbox-qa.yml:191-205

priority: "critical" adds a third label, critical. Unlike ai-automated and prLabel, that label is not created by the loop at :144-150 — it is assumed to already exist in the repository. If it does not, the issues.create call fails and, because there is no try/catch here, the whole publish step fails after having already created some issues. Worth confirming the label exists before the first critical-priority story shows up.

The done predicate

const done = s.result === "PASS" || s.result === "FIXED" || s.result === "SKIPPED";

— .github/workflows/sandbox-qa.yml:176

SKIPPED counting as done is the behavior called out in the Business view. It is defensible — an unrun story is not a defect — but it means the open-issue count understates untested surface. The summary tally is the honest number.

Non-story findings

const issuesFile = "${{ steps.qa.outputs.QA_ISSUES }}";
if (issuesFile && fs.existsSync(issuesFile)) {
  try {
    for (const i of JSON.parse(fs.readFileSync(issuesFile, "utf8"))) {
      if (!i || !i.title) continue;
      await github.rest.issues.create({ title: `[QA] ${i.title}`, body: `${i.body || ""}\n\n---\n_Found by Sandbox QA on \`${branch}\` · ${now} · [run](${runUrl}) · severity ${i.severity || "normal"}_`, labels: [...] });
    }
  } catch (e) { core.warning("issues.json unparsable: " + e); }
}

— .github/workflows/sandbox-qa.yml:209-223

Technical findings not tied to a story — [QA]-prefixed, always created open, never matched against prior runs even under fix. These duplicate on every run of every reporting action.

The summary

const counts = { PASS: 0, FAIL: 0, FIXED: 0, SKIPPED: 0 };
for (const s of Array.isArray(stories) ? stories : []) if (counts[s.result] !== undefined) counts[s.result]++;
await upsertComment("<!-- sandbox-qa-report -->", [ ... `mode **${mode}**` ... ]);

— .github/workflows/sandbox-qa.yml:225-239

Two details:

• stories.length is used unguarded in the tally line at :234, while the

loops that consume stories are guarded with Array.isArray (:173, :229). A stories.json containing valid JSON that isn't an array survives the try/catch at :156 and reaches :234: {} renders as undefined stories, and null throws a TypeError that fails the publish step outright — after the issues have already been filed. Low-probability, but the failure lands in the worst place.

• mode at :234 comes from const mode = action; (:101) — a compatibility

alias left over from the rename whose only surviving use is this one string. It prints the new action name, so the summary says mode **test**, which is correct if slightly inconsistent in vocabulary.