Operating notes & limits

The things that will bite, ordered roughly by how likely they are to bite.

Business view

This workflow can write to your branch

Two of the five actions — fix and full-auto — push commits to the pull request branch. That is by design and it is stated up front, but it is the most consequential fact about the tool: an AI has commit access to feature branches in this repository.

Guardrails that exist:

• Only people who can already push to the repo can dispatch a workflow, so this

does not widen who can write code — it widens what writes code.

• The default action in the dropdown is test, which never commits.
• Fix commits carry [skip ci] so they don't trigger a cascade of builds.
• Fix commits are described as "bounded" — the agent stops after some limit.

Guardrails that do not exist in this repository: nothing here constrains what fix may change, and the bound itself lives in the host-side script. Review AI fix commits like any other contributor's.

One QA run blocks the branch's queue for up to five hours

The job may run for 300 minutes and will not be cancelled by a newer dispatch of itself. The homelab runner has a single slot. A wedged QA run is therefore a capacity problem for everyone, not just for the person who dispatched it — sandbox deploys queue behind it.

If a run appears stuck, cancel it from the Actions tab rather than waiting.

Running test more than once piles up issues

There is no deduplication. See Issues & reporting. Close the old label's issues in bulk before re-testing, or use fix, which updates in place.

Failures can be quiet

Several things degrade silently instead of failing:

• A deleted plan comment → test invents a new plan and tests that.
• An unparsable stories.json → a green run with "0 stories".
• An agent crash before it emits its state directory → no artifacts at all,

including the log the error message tells you to read.

The habit that catches all three: read the summary comment's tally, not the green checkmark.

Technical view

Permissions

permissions:
  contents: write        # AI fixes are pushed to the PR branch
  pull-requests: write
  issues: write

— .github/workflows/sandbox-qa.yml:50-53

Unchanged by this PR. contents: write is the notable one — it is granted for all five actions, including the three that never commit. Narrowing it per-action is not expressible in a permissions: block (it is job-level, and there is one job), so tightening would mean splitting into separate jobs with an if: on each. Given that workflow_dispatch already requires write access to the repository, the practical exposure is the AI's own behavior rather than an access-control hole.

Note also that GITHUB_TOKEN is passed explicitly into the agent step (:87), so the agent — not just the workflow — acts with these permissions.

Concurrency

concurrency:
  group: sandbox-qa-${{ github.ref_name }}
  cancel-in-progress: false

— .github/workflows/sandbox-qa.yml:46-48

Unchanged, and correct for the new design: serializing per branch is exactly what makes a review → plan → test sequence safe to fire in quick succession. cancel-in-progress: false prevents a queued plan from killing a running review mid-write.

The group is keyed on ref_name, so two different branches can run QA concurrently as far as this block is concerned — but they cannot in practice, because they compete for the same single runner slot.

Runner and timeout

runs-on: [self-hosted, homelab, dentolize]
timeout-minutes: 300

— .github/workflows/sandbox-qa.yml:57-58

Five hours. Combined with cancel-in-progress: false and a single runner, the worst case is a five-hour repo-wide stall on sandbox operations. The header comment acknowledges the single-slot constraint explicitly as the reason fix commits use [skip ci] and the agent self-redeploys (.github/workflows/sandbox-qa.yml:16-19).

No actions/checkout step — consistent with sandbox.yml, which documents the reason: the sandbox CLI clones into its own shared tree, and adding a checkout would grow the runner's persistent volume per PR.

The input rename is a breaking change

mode → action, and every option value changed (.github/workflows/sandbox-qa.yml:26-36). workflow_dispatch inputs are part of the workflow's public interface. Anything that invoked this workflow programmatically —

gh workflow run sandbox-qa.yml --ref my-branch -f mode=test-only

— now fails with an unknown-input error. There is no alias, no deprecation window, and no mode input retained for compatibility. For a workflow dispatched by hand from the UI this is likely fine; it is only a problem if scripts, bots, or documentation elsewhere reference the old names. See Migration.

scope is over-declared

scope:
  description: Test scope (test action only)
  type: choice
  required: true
  default: all

— .github/workflows/sandbox-qa.yml:37-44

required: true on an input that applies to one of five actions means GitHub renders the dropdown unconditionally. GitHub Actions has no conditional-input mechanism, so the alternatives were required: false (same UI, no default) or folding scope into the action list as a sixth option (test-failed-only). The chosen shape is the least bad; the description carries the caveat.

The value reaches the agent only as QA_SCOPE in the environment (:88), never as an argument (:93). If the host script does not read that variable, failed-only degrades to all with no visible signal.

full-auto ignores your edits

if: inputs.action == 'plan' || inputs.action == 'test'

— .github/workflows/sandbox-qa.yml:61

full-auto is excluded from the fetch step. If you have curated a review and a plan on the PR and then dispatch full-auto, it will regenerate both from scratch and test its own version. This is internally consistent (full-auto means "do everything yourself") but it is a foot-gun for anyone who reads the header's promise that "each action's PR-comment output is editable and the next action uses your edits" (:3-5) and assumes it holds universally.

Use test, not full-auto, when you have edited a plan.

Leftovers from the rename

Two cosmetic pieces of drift, both harmless and both worth a follow-up commit:

const mode = action;

— .github/workflows/sandbox-qa.yml:101

A compatibility alias whose only remaining consumer is the mode **${mode}** string in the summary comment (:234). Since it is assigned from action, the summary prints the new vocabulary — the alias buys nothing.

// fix-issues updates the EXISTING story issues (matched by [QA-n]
// title prefix under the PR label); every other mode creates them.

— .github/workflows/sandbox-qa.yml:158-159

Comment still uses the pre-PR mode name fix-issues; the code beneath it tests action === "fix" (:161).

The critical label assumption

.github/workflows/sandbox-qa.yml:199 attaches a critical label to critical-priority stories, but the on-demand label creation loop at :144-150 only creates ai-automated and the PR label. If critical does not exist in the repository, issues.create throws, and — with no try/catch around it — the publish step fails partway through, after some issues have already been filed. Confirm the label exists.

What this diff cannot tell you

The workflow is a wrapper. Everything below is asserted by the header comment or the PR description and implemented outside this repository, in /opt/homelab/sandbox/bin/sandbox-qa:

None of this makes the claims false — the PR author reports a real review run succeeded on another branch. It means a reviewer of this PR is reviewing the GitHub-side contract only, and that the two halves can drift apart without any signal in this repository. If the script's output keys or file layout change, the failure surfaces as a silently degraded run, not a build break.