Stopping missed spec updates with process, not attention, in GitHub Actions: A PoC for auto PR/Noop with gh-aw

tl;dr

• Missed documentation updates cannot be solved by attention alone
• The baseline is "update docs in the same PR," but leaks still happen, so you need a second line of defense
• With gh-aw, automating "create a PR if docs updates are needed / emit noop if not" makes human review easier
• I only got stuck in two places: Safe Outputs and GitHub Actions PR creation permissions
• Polishing PR output (Japanese text + non-draft PR) also removes day-to-day operational friction

Background: docs will drift if your strategy is only "be careful"

Then and now, one unavoidable challenge is documentation management.
Code gets updated while specs and runbooks stay stale. It happens all the time.

This is even easier to trigger in the AI era.
As AI speeds up source code generation and edits, both change volume and velocity increase, and the chance that docs get left behind rises with them.

"Let's be careful" is not a winning strategy here.
So it is more realistic to first build a flow that automatically detects drift and makes misses harder.

Of course, the cleanest approach is still to bundle docs updates into the same code-change PR.
That is how we normally operate too (one place to review, one place to inspect diffs).

But even then, leaks happen.

• Busy review days
• Days when changes impact more than expected
• Days when AI updates code quickly and creates more follow-up tasks

On days like these, "correct code" passes and only docs get left behind.
So this is not about rejecting "fix it in the PR." It is about adding a secondary defense that detects misses automatically.

Incidents always arrive looking like "the code is correct"

We changed shipping-fee calculation logic.

• The code works
• Tests pass
• The PR is merged

...yet the spec stayed outdated.

Not rare at all. If anything, common.
That is why I wanted to move from "people should remember" to a system where drift automatically gets blocked.

As a mitigation, I used GitHub Agentic Workflows (gh-aw) to:

• Trigger from changes under src/ and related files
• Re-check impacted docs
• Create a PR if updates are needed
• Emit noop if nothing is needed, to make "no change" explicit

This PoC is about building that flow.

The goal here is not "fully auto-generate perfect documentation."
It is to build a mechanism that makes missed updates harder to overlook.

Approach: first defense (fix in PR) + second defense (auto-catch misses)

• First defense: update docs in the same PR

  • This is the cleanest option. Diff and review both finish in one pass.

• Second defense: if something still leaks, trigger on push to main and auto-create a follow-up PR

  • Prevents "a miss quietly moving forward."
  • If no fix is needed, emit noop to keep a visible record that the workflow ran and did nothing.

This PoC is specifically about implementing the second defense as a reusable process.

What I wanted gh-aw to do (encode human review judgment as a repeatable machine flow)

The intent is simple:

1. Trigger on src/ changes
2. Re-check impacted docs
3. Open a PR if changes are required
4. Emit noop explicitly if no changes are required

The key point is not "AI is smart, so it will find everything."

I want the same judgment humans make during review
to run through the same procedure every time.

First attempt: it detected drift, but produced zero artifact (No Safe Outputs)

The first experiment was interesting.

AI judged that docs updates were needed.
The logs even showed a plausible completion summary. That summary is human-readable text; actual actions like PR creation require separate structured data via safe outputs.

...but the workflow still failed.

Why:

Safe Outputs were not generated
(No Safe Outputs Generated)

At first it feels strange: "AI did work, so why is it invalid?"
But once you understand gh-aw's design, it makes sense.

• The agent itself is basically read-only
• Write operations (for example PR creation) must be explicitly emitted through safe outputs

So even if AI "feels like it committed locally," it does not count as a valid artifact.

I changed the instructions like this:

• If changes exist, always call create-pull-request
• If no changes exist, always call noop
• Do not commit directly (emit artifacts through safe outputs)

Second attempt: PR creation failed because of permissions

Next, create-pull-request was invoked.

I thought that would finish it, but then another mine appeared:

GitHub Actions is not permitted to create or approve pull requests.

This is not a code issue. It is a GitHub settings issue.

If Actions workflow permissions are read-only, PR creation fails.
In other words:

The mechanism is correct, but it still fails because permissions are insufficient.

Classic CI behavior.

This was resolved by enabling two settings in the UI (details below).

Final form: enforce PR/Noop, create non-draft PRs, and keep PR text in Japanese

At this point, I wanted more than "it works." I wanted something easy to run in daily operations.

• Draft PRs add unnecessary friction
• English PR descriptions are easier to skim past in our context
• Silent completion when there are no changes leaves people wondering whether it ran

So the final form is:

• If docs are updated, open a PR (Ready for review)
• If no update is needed, emit noop
• Keep PR title/body in Japanese
• Restrict edits to /docs and README.md only (never touch source)

The final docs-auto-fix.md looked like this and felt operationally solid:

Note: ideally this should run on PR events and fix before merge.
Here, because we already do that and still see occasional misses, we use pushes to main as a second-defense trigger.

1234567891011121314151617181920212223242526272829303132333435363738394041---
on:
  push:
    branches:
      - main
    paths:
      - "src/**"
      - "test/**"
      - "package.json"
      - "tsconfig*.json"
      - "nest-cli.json"
permissions:
  contents: read
  issues: read
  pull-requests: read
safe-outputs:
  create-pull-request:
    draft: false
  noop:
    max: 1
tools:
  edit:
---
# Docs Auto-Fix

## Goal
Keep documentation in `/docs` and `README.md` consistent with recent source code changes.

## Instructions
1. Identify what changed in this push (check the git diff or review the modified files under `src/`, `test/`, and config files).
2. Determine which docs are impacted and update them for accuracy.
3. Only modify documentation files under `/docs` and `README.md`. Do not change source code.
4. Do NOT commit directly. If documentation updates are made, use the safe output tool `create-pull-request` to propose changes.
5. If no documentation updates are needed, use the safe output tool `noop` to explicitly report no action.
6. When creating a pull request, write the PR title and body in Japanese. Include a short summary and test status.
7. Keep updates minimal, precise, and aligned with existing formatting.

## Repo Notes
> Note: "Repo Notes" is an in-prompt memo to pin SSOT locations, not a built-in gh-aw feature.

- If shipping fee rules or API behavior change, update `/docs/shipping-fee-spec.md`.

When docs updates are needed, create-pull-request runs and a PR is created (see image below).

Setup Steps (copy-paste checklist)

This is where people usually get stuck, so here is a checklist including settings.

1) Add the workflow file

• Create .github/workflows/docs-auto-fix.md (use the content above)

2) Generate lock file

gh aw compile

Also commit .github/workflows/docs-auto-fix.lock.yml.

3) Configure GitHub Actions (common failure point)

In repo Settings → Actions → General:

• Workflow permissions: Read and write permissions
• Allow GitHub Actions to create and approve pull requests: ON

Without this, create-pull-request fails with a permissions error.

4) Add secret (if using Copilot)

Repo → Settings → Secrets and variables → Actions → New repository secret

• Add COPILOT_GITHUB_TOKEN

(Use a fine-grained PAT with "Copilot Requests." I prefer this because it keeps permissions minimal.)

5) Verify behavior

• Modify files under src/ and push to main
• If docs updates are needed, a PR is created
• If not needed, noop is emitted (you can see that it ran and intentionally did nothing)

Lessons learned: build a system that blocks misses, not a reminder to be careful

The most interesting part was this:

• AI can make judgments
• But structured protocol is required for artifacts to be accepted (safe outputs)
• And permissions are reality (Actions PR permissions)

So the winning path is not "let AI do everything."

Humans design the lane where AI is allowed to act.

Once that lane exists, missed spec updates stop being an "attention" problem and become a "process" problem you can systematically reduce.

As speed increases, drift increases too.
So instead of "let's remember to update docs," it is more effective to set up fix in PR (first defense) + automatic post-merge detection (second defense) upfront.

Sample repository:

https://github.com/thundermiracle/ai-auto-docs