📚 Documentation Noob Test Report - March 11, 2026

[github-actions[bot]]

Summary

• Date of test: March 11, 2026
• Pages visited: Homepage, Quick Start, Creating Workflows, CLI Commands, How They Work, About Workflows (Overview), Engines, Frontmatter, Safe Outputs, Workflow Structure, Auth, Glossary, FAQ, Common Issues, Manual Examples, Comment-Triggered Examples
• Overall impression: The homepage is welcoming and the concept is exciting, but getting started hits a significant wall immediately: the docs site cannot be statically built, and a key command in the Quick Start guide (gh aw add-wizard) is missing from the CLI reference entirely.

---

🔴 Critical Issues Found

1. npm run build fails — 226 invalid anchor links in 79 files

The documentation site cannot produce a static build at all:

[ERROR] ✗ Found 226 invalid links in 79 files.

The root cause appears to be a discrepancy between how the Astro dev server and the static build generate anchor IDs for headings that contain backtick code spans.

For example, ### Issue Creation (\create-issue:`)` generates:

• Dev server: #issue-creation-create-issue ✅ (includes the YAML key)
• Static build: #issue-creation ❌ (strips code span content)

Pages like staged-mode.md (22 anchor links to safe-outputs.md) and footers.md link to the longer form, which breaks at build time.

Impact: Production deployment is blocked. CI/CD and GitHub Pages publishing will fail.

Affected files include (partial list): staged-mode.md, footers.md, triggering-ci.mdx, and 76 other files.

Suggested fix: Add explicit heading IDs using \{\#anchor-id} syntax to the key sections in safe-outputs.md that are widely linked, e.g.:

### Issue Creation (`create-issue:`) \{\#issue-creation-create-issue}

---

2. gh aw add-wizard command is undocumented in CLI reference

The Quick Start guide (Step 2) instructs new users to run:

gh aw add-wizard githubnext/agentics/daily-repo-status
```

This is the **main onboarding command** — the single most important step for new users. Yet there is **no `add-wizard` entry** in the [CLI Commands reference](/gh-aw/setup/cli/). The CLI page has `add`, `init`, `new`, etc. but not `add-wizard`.

A beginner will:
1. Run `gh aw add-wizard` (from Quick Start)
2. Want to understand what it does, what options it has, or how to use it for other workflows
3. Search the CLI docs — find nothing
4. Be confused

**Suggested fix**: Add an `#### \`add-wizard\`` section to `docs/src/content/docs/setup/cli.md` explaining what the wizard does, its syntax, and available options.

---

### 3. Typo in `introduction/overview.mdx` (missing verb)

Line 44:
> "The `gh aw compile` command **this** markdown file into a hardened GitHub Actions Workflow..."

Should read: "The `gh aw compile` command **compiles** this markdown file into..."

This is the first page many users read to understand how the tool works. A grammatical error undermines confidence.

**File**: `docs/src/content/docs/introduction/overview.mdx`, line 44

---

### 4. Code block syntax error in `introduction/how-they-work.mdx`

Line 14 has:

```
````
Should be `` ```aw wrap `` (not `warp`). This could cause the code block to render incorrectly in the static build.

**File**: `docs/src/content/docs/introduction/how-they-work.mdx`, line 14

---

## 🟡 Confusing Areas

### 5. `frontmatter` is used before it's explained

The very first code example on the homepage shows YAML frontmatter between `---` markers. The term is linked to the glossary, but:
- First-time users may not notice the tooltip link
- The link text appears mid-sentence without emphasis

**Suggested fix**: Add a one-sentence inline explanation on first use: *"The YAML section between `---` markers (called **frontmatter**) configures triggers and permissions..."*

### 6. The "lock file" concept needs more emphasis

The Quick Start mentions "regenerate the workflow YAML from the frontmatter" and "lock file" without clearly explaining what a lock file is or why it exists. A beginner might think `.github/workflows/` should only contain `.yml` files (standard GitHub Actions knowledge), not `.md` + `.lock.yml` pairs.

The concept IS explained in the FAQ and Glossary, but it needs a brief callout near the first mention in the Quick Start.

### 7. "What's next?" link in Quick Start leads to an unclear URL

Line 121 of `quick-start.mdx`:
```
Explore some of these in [Peli's Agent Factory](https://github.github.com/gh-aw/blog/2026-01-12-welcome-to-pelis-agent-factory/)

The github.github.com domain (the production docs URL per astro.config.mjs) is not yet publicly known/familiar. When reading the source, this link appears to be a circular reference within the docs. A note or context about what "Peli's Agent Factory" is would help.

8. Secrets setup in Quick Start lacks direct guidance

Step 2 says the wizard will "set up the required secret" with links to COPILOT_GITHUB_TOKEN, ANTHROPIC_API_KEY, or OPENAI_API_KEY. But the wizard isn't documented (see Issue #2), so users don't know if this is fully automated or requires manual steps.

Suggested fix: Add a brief note explaining whether the wizard sets secrets automatically or guides the user through it.

9. gh aw compile step 4 customization section is unclear about when to run it

The Quick Start says "If you have changed the frontmatter, regenerate the workflow YAML by running gh aw compile" — but beginners won't know what "frontmatter" is at this point, or what counts as a frontmatter change vs. a markdown body change.

---

🟢 What Worked Well

1. Homepage is compelling and clear — The tagline "Repository automation, running the coding agents you know and love, with strong guardrails in GitHub Actions" is excellent. Feature cards are concise and the code example immediately shows what a workflow looks like.

2. Quick Start has a great structure — The 4-step format with a ⏱️ time estimate is exactly right for a new user. Steps are numbered and concrete.

3. Glossary with tooltips throughout — Every technical term (frontmatter, lock file, safe outputs, MCP, etc.) is linked to a clear definition. The glossary is excellent and well-organized.

4. FAQ addresses real concerns upfront — Questions about determinism, costs, security, and whether you can write code are answered clearly. The "100% additive" framing for determinism is reassuring.

5. Auth page is beginner-friendly — Clear breakdown per engine (Copilot/Claude/Codex/Gemini), with a video, CLI commands, and UI screenshots. Covers both CLI and UI approaches.

6. Troubleshooting section is practical — The common issues page covers real errors with specific messages and step-by-step fixes. Shows the team has learned from real user problems.

7. Security explanation builds trust — The early explanation that workflows run read-only by default with safe outputs is reassuring for enterprise users worried about AI having write access.

8. CLI reference is comprehensive — The commands table at the top of the CLI page is a great quick-reference. Each command has clear examples.

---

Recommendations

Immediate (Blocks Getting Started)

1. Fix build failure — Investigate the anchor ID discrepancy between dev server and static build for headings with code spans. Add explicit \{\#anchor-id} to the most-linked sections in safe-outputs.md, or audit all 226 links.
2. Document add-wizard — Add #### \add-wizard`` to the CLI reference. This is the primary onboarding command.
3. Fix typo in overview.mdx — Add "compiles" to "The gh aw compile command compiles this markdown file..."
4. Fix warp typo — Change ```aw warp to ```aw wrap in how-they-work.mdx.

Quick Wins

5. Add inline frontmatter explanation on first use in Quick Start.
6. Add a "What is a lock file?" callout near the first mention in Quick Start.
7. Clarify secrets wizard behavior — Does add-wizard set secrets automatically?
8. Clarify when to recompile — Add a small note in the customize step explaining "frontmatter = top YAML section = triggers, permissions, tools".

Longer Term

9. Review all 226 broken anchor links — Systematically fix or add explicit anchor IDs to avoid future build failures.
10. Consider a "Your First Workflow Explained Line by Line" tutorial page — Walking through the daily-repo-status example with annotations would help noobs understand every part.
11. Add a "Before You Start" checklist to Quick Start — explicitly list: GitHub Copilot subscription OR Anthropic/OpenAI API key, GitHub CLI installed, repo with Actions enabled.

AI generated by Documentation Noob Tester · history

• expires on Mar 12, 2026, 8:11 AM UTC

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-03-12T08:11:08.616Z.

Closed by Workflow