📚 Documentation Noob Test Report - November 20, 2025

[github-actions[bot]]

📚 Documentation Noob Test Report - November 20, 2025

Summary

Test Date: November 20, 2025
Pages Visited: 7 core documentation pages
Overall Impression: The documentation is well-organized and visually appealing, but assumes too much prior knowledge for true beginners. Users familiar with GitHub Actions and AI concepts will navigate fine, but newcomers will hit several roadblocks.

Key Finding: The docs need a "true beginner" path with clearer prerequisites, simpler first examples, and jargon definitions.

---

🔴 Critical Issues Found

Issue #1: Prerequisites Exist But Buried in Text ⭐ HIGHEST PRIORITY

Page: Quick Start
Current State: Prerequisites are mentioned in a paragraph, easy to miss
Problem: A beginner skimming the page might miss critical requirements

Current text:

"You'll need a repository where you can push changes and add Actions secrets, with Actions, Discussions and Issues enabled. You'll also need a Personal Access Token (PAT) for GitHub Copilot CLI."

Why it's confusing:

• Assumes I know what "Actions secrets" are
• Doesn't mention GitHub CLI installation
• No mention of OS requirements
• All crammed into one dense sentence

Recommendation: Create a prominent checklist at the top:

## ✅ Prerequisites

Before you begin, make sure you have:

- [ ] **GitHub account** with access to a repository
- [ ] **GitHub CLI (gh)** installed ([installation guide](https://cli.github.com))
  - Verify: Run `gh --version` (requires v2.0.0 or higher)
- [ ] **Repository permissions:** Admin or write access
- [ ] **Repository features enabled:** 
  - GitHub Actions
  - Issues
  - Discussions
- [ ] **Operating System:** Linux, macOS, or Windows with WSL

Issue #2: No Success Indicators After Installation

Page: Quick Start - Step 1
Current State: Says to run gh extension install githubnext/gh-aw
Problem: Doesn't show what success looks like

As a noob, I'm asking:

• "Did it work?"
• "How do I know it installed?"
• "What if nothing happened?"

Recommendation: Add expected output:

gh extension install githubnext/gh-aw

# ✅ Expected output:
# ✓ Installed extension githubnext/gh-aw

# Verify installation:
gh aw --version

# ✅ Expected output:
# gh-aw version x.x.x

Issue #3: "Frontmatter" Used Without Definition

Page: Quick Start - Understanding Your First Workflow
Current text: "The section between the --- markers (called frontmatter)"
Problem: Never explains WHAT frontmatter is or WHY it exists

As a noob, I'm thinking:

• "Is frontmatter a GitHub thing? A YAML thing?"
• "Why is it called frontmatter?"
• "What happens if I mess it up?"

Recommendation: Add inline explanation or tooltip:

The configuration section at the top (called "frontmatter" - borrowed from 
blogging/documentation systems, meaning metadata at the front of the file) 
contains the YAML settings that control...

Or link to a glossary on first use.

Issue #4: "Compile" Step Needs Context

Page: Quick Start - Customize Your Workflow
Current text: "Customize your workflow by editing the .md file and recompiling with gh aw compile"
Problem: Doesn't explain WHY compilation is needed

As a noob, I'm confused:

• "I thought the .md file WAS the workflow?"
• "Why does it need compiling?"
• "What does compile actually do?"

Recommendation: Add brief explanation:

## Why Compile?

The `.md` file is human-friendly (natural language + simple config). GitHub Actions 
requires `.yml` format. The compile step translates your markdown into the YAML 
workflow file that GitHub Actions can execute.

Think of it like: Markdown (what you write) → YAML (what Actions runs)

Issue #5: First Example Too Complex

Page: Quick Start
Current state: "Daily Team Status" is the first example
Problem: Does too much for a beginner's first workflow

The example:

• Runs on cron schedule
• Reads multiple data sources
• Has complex permissions
• Uses safe-outputs
• Creates discussions

As a noob, I'm overwhelmed:

• "This seems complicated for my first try"
• "Can I see something simpler first?"

Recommendation: Add a "Hello World" example FIRST:

## Your First Workflow: Hello World

Before we dive into the daily status example, let's start with the simplest possible workflow:

---
name: hello-world
on:
  workflow_dispatch:
permissions:
  issues: write
tools:
  github:
---

# Hello World

Create a GitHub issue with title "Hello from AI!" and body "This is my first agentic workflow!"

Then close this issue and add a comment saying "Mission accomplished! 🎉"

Then show the complex example as "Next: A Real-World Example"

---

🟡 Confusing Areas

Confusing #1: "Agentic" Never Defined

Pages: Used throughout documentation
Problem: Core term never defined

Quick fix: Add to glossary and link on first use:

**Agentic workflows** = AI-powered workflows that can reason, make decisions, 
and take autonomous actions (unlike traditional workflows with fixed if/then rules)

Confusing #2: CLI vs Copilot CLI Relationship Unclear

Page: Quick Start - Customize section
Problem: Switches between gh aw and copilot commands without explaining

As a noob:

• "Are these different tools?"
• "Do I need both?"
• "When do I use which?"

Recommendation: Add callout:

## 🔧 Two Tools, Different Purposes

- **GitHub Agentic Workflows CLI** (`gh aw`) - Manages, compiles, and runs your workflows
  - Install: `gh extension install githubnext/gh-aw`
  - Use when: Running workflows, adding samples, checking status

- **GitHub Copilot CLI** (`copilot`) - Helps you write workflows with AI assistance (OPTIONAL)
  - Install: `npm install -g `@github/copilot-cli``
  - Use when: Want AI help creating/editing workflows
  - Not required: You can write workflows in any text editor

Confusing #3: "Safe Outputs" Not Explained

Page: Quick Start example
Problem: Shows safe-outputs: create-discussion but doesn't explain it

Recommendation: Add tooltip or inline note:

safe-outputs:          # Actions the AI can safely take
  create-discussion:   # Without this, the workflow would need write permissions

Confusing #4: Token Permissions "see note below" Confusing

Page: Quick Start - Step 3
Current text: "If you are not finding this option, review steps 2 and 3"
Problem: Circular reference - we're already IN step 3!

Recommendation: Rewrite:

⚠️ Don't see "Copilot Requests" permission?
- Make sure you selected YOUR user account (not an organization)
- Organization accounts require different setup - see [Enterprise Setup Guide]

Confusing #5: No Visual Workflow Structure

Page: Reference > Workflow Structure
Problem: Lots of text, no diagram

Recommendation: Add visual diagram:

┌─────────────────────────────────┐
│  Frontmatter (YAML Config)      │
│  ─────────────────────────      │
│  • When to run (triggers)       │
│  • What permissions needed      │
│  • What tools available         │
└─────────────────────────────────┘
           ↓
┌─────────────────────────────────┐
│  Instructions (Natural Language)│
│  ─────────────────────────────  │
│  • What to do                   │
│  • How to do it                 │
│  • What output to create        │
└─────────────────────────────────┘
           ↓
        Compile
           ↓
┌─────────────────────────────────┐
│  .lock.yml (GitHub Actions)     │
│  Generated automatically        │
└─────────────────────────────────┘

---

✅ What Worked Well

Good: Prerequisites Section Exists!

The Quick Start DOES have a prerequisites section - this is great! Just needs to be more prominent and detailed.

Good: Clear Navigation

Sidebar organization (Setup → Introduction → Reference → Examples) makes logical sense.

Good: Syntax Highlighting

Code blocks are well-formatted and easy to read.

Good: "What's Next" Section

Quick Start ends with clear next steps - helps users know where to go.

Good: Search Functionality

Prominent search bar makes finding topics easy.

Good: Warning Callouts

The "Caution" box about research demonstrator sets expectations well.

---

📊 Recommendations (Prioritized)

🏃 Quick Wins (< 1 hour each)

1. Expand Prerequisites checklist (30 min)

   • Add OS requirements
   • Add gh CLI version check
   • Make it a visual checklist

2. Add "Hello World" example before complex example (30 min)

   • Create simple workflow that just creates an issue
   • Progressive disclosure: simple → complex

3. Add expected output after install commands (15 min)

   • Show what success looks like
   • Show how to verify installation

4. Define "agentic" in glossary (15 min)

   • Link on first use
   • Add to reference section

5. Fix circular reference in token permissions (10 min)

   • Rewrite confusing "review steps 2 and 3" note

🚶 Medium-Term (1-3 hours each)

6. Add workflow structure diagram (1-2 hours)

   • Visual representation of .md → .yml
   • Annotated example showing each part

7. Clarify CLI vs Copilot CLI relationship (1 hour)

   • Add comparison table
   • Explain when to use each

8. Expand "frontmatter" explanation (30 min)

   • Why it's called frontmatter
   • What happens if you mess it up
   • Link to YAML basics if needed

9. Add "Common Issues" section to Quick Start (2 hours)

   • Installation problems
   • Permission issues
   • Token errors

10. Create troubleshooting flowchart (2 hours)

    • "My workflow didn't run" → check X, Y, Z
    • "Permission denied" → check A, B, C

🏗️ Long-Term Enhancements

11. Video walkthrough of Quick Start (half day)

    • Screen recording showing each step
    • Real-time troubleshooting

12. Interactive tutorial (multiple days)

    • Like GitHub Learning Lab
    • Hands-on exercises

13. Beginner vs Advanced mode toggle (multiple days)

    • Show/hide advanced details
    • Let users choose their depth

14. Example difficulty ratings (half day)

    • Tag examples: Beginner, Intermediate, Advanced
    • Filter by difficulty

---

🎯 Success Metrics

After implementing these changes, a beginner should be able to:

1. ✅ Install gh-aw in < 5 minutes (with clear success confirmation)
2. ✅ Run their first workflow in < 10 minutes (Hello World)
3. ✅ Understand what "agentic" means
4. ✅ Know when to use gh aw vs copilot
5. ✅ Find help when something goes wrong
6. ✅ Customize the example workflow without fear

Current state: Steps 1, 3, 4, 5 would be challenging
Target state: All 6 steps should be straightforward

---

🧪 Test Methodology

This review simulated a complete beginner by:

• Reading documentation in order (as a new user would)
• Noting every point of confusion or uncertainty
• Not assuming knowledge of GitHub Actions, YAML, or AI workflows
• Identifying where I'd get stuck if following step-by-step
• Focusing on the "getting started" path (most critical for adoption)

Tools used:

• Manual review of 7 core pages
• Automated HTML parsing for jargon detection
• Text analysis for clarity and completeness

---

🙏 Final Thoughts

The GitHub Agentic Workflows documentation is well-structured and comprehensive for users with GitHub Actions experience. However, to reach a wider audience and enable true beginners to succeed, the docs need:

1. More hand-holding in the Quick Start (clearer prerequisites, simpler first example)
2. Less jargon or better jargon definitions (agentic, frontmatter, safe-outputs)
3. Better success indicators (show what success looks like)
4. More visuals (diagrams, flowcharts, annotated screenshots)
5. Clearer tool relationships (gh aw vs copilot)

Bottom line: The content is there, it just needs to be more accessible to newcomers.

---

This test was conducted on November 20, 2025, as an automated documentation review workflow.

Labels: documentation, user-experience, automated-testing

AI generated by Documentation Noob Tester

[pelikhan]

/plan

[github-actions[bot]]

✅ Agentic Plan Command completed successfully.