🎯 Repository Quality Improvement - Workflow Compilation Health & Synchronization

[github-actions[bot]]

🎯 Repository Quality Improvement Report - Workflow Compilation Health & Synchronization

Analysis Date: 2025-11-16
Focus Area: Workflow Compilation Health & Synchronization
Strategy Type: Custom (Repository-Specific)
Custom Area: Yes - This focus area is unique to gh-aw's workflow compilation system, addressing the synchronization and health monitoring of the .md → .lock.yml compilation pipeline

Executive Summary

This analysis examined the health and reliability of gh-aw's unique workflow compilation system that transforms markdown workflow files (.md) into GitHub Actions workflows (.lock.yml). With 118 workflow source files and 83 compiled outputs, the compilation system is a critical component of the gh-aw developer experience.

Key findings reveal strong compilation infrastructure with 65% safe-outputs adoption, 20+ compilation test files, and automated CI/CD recompilation. However, opportunities exist to improve synchronization monitoring, enhance error reporting, and create better developer tooling for compilation health checks. Six workflows show potential staleness, and the lack of real-time compilation health monitoring could lead to deployment of outdated workflows.

The analysis identified 5 high-impact tasks for Copilot agent execution focused on compilation monitoring, automated synchronization checks, enhanced error messages, and developer tooling improvements.

Full Analysis Report

Focus Area: Workflow Compilation Health & Synchronization

Rationale for Custom Focus Area

The workflow compilation system (.md → .lock.yml) is unique to gh-aw and represents a critical quality dimension not covered by standard categories. This system:

• Transforms 118 natural language workflow files into executable GitHub Actions
• Requires perfect synchronization between source and compiled files
• Has complex validation spanning YAML schema, GitHub Actions limits, and MCP configurations
• Directly impacts developer productivity and workflow reliability
• Involves 444 Go files in pkg/workflow/ with sophisticated compilation logic

This custom focus area addresses repository-specific quality concerns around compilation reliability, synchronization health, and developer experience that would not be captured by generic quality metrics.

Current State Assessment

Metrics Collected:

Metric	Value	Status
Total .md workflow files	118	✅
Compiled .lock.yml files	83	✅
Shared components (non-compiled)	36	✅
Main workflows	80	✅
Uncompiled main workflows	0	✅
Stale .lock.yml files	6	⚠️
Orphaned .lock.yml files	0	✅
Safe-outputs adoption	65% (77/118)	✅
Workflows with imports	53	✅
Strict mode adoption	25% (30/118)	⚠️
Compilation test files	20+	✅
Console error messages in compiler	48	✅
Average workflow complexity	194 lines	✅
Largest compiled workflow	8,145 lines	⚠️
Validation mechanisms	10+ files	✅

Findings

Strengths

• No orphaned lock files: Perfect cleanup - all .lock.yml files have corresponding .md sources
• No uncompiled main workflows: All production workflows are compiled (shared components correctly excluded)
• Robust test coverage: 20+ compilation test files covering various scenarios
• Strong error handling: 48 console-formatted error messages, 22 error handling points in compiler.go
• CI/CD integration: Automated recompilation in CI pipeline ensures consistency
• Comprehensive validation: 10+ validation files covering schema, runtime, strict mode, MCP, pip, npm, expressions
• High safe-outputs adoption: 65% of workflows use safe outputs for secure GitHub API interactions
• Well-documented compilation process: Clear guidance in AGENTS.md with multiple "ALWAYS RECOMPILE" reminders
• Organized codebase: Compiler split into focused files (compiler.go, compiler_yaml.go, compiler_jobs.go, etc.)
• Size limits enforced: MaxLockFileSize (1MB), MaxExpressionSize (21KB), MaxPromptChunkSize (20KB) prevent bloat

Areas for Improvement

• Stale workflow detection (6 stale files) - Workflows modified after their lock files compiled, indicating potential drift
• No real-time health monitoring - Missing dashboard or command to check compilation health at a glance
• Large compiled workflows - Largest workflow is 8,145 lines, approaching complexity limits
• Strict mode adoption low - Only 25% use strict mode, missing security benefits
• No automated staleness detection - Developers must manually track if workflows need recompilation
• Limited compilation performance metrics - No tracking of compilation time or performance bottlenecks
• MCP server adoption low - Only 1% of workflows use MCP servers, underutilizing capabilities

Detailed Analysis

1. Compilation Synchronization

The analysis revealed 6 potentially stale workflows where the .md file was modified after the .lock.yml compilation:

• archie.md / archie.lock.yml
• blog-auditor.md / blog-auditor.lock.yml
• dictation-prompt.md / dictation-prompt.lock.yml
• duplicate-code-detector.md / duplicate-code-detector.lock.yml
• pr-nitpick-reviewer.md / pr-nitpick-reviewer.lock.yml
• schema-consistency-checker.md / schema-consistency-checker.lock.yml

Note: Timestamp analysis shows these were modified at the same timestamp (2025-11-16 17:57:36), suggesting they may have been updated during repository checkout. However, this highlights the need for automated staleness detection.

2. Compilation Process Architecture

Strengths:

• Modular design: Compiler split across multiple focused files
  • compiler.go - Main orchestration (21 error points)
  • compiler_yaml.go - YAML generation
  • compiler_jobs.go - Job generation
  • compiler_test.go - Comprehensive tests
• Multiple validation layers:
  • Schema validation (JSON Schema for frontmatter)
  • Runtime validation (GitHub Actions limits)
  • Expression validation (safety checks)
  • Strict mode validation (security enforcement)
  • MCP config validation
  • Package validation (pip, npm)
• Developer-friendly:
  • make recompile - One command to recompile all
  • --purge flag to clean orphaned files
  • --validate flag for validation-only
  • --verbose flag for debugging

Gaps:

• No gh aw compile --check command to verify health without recompiling
• No compilation performance tracking
• No easy way to list stale workflows
• No automated pre-commit hooks to ensure compilation

3. Large Workflow Management

Five workflows exceed 6,000 lines in compiled form:

• poem-bot.lock.yml: 8,145 lines
• cloclo.lock.yml: 6,724 lines
• q.lock.yml: 6,555 lines
• technical-doc-writer.lock.yml: 6,213 lines
• unbloat-docs.lock.yml: 6,185 lines

These large files may indicate:

• Complex workflow logic that could be modularized
• Extensive safe-output processing
• Large embedded scripts or configurations
• Potential performance concerns during compilation

4. Developer Experience Gaps

Current DX Strengths:

• Clear "ALWAYS RECOMPILE" messaging in AGENTS.md
• Makefile shortcuts for compilation
• CI enforcement prevents forgotten recompilation
• Helpful error messages with console formatting

DX Improvement Opportunities:

• No quick health check command
• Stale workflow detection requires manual timestamp comparison
• No guidance on when to use strict mode
• Limited visibility into compilation performance
• No automated suggestions for workflow optimization

---

🤖 Tasks for Copilot Agent

NOTE TO PLANNER AGENT: The following tasks are designed for GitHub Copilot agent execution. Please split these into individual work items for Claude to process.

Improvement Tasks

The following code regions and tasks should be processed by the Copilot agent. Each section is marked for easy identification by the planner agent.

---

Task 1: Create Compilation Health Check Command

Priority: High
Estimated Effort: Medium
Focus Area: Workflow Compilation Health

Description:
Implement a gh aw compile --health command that provides a comprehensive health report on workflow compilation status without recompiling. The command should:

• List all .md files and their compilation status
• Identify stale workflows (where .md is newer than .lock.yml)
• Report orphaned .lock.yml files (no corresponding .md)
• Show uncompiled workflows
• Display summary statistics (total workflows, compiled %, stale %, etc.)
• Exit with non-zero code if health issues detected (for CI integration)

Acceptance Criteria:

• New --health flag added to compile command
• Health check runs without modifying any files
• Identifies all 6 categories: compiled, stale, orphaned, uncompiled, shared components, total
• Colored console output using pkg/console for readability
• Exit code 0 if healthy, 1 if issues found
• JSON output option with --json flag for scripting
• Documentation added to CLI help and README
• Unit tests covering all health check scenarios

Code Region: pkg/cli/compile_command.go, pkg/workflow/compiler.go

Create a workflow compilation health check feature in the gh aw CLI.

Add a new `--health` flag to the `gh aw compile` command that performs non-invasive health checking:

1. In `pkg/cli/compile_command.go`:
   - Add `health` boolean flag to compile command
   - When `--health` is set, call new `compiler.CheckHealth()` method instead of `Compile()`
   - Format output using `console` package for colored terminal output
   - Return appropriate exit codes (0=healthy, 1=issues)

2. In `pkg/workflow/compiler.go`:
   - Implement `CheckHealth() (*HealthReport, error)` method
   - Scan .github/workflows directory for .md files
   - For each .md file, check if .lock.yml exists and compare timestamps
   - Identify shared components in .github/workflows/shared/ (should not have .lock.yml)
   - Find orphaned .lock.yml files (no corresponding .md)
   - Calculate statistics and categorize workflows

3. Create `pkg/workflow/health_report.go`:
   - Define `HealthReport` struct with fields:
     - TotalWorkflows int
     - CompiledWorkflows []string
     - StaleWorkflows []WorkflowStatus
     - UncompiledWorkflows []string
     - OrphanedLockFiles []string
     - SharedComponents []string
     - Summary HealthSummary
   - Implement `PrintReport()` method for console output
   - Implement `ToJSON()` method for JSON output

4. Add tests in `pkg/workflow/compiler_health_test.go`:
   - Test detection of stale workflows
   - Test detection of orphaned files
   - Test detection of uncompiled workflows
   - Test proper handling of shared components
   - Test summary statistics calculation

Use the console package for all output formatting. Ensure the feature is well-documented with examples.

---

Task 2: Implement Pre-Commit Compilation Validation Hook

Priority: High
Estimated Effort: Small
Focus Area: Developer Experience

Description:
Create a Git pre-commit hook that automatically detects when workflow .md files are modified and ensures their .lock.yml files are up-to-date before allowing the commit. This prevents developers from accidentally committing stale workflows.

Acceptance Criteria:

• Pre-commit hook script created in .git-hooks/pre-commit
• Hook detects modified .md files in .github/workflows/
• Hook verifies corresponding .lock.yml files exist and are current
• Hook provides clear error messages when compilation needed
• Hook skips shared components in .github/workflows/shared/
• Installation instructions added to CONTRIBUTING.md
• Hook can be bypassed with --no-verify for emergency commits
• Hook runs quickly (<2 seconds for typical commits)

Code Region: .git-hooks/pre-commit, CONTRIBUTING.md

Create a Git pre-commit hook to enforce workflow compilation before commits.

1. Create `.git-hooks/pre-commit` shell script:
   - Use `git diff --cached --name-only` to find modified .md files in .github/workflows/
   - Exclude files in .github/workflows/shared/ (shared components)
   - For each modified .md file, check if corresponding .lock.yml exists
   - If .lock.yml exists, compare timestamps (fail if .md is newer)
   - If .lock.yml doesn't exist, fail with error
   - Provide helpful error message with `make recompile` command
   - Exit 0 if all checks pass, exit 1 if any workflow needs compilation

2. Make hook executable:
   ```bash
   chmod +x .git-hooks/pre-commit

3. Update CONTRIBUTING.md:

   • Add section "Setting Up Git Hooks"
   • Provide installation command: git config core.hooksPath .git-hooks
   • Explain what the pre-commit hook validates
   • Document bypass method for emergencies: git commit --no-verify

4. Add .git-hooks/README.md:

   • Document purpose of each hook
   • Explain installation process
   • Provide troubleshooting tips

Example error message format:

ERROR: Workflow files modified but not compiled:
  - .github/workflows/example.md

Please run 'make recompile' before committing.
To bypass this check (not recommended): git commit --no-verify

Ensure the hook is fast and provides actionable feedback to developers.

---

#### Task 3: Add Compilation Performance Metrics

**Priority**: Medium  
**Estimated Effort**: Medium  
**Focus Area**: Compilation Performance

**Description:**
Instrument the compilation process to track and report performance metrics. This will help identify bottlenecks and optimize the compilation pipeline as the number of workflows grows.

**Acceptance Criteria:**
- [ ] Compilation time tracked per workflow
- [ ] Overall compilation time for batch operations
- [ ] Performance metrics logged when `--verbose` flag is used
- [ ] Summary report shows slowest workflows to compile
- [ ] Metrics include: parse time, validation time, YAML generation time, file I/O time
- [ ] Optional `--profile` flag for detailed performance profiling
- [ ] Performance regression tests to catch slowdowns
- [ ] Documentation in DEVGUIDE.md about performance optimization

**Code Region:** `pkg/workflow/compiler.go`, `pkg/workflow/compiler_performance.go`

```markdown
Add performance tracking and profiling to the workflow compilation system.

1. Create `pkg/workflow/compiler_performance.go`:
   - Define `CompilationMetrics` struct:
     ```go
     type CompilationMetrics struct {
         WorkflowPath     string
         ParseDuration    time.Duration
         ValidateDuration time.Duration
         GenerateDuration time.Duration
         WriteDuration    time.Duration
         TotalDuration    time.Duration
     }
     ```
   - Define `PerformanceTracker` interface
   - Implement `DefaultPerformanceTracker` with reporting methods

2. Update `pkg/workflow/compiler.go`:
   - Add `performanceTracker` field to Compiler struct
   - Instrument `Compile()` method with timing measurements:
     - Start timer before parsing
     - Record parse time
     - Record validation time
     - Record YAML generation time
     - Record file write time
     - Calculate total time
   - Add `WithPerformanceTracking()` option method

3. Update `pkg/cli/compile_command.go`:
   - When `--verbose` is set, enable performance tracking
   - After compilation batch, print performance summary:
     - Total compilation time
     - Average time per workflow
     - Slowest 5 workflows
     - Time breakdown by phase (parse, validate, generate, write)
   - Add optional `--profile` flag for detailed profiling output

4. Add performance tests in `pkg/workflow/compiler_performance_test.go`:
   - Benchmark compilation of small, medium, large workflows
   - Test that compilation time scales linearly with workflow count
   - Add regression test that fails if compilation time increases >20%

Output example:

Compilation Performance Summary
Total time: 2.34s
Workflows compiled: 118
Average per workflow: 19.8ms

Slowest workflows:

1. poem-bot.md: 234ms
2. cloclo.md: 189ms
3. q.md: 176ms

Phase breakdown:
Parse: 23% (538ms)
Validate: 45% (1.05s)
Generate: 28% (655ms)
Write: 4% (93ms)

---

Task 4: Create Workflow Complexity Analyzer

Priority: Medium
Estimated Effort: Large
Focus Area: Workflow Quality

Description:
Build a complexity analyzer that identifies workflows approaching size limits, complexity thresholds, or best practice violations. This helps prevent workflows from becoming unmaintainable and identifies refactoring opportunities.

Acceptance Criteria:

• Analyze compiled .lock.yml file sizes and warn when approaching 1MB limit
• Count steps per workflow and flag workflows with >50 steps
• Identify workflows with deep nesting (>3 levels in jobs/steps)
• Check for duplicate code patterns across workflows
• Measure expression complexity (nested expressions, long expressions)
• Generate refactoring suggestions for complex workflows
• Exportable report in JSON and markdown formats
• Integration with gh aw compile --analyze command

Code Region: pkg/workflow/analyzer.go, pkg/cli/compile_command.go

Create a workflow complexity analyzer to identify quality issues and refactoring opportunities.

1. Create `pkg/workflow/analyzer.go`:
   - Define `ComplexityReport` struct:
     ```go
     type ComplexityReport struct {
         WorkflowPath       string
         LockFileSize       int64
         LineCount          int
         StepCount          int
         JobCount           int
         NestingDepth       int
         ExpressionCount    int
         LongExpressions    []string
         Warnings           []Warning
         Suggestions        []Suggestion
         ComplexityScore    int
     }
     ```
   - Implement `AnalyzeWorkflow(lockFilePath string) (*ComplexityReport, error)`
   - Define complexity scoring algorithm (0-100 scale)
   - Implement warning generators:
     - Size warnings (>800KB = warn, >900KB = error)
     - Step count warnings (>50 steps)
     - Nesting warnings (>3 levels)
     - Expression warnings (>15KB expressions)

2. Create `pkg/workflow/analyzer_suggestions.go`:
   - Implement suggestion generators:
     - "Split into multiple workflows" for >100 steps
     - "Use shared components" for duplicate patterns
     - "Simplify expressions" for long/complex expressions
     - "Enable strict mode" for non-strict workflows
   - Each suggestion includes: reason, impact, effort estimate

3. Update `pkg/cli/compile_command.go`:
   - Add `--analyze` flag to compile command
   - After compilation, run complexity analysis on all workflows
   - Display summary report:
     - Top 10 most complex workflows
     - Count of warnings by severity
     - Count of suggestions by type
     - Overall health score
   - Support `--format json` for machine-readable output

4. Add tests in `pkg/workflow/analyzer_test.go`:
   - Test size detection for various file sizes
   - Test step counting accuracy
   - Test nesting depth calculation
   - Test suggestion generation logic
   - Test complexity scoring algorithm

Example output:

Workflow Complexity Analysis

High Complexity Workflows:

1. poem-bot.md (Score: 87/100) ⚠️

   • Size: 8,145 lines (approaching limit)
   • Steps: 76 (consider splitting)
   • Suggestions: 3

2. cloclo.md (Score: 72/100) ⚠️

   • Size: 6,724 lines
   • Steps: 58
   • Suggestions: 2

Warnings: 12 total

• Size warnings: 5
• Complexity warnings: 7

Suggestions: 18 total

• Split workflows: 3
• Use shared components: 8
• Simplify expressions: 4
• Enable strict mode: 3

Overall Repository Health: 78/100 ✅

---

Task 5: Enhanced Compilation Error Messages with Examples

Priority: High
Estimated Effort: Small
Focus Area: Developer Experience

Description:
Improve error messages during workflow compilation by adding concrete examples of valid syntax and pointing to specific lines in the workflow file. This reduces debugging time and improves the learning curve for new contributors.

Acceptance Criteria:

• All validation errors include line numbers from source .md file
• Errors show the problematic YAML snippet
• Errors provide example of correct syntax
• Errors link to relevant documentation when available
• Common errors have detailed troubleshooting guides
• Error messages use console formatting for readability
• Validation errors grouped logically (schema errors, GitHub Actions errors, etc.)
• Unit tests verify error message quality

Code Region: pkg/workflow/validation*.go, pkg/workflow/compiler.go, pkg/parser/*.go

Enhance workflow compilation error messages with context, examples, and actionable guidance.

1. Update `pkg/workflow/validation.go`:
   - Modify error creation to include line numbers from source
   - Use error wrapping with `%w` to preserve error chains
   - Add helper function `formatValidationError(err error, line int, context string) error`
   - Include YAML snippet in error output (3 lines before/after)

2. Update validation files to use new error format:
   - `pkg/workflow/strict_mode_validation.go`
   - `pkg/workflow/expression_validation.go`
   - `pkg/workflow/mcp_config_validation.go`
   - `pkg/workflow/runtime_validation.go`
   
   For each validation error:
   - Extract line number from YAML parser
   - Include example of correct syntax
   - Add link to docs when applicable

3. Create `pkg/workflow/error_examples.go`:
   - Define map of common errors to example fixes:
     ```go
     var ErrorExamples = map[string]string{
         "invalid engine": "engine: copilot\n# Valid: copilot, claude, codex, custom",
         "missing permissions": "permissions:\n  contents: read\n  issues: write",
         // ... more examples
     }
     ```
   - Implement `GetErrorExample(errorType string) string`

4. Update `pkg/console` package:
   - Add `FormatValidationError()` function for consistent formatting
   - Include color coding: red for errors, yellow for warnings, green for suggestions
   - Format example code with syntax highlighting

5. Add comprehensive tests in `pkg/workflow/validation_errors_test.go`:
   - Verify all error messages include line numbers
   - Verify examples are syntactically correct
   - Verify error messages are under 500 characters
   - Test error message rendering in console

Example error output:

✗ Validation Error in workflow.md:15

13 | on: issues
14 | permissions:
15 | issue: write
^^^^^
16 | engine: copilot
17 |

Invalid permission: 'issue'

Valid GitHub permissions are:

• contents
• issues (note the 's')
• pull-requests
• discussions
• actions

Example:
permissions:
contents: read
issues: write

Documentation: https://docs.github.com/actions/security-guides/automatic-token-authentication#permissions-for-the-github_token

Ensure error messages are helpful, actionable, and include enough context for quick resolution.

---

📊 Historical Context

Previous Focus Areas

Date	Focus Area	Type	Custom	Key Outcomes
2025-11-16	Workflow Compilation Health & Synchronization	Custom	Y	First quality improvement run - established baseline metrics, identified 6 stale workflows, created 5 high-priority tasks for compilation health monitoring

---

🎯 Recommendations

Immediate Actions (This Week)

1. Implement compilation health check command (gh aw compile --health) - Priority: High

   • Provides visibility into compilation status
   • Enables quick checks without full recompilation
   • Foundation for CI integration

2. Create pre-commit hook for compilation validation - Priority: High

   • Prevents stale workflow commits
   • Improves developer experience
   • Reduces compilation-related bugs

3. Enhance error messages with examples - Priority: High

   • Reduces debugging time for workflow authors
   • Lowers barrier to entry for contributors
   • Improves overall developer experience

Short-term Actions (This Month)

1. Add compilation performance metrics - Priority: Medium

   • Track performance trends as workflow count grows
   • Identify optimization opportunities
   • Establish performance baselines

2. Build workflow complexity analyzer - Priority: Medium

   • Prevent workflows from becoming unmaintainable
   • Identify refactoring opportunities early
   • Maintain code quality standards

Long-term Actions (This Quarter)

1. Create compilation health dashboard - Priority: Low

   • Visual overview of compilation health across all workflows
   • Trend analysis over time
   • Integration with repository insights

2. Implement automated workflow optimization - Priority: Low

   • Auto-suggest shared components for duplicate patterns
   • Auto-refactor simple complexity issues
   • Generate workflow templates from best practices

3. Build workflow testing framework - Priority: Low

   • Unit test individual workflow logic
   • Integration test workflow interactions
   • Regression test workflow behavior

---

📈 Success Metrics

Track these metrics to measure improvement in Workflow Compilation Health & Synchronization:

• Compilation Synchronization: 0 stale workflows → Target: Maintain 0
• Health Check Adoption: 0% → Target: 80% of developers use --health weekly
• Pre-commit Hook Usage: 0% → Target: 100% of active contributors
• Compilation Error Resolution Time: Unknown → Target: <5 minutes average
• Workflow Complexity Score: Unknown → Target: 85/100 repository average
• Developer Satisfaction: Unknown → Target: Survey developers quarterly

Secondary Metrics:

• Number of compilation-related bugs/issues → Target: <2 per month
• Average compilation time per workflow → Target: <25ms
• Strict mode adoption rate → Target: 60% (from current 25%)
• MCP server adoption rate → Target: 15% (from current 1%)

---

Next Steps

1. Review and prioritize the 5 tasks above based on immediate developer pain points
2. Assign tasks to Copilot agent via planner agent workflow
3. Track progress on improvement items in dedicated project board
4. Re-evaluate this focus area in 30 days to measure impact
5. Next quality run on 2025-11-17 will select a different focus area using the diversity algorithm (60% custom, 30% standard, 10% reuse)

---

Generated by Repository Quality Improvement Agent
Next analysis: 2025-11-17 - Focus area will be selected based on diversity algorithm to maintain 60%+ custom areas

AI generated by Repository Quality Improvement Agent