🎯 Repository Quality Improvement Report - Workflow Compilation Error Messages

[github-actions[bot]]

🎯 Repository Quality Improvement Report - Workflow Compilation Error Messages

Analysis Date: 2025-11-15
Focus Area: Workflow Compilation Error Messages and Developer Feedback
Strategy Type: Custom
Custom Area: Yes - This focus area was specifically chosen to address gh-aw's core user experience challenge: helping developers debug workflow compilation and validation issues efficiently.

Executive Summary

This analysis examines the quality, clarity, and actionability of error messages throughout the gh-aw workflow compilation and validation system. With 839 total error points across the codebase and only 17.28% adoption of the error message style guide (145 errors with examples), there's significant opportunity to improve the developer experience when workflows fail to compile or validate.

The repository has excellent infrastructure in place: a comprehensive error message style guide (.github/instructions/error-messages.instructions.md), console formatting utilities, and dedicated error message quality tests. However, the relatively low adoption rate indicates that many error messages could benefit from enhancement to include actionable examples and clearer guidance. Given that gh-aw is a developer-facing tool where compilation errors directly impact productivity, investing in error message quality will yield high returns in user satisfaction and reduced support burden.

Key findings show strong patterns in recently-updated files (like time_delta.go and mcp-config.go) following best practices, while older validation code may need modernization.

Full Analysis Report

Focus Area: Workflow Compilation Error Messages

Current State Assessment

Codebase Error Inventory:

• Total Error Points: 839 (across pkg/workflow/ and pkg/cli/)
• Errors with Examples: 145 (17.28% of total)
• Validation Files: 40 dedicated validation Go files
• Error Test Files: 9 dedicated error testing files
• Console Error Formatting Usage: 35 instances using console.FormatError* functions

Infrastructure Assets:

• ✅ Comprehensive error message style guide (.github/instructions/error-messages.instructions.md)
• ✅ Console formatting package with structured error display (pkg/console/)
• ✅ Dedicated error message quality tests (error_message_quality_test.go, error_validation_test.go)
• ✅ Clear three-part template: [what's wrong] + [what's expected] + [example]
• ❌ No centralized error message documentation for users
• ❌ No error code or categorization system

Metrics Collected:

Metric	Value	Status
Total Error Messages	839	ℹ️ Baseline
Error Messages with Examples	145	⚠️ 17.28% adoption
Validation Files	40	✅ Well-organized
Error Test Coverage	9 files	✅ Good testing
Console Formatting Usage	35	⚠️ Inconsistent usage
Error Documentation	0 pages	❌ Missing

Findings

Strengths

• Excellent Style Guide: The error message style guide is comprehensive, well-documented, and provides clear examples of good vs. bad error messages
• Best-in-Class Examples: Files like time_delta.go demonstrate exemplary error messages:

  fmt.Errorf("invalid time delta format: +%s. Expected format like +25h, +3d, +1w, +1mo, +1d12h30m", deltaStr)

• Strong Testing Culture: Dedicated error message quality tests ensure new errors follow guidelines
• Console Formatting Infrastructure: The console package provides structured error formatting with FormatError, FormatErrorMessage, and FormatErrorWithSuggestions
• Recent Improvements: MCP configuration errors show excellent multi-line YAML examples

Areas for Improvement

• Low Adoption Rate (17.28%): 82.72% of error messages lack actionable examples - High Impact
• Generic "failed to" Messages: Many errors use generic patterns like "failed to parse: %w" without context - Medium Impact
• Inconsistent Console Formatting: Only 35 uses of console formatters vs. 839 total errors - Medium Impact
• No Error Documentation: Users have no central reference for common errors and solutions - Medium Impact
• Large Validation Files: Files like compiler.go (1617 lines) and permissions.go (905 lines) may contain older error patterns - Low Impact

Detailed Analysis

Error Message Quality Distribution

Exemplary Files (>30% adoption):

• time_delta.go - Nearly all errors include examples and clear guidance
• mcp-config.go - Recent updates show multi-line YAML examples
• manual_approval.go - Type validation with examples

Files Needing Improvement:

• compiler.go - 23 errors without examples (core compilation logic)
• runtime_setup.go - Generic "failed to" messages
• repository_features_validation.go - Wrapped errors without actionable guidance
• Various validation files using minimal error context

Error Message Categories

Category 1: Format/Syntax Errors (Good Coverage)

• Time delta parsing ✅
• Date format validation ✅
• Repository format validation ✅

Category 2: Type Validation (Moderate Coverage)

• Manual approval types ✅
• MCP configuration types ✅
• Generic type mismatches ⚠️

Category 3: Configuration Errors (Mixed Coverage)

• MCP server configuration ✅
• Permission configuration ⚠️
• Tool configuration ⚠️

Category 4: Wrapped Errors (Needs Improvement)

• Runtime setup errors ❌
• Repository validation errors ❌
• Compiler job errors ❌

Console Formatting Adoption

Current Usage:

• console.FormatError() - Used for structured compiler errors
• console.FormatErrorMessage() - Used for simple error display
• console.FormatErrorWithSuggestions() - Used for actionable errors with hints

Gap Analysis:

• Most errors in pkg/workflow/ use plain fmt.Errorf without console formatting
• CLI commands in pkg/cli/ show better console formatting adoption
• Opportunity to standardize error output through console package

---

🤖 Tasks for Copilot Agent

NOTE TO PLANNER AGENT: The following tasks are designed for GitHub Copilot agent execution. Please split these into individual work items for Claude to process.

Improvement Tasks

The following code regions and tasks should be processed by the Copilot agent. Each section is marked for easy identification by the planner agent.

---

Task 1: Enhance Permission Validation Error Messages

Priority: High
Estimated Effort: Medium
Focus Area: Configuration Validation

Description:
Improve error messages in pkg/workflow/permissions.go to follow the three-part template: what's wrong + what's expected + example. The file is 905 lines and handles critical permission validation logic. Currently, many errors don't provide examples or clear guidance on valid permission configurations.

Acceptance Criteria:

• All permission-related errors include examples showing valid YAML syntax
• Type mismatch errors show both received type and expected type
• Enum validation errors list all valid permission levels
• Invalid permission scope errors include example configurations
• Tests updated to verify error message content

Code Region: pkg/workflow/permissions.go

Review all error messages in `pkg/workflow/permissions.go` and update them to follow the error message style guide. Focus on:

1. Permission level validation - show valid options (read, write, none)
2. Permission scope errors - provide example YAML configurations
3. Type validation errors - show expected types with examples
4. Missing required permission errors - include complete examples

Follow the pattern from `.github/instructions/error-messages.instructions.md`:
- Format: [what's wrong]. [what's expected]. [example of correct usage]
- Use %T for type display in type mismatch errors
- Include multi-line YAML examples for configuration errors
- Reference the style guide for template patterns

Example transformation:
- Before: `fmt.Errorf("invalid permission level: %s", level)`
- After: `fmt.Errorf("invalid permission level: %s. Valid levels: read, write, none. Example: permissions:\n  contents: read", level)`

---

Task 2: Add Examples to Compiler Error Messages

Priority: High
Estimated Effort: Large
Focus Area: Core Compilation Logic

Description:
The pkg/workflow/compiler.go file (1617 lines) contains 23 error messages without examples. This is a critical file for the user experience as compilation errors are frequently encountered. Enhance these errors to provide actionable guidance.

Acceptance Criteria:

• All compilation errors include at least one example of correct usage
• Schema validation errors show valid field names and types
• Workflow structure errors include example frontmatter
• Import resolution errors suggest valid import paths
• Error messages use console formatting functions where appropriate

Code Region: pkg/workflow/compiler.go

Improve error messages in `pkg/workflow/compiler.go` to include actionable examples. Focus on:

1. Schema validation errors - show valid field names and example configurations
2. Frontmatter parsing errors - provide example YAML frontmatter
3. Import resolution errors - suggest valid import paths and formats
4. Workflow structure validation - include complete workflow examples
5. Tool configuration errors - show valid tool setups

Use console formatting functions for consistency:
- `console.FormatError()` for structured compiler errors
- `console.FormatErrorWithSuggestions()` for errors with actionable hints

Apply the three-part error message template throughout.

---

Task 3: Standardize Runtime Setup Error Messages

Priority: Medium
Estimated Effort: Small
Focus Area: Error Context and Wrapping

Description:
The pkg/workflow/runtime_setup.go file contains several generic "failed to" error messages that wrap underlying errors without adding sufficient context. Enhance these to provide better debugging information.

Acceptance Criteria:

• Wrapped errors include context about what operation was being attempted
• Error messages explain the impact of the failure
• Suggestions for resolution are provided where applicable
• Error messages reference relevant documentation when complex

Code Region: pkg/workflow/runtime_setup.go

Update error messages in `pkg/workflow/runtime_setup.go` to provide better context:

1. "failed to parse custom steps" → Explain what custom steps are and show example
2. "failed to marshal deduplicated steps" → Explain deduplication and its purpose
3. Add context about which part of runtime setup failed
4. Provide examples of valid runtime configurations

Even for wrapped errors, add meaningful context:
- Before: `fmt.Errorf("failed to parse: %w", err)`
- After: `fmt.Errorf("failed to parse custom workflow steps from frontmatter. Custom steps must be valid GitHub Actions step syntax. Example:\nsteps:\n  - name: Setup\n    run: echo 'hello'\nError: %w", err)`

---

Task 4: Create Error Reference Documentation

Priority: Medium
Estimated Effort: Medium
Focus Area: User-Facing Documentation

Description:
Create a comprehensive error reference page in the documentation (docs/src/content/docs/reference/) that catalogs common workflow compilation and validation errors with solutions.

Acceptance Criteria:

• Documentation page created at docs/src/content/docs/reference/errors.md
• Common errors organized by category (syntax, validation, configuration, etc.)
• Each error includes: description, cause, example, and solution
• Cross-references to relevant documentation sections
• Search-friendly error codes or titles

Code Region: docs/src/content/docs/reference/errors.md (new file)

Create a new documentation file at `docs/src/content/docs/reference/errors.md` following the Astro Starlight format and Diátaxis framework (Reference category).

Structure the documentation as follows:

---
title: Error Reference
description: Common workflow compilation and validation errors
---

# Error Reference

Comprehensive reference for gh-aw workflow compilation and validation errors.

## Frontmatter Errors

### Invalid Time Delta Format
**Error**: `invalid time delta format: +[value]. Expected format like +25h, +3d, +1w, +1mo, +1d12h30m`

**Cause**: Time delta in `stop-after` field uses invalid format or units.

**Example**:
```yaml
on:
  issues:
    stop-after: +2d  # Valid: 2 days

Solution: Use combinations of h (hours), d (days), w (weeks), mo (months). Minimum unit is hours.

[Continue for 15-20 most common errors...]

MCP Configuration Errors

[...]

Permission Errors

[...]

Validation Errors

[...]

Use tables, code examples, and cross-links to related documentation. Follow the Diátaxis reference documentation style: organized by structure, comprehensive, authoritative.

---

#### Task 5: Implement Console Formatting Standardization

**Priority**: Low  
**Estimated Effort**: Medium  
**Focus Area**: Error Display Consistency

**Description:**
Standardize error output across the codebase by converting plain `fmt.Errorf` calls to use console formatting functions. This improves consistency in error display and makes errors more visually distinctive in terminal output.

**Acceptance Criteria:**
- [ ] CLI command errors use `console.FormatErrorMessage()` consistently
- [ ] Compiler errors use `console.FormatError()` with structured error types
- [ ] Validation errors with suggestions use `console.FormatErrorWithSuggestions()`
- [ ] Plain error strings converted to formatted output in user-facing code
- [ ] Tests updated to check for formatted error output

**Code Region:** `pkg/cli/*.go` and `pkg/workflow/*.go`

```markdown
Audit and standardize error formatting across the codebase:

1. **Identify User-Facing Errors**: Focus on CLI commands and top-level compiler functions
2. **Apply Console Formatting**:
   - Simple errors → `console.FormatErrorMessage(err.Error())`
   - Compiler errors → `console.FormatError(console.CompilerError{...})`
   - Errors with hints → `console.FormatErrorWithSuggestions(message, suggestions)`

3. **Priority Files**:
   - `pkg/cli/compile_command.go`
   - `pkg/cli/logs.go`
   - `pkg/workflow/compiler.go` (user-facing compilation errors)

4. **Pattern**:
   ```go
   // Before
   fmt.Fprintf(os.Stderr, "Error: %v\n", err)
   
   // After
   fmt.Fprintln(os.Stderr, console.FormatErrorMessage(err.Error()))

Leave internal/wrapped errors unchanged. Focus on errors displayed directly to users.

---

## 📊 Historical Context

<details>
<summary><b>Previous Focus Areas</b></summary>

| Date | Focus Area | Type | Custom | Key Outcomes |
|------|------------|------|--------|--------------|
| 2025-11-15 | Workflow Compilation Error Messages | Custom | Y | First analysis - identified 17.28% adoption rate of error message guidelines |

</details>

---

## 🎯 Recommendations

### Immediate Actions (This Week)

1. **Enhance High-Impact Error Messages** - Focus on `permissions.go` and `compiler.go` (Task 1 & 2) - Priority: High
   - These files handle the most common user-facing errors
   - Improving these will have immediate impact on developer experience

2. **Create Error Reference Documentation** - Add `docs/reference/errors.md` (Task 4) - Priority: High
   - Provides users with self-service error resolution
   - Reduces support burden and improves onboarding

### Short-term Actions (This Month)

1. **Standardize Runtime Errors** - Update `runtime_setup.go` (Task 3) - Priority: Medium
   - Improves debugging experience for advanced users
   - Completes coverage of critical compilation path

2. **Implement Console Formatting** - Apply consistent formatting (Task 5) - Priority: Medium
   - Visual consistency improves professional appearance
   - Makes errors easier to spot in terminal output

### Long-term Actions (This Quarter)

1. **Error Message Linting** - Create automated checks for error message quality
   - Prevent regression on error message standards
   - Enforce style guide during PR reviews

2. **Error Categorization System** - Implement error codes (e.g., E001, E002)
   - Makes errors searchable and documentable
   - Enables better error tracking and analytics

3. **Interactive Error Helper** - Add `gh aw explain (error-code)` command
   - Provides context-sensitive help for errors
   - Further reduces support burden

---

## 📈 Success Metrics

Track these metrics to measure improvement in **Workflow Compilation Error Messages**:

- **Error Message Adoption Rate**: 17.28% → 60%+ (target: all new errors, 50%+ of existing errors)
- **Console Formatting Usage**: 35 instances → 150+ instances (all user-facing errors)
- **Error Documentation Pages**: 0 → 1 comprehensive reference page
- **User Support Issues Related to Errors**: Baseline → -30% (track via GitHub issues)
- **Error Message Test Coverage**: 9 test files → 15+ test files (ensure quality maintenance)

---

## Next Steps

1. Review and prioritize the tasks above
2. Assign tasks to Copilot agent via planner agent
3. Track progress on improvement items
4. Re-evaluate this focus area in 3 months to measure adoption and impact
5. Next daily analysis (2025-11-16) will focus on a different area per diversity algorithm

---

*Generated by Repository Quality Improvement Agent*  
*Next analysis: 2025-11-16 - Focus area will be selected based on diversity algorithm (60% custom, 30% standard, 10% reuse)*


> AI generated by [Repository Quality Improvement Agent](https://github.com/githubnext/gh-aw/actions/runs/19395189205)

[pelikhan]

/plan

[github-actions[bot]]

✅ Agentic Plan Command completed successfully.