🎯 Repository Quality Improvement - Workflow Compilation Error Messages

[github-actions[bot]]

🎯 Repository Quality Improvement Report - Workflow Compilation Error Messages

Analysis Date: 2025-11-19
Focus Area: Workflow Compilation Error Messages and User Experience
Strategy Type: Custom
Custom Area: Yes - This focus area is specific to gh-aw's core purpose as a workflow compilation system. With 119 agentic workflow files and 268 error generation sites, error message quality directly impacts the developer experience when creating and debugging workflows.

Executive Summary

The gh-aw project includes a comprehensive Error Message Style Guide that defines a three-part template for validation errors: [what's wrong] + [what's expected] + [example of correct usage]. However, analysis reveals that only 8.2% (22 of 268) of error messages follow this guideline by including examples. This creates a significant gap between the established best practice and actual implementation.

The impact is particularly noticeable in the five largest validation files (totaling 1,368 lines of code) where users encounter the majority of compilation errors. Improving error message quality will reduce debugging time, decrease support burden, and enhance the developer experience for the 119+ workflows currently in the repository.

Full Analysis Report

Focus Area: Workflow Compilation Error Messages

Current State Assessment

The gh-aw codebase has established clear guidelines for error message quality through the Error Message Style Guide (.github/instructions/error-messages.instructions.md). The guide emphasizes three key components every error should include:

1. What's wrong - Clearly state the validation error
2. What's expected - Explain the valid format or values
3. How to fix it - Provide a concrete example of correct usage

Metrics Collected:

Metric	Value	Status
Total error generation sites	268	ℹ️
Errors with examples	22	⚠️
Errors without examples	246	❌
Example coverage	8.2%	❌
Validation files	43	ℹ️
Agentic workflows	119	✅
Compiled workflows	83	✅
Test files	319	✅
Console formatting usage	833	✅

Findings

Strengths

• Clear Style Guide: The repository has an excellent Error Message Style Guide with concrete examples of good and bad error messages
• Some Exemplary Implementations: Files like time_delta.go and manual_approval.go demonstrate high-quality error messages with examples
• Good Error Infrastructure: Uses fmt.Errorf appropriately with proper error wrapping via %w
• Console Formatting: 833 uses of console.Format* functions show strong commitment to user-facing message quality
• Test Coverage: 319 test files indicate comprehensive testing infrastructure that can validate error messages

Areas for Improvement

• Low Example Coverage (8.2%): 246 of 268 errors lack concrete examples, leaving users to guess valid formats
• Inconsistent Application: Style guide exists but isn't systematically applied across validation files
• High-Impact Files Need Attention: The five largest validation files (1,368 LOC total) are where users encounter most errors:
  • repository_features_validation.go (328 lines)
  • runtime_validation.go (283 lines)
  • agent_validation.go (273 lines)
  • mcp_config_validation.go (258 lines)
  • expression_validation.go (226 lines)
• Error Wrapping Overuse: 22 uses of errors.New instead of fmt.Errorf provide less contextual information

Detailed Analysis

Error Message Quality Patterns:

Good Examples (following the style guide):

// time_delta.go - Excellent error with multiple format examples
fmt.Errorf("invalid time delta format: +%s. Expected format like +25h, +3d, +1w, +1mo, +1d12h30m", deltaStr)

// manual_approval.go - Type error with concrete YAML example
fmt.Errorf("manual-approval value must be a string, got %T. Example: manual-approval: \"production\"", val)

// time_delta.go - Helpful error with conversion suggestion
fmt.Errorf("minute unit 'm' is not allowed for stop-after. Minimum unit is hours 'h'. Use +%dh instead of +%dm", (value+59)/60, value)

Areas Needing Improvement:

// repository_features_validation.go - Good explanation but missing example
fmt.Errorf("workflow uses safe-outputs.create-discussion but repository %s does not have discussions enabled. Enable discussions in repository settings or remove create-discussion from safe-outputs", repo)
// Could add: Example: safe-outputs:\n  create-issue:\n    # Use create-issue instead

// mcp_config_validation.go - Generic wrapper error
fmt.Errorf("tool '%s' has invalid MCP configuration: %w", toolName, err)
// Could add specific examples of valid MCP configurations

// action_pins.go - Pass-through error
fmt.Errorf("%s", errMsg)
// Could enhance with context about where the error occurred

Impact on User Experience:

When developers encounter compilation errors without examples, they must:

1. Search through documentation
2. Look at existing workflow files for patterns
3. Trial-and-error with different formats
4. Potentially create support requests

With 119 agentic workflow files in the repository, improving error messages could significantly reduce friction in the workflow development cycle.

---

🤖 Tasks for Copilot Agent

NOTE TO PLANNER AGENT: The following tasks are designed for GitHub Copilot agent execution. Please split these into individual work items for Claude to process.

Improvement Tasks

The following code regions and tasks should be processed by the Copilot agent. Each section is marked for easy identification by the planner agent.

---

Task 1: Enhance Error Messages in Top 5 Validation Files

Priority: High
Estimated Effort: Large
Focus Area: Workflow Compilation Error Messages

Description:
Systematically improve error messages in the five largest validation files to follow the Error Message Style Guide template: [what's wrong] + [what's expected] + [example]. These files contain the majority of user-facing compilation errors.

Acceptance Criteria:

• Each error in the five target files includes a concrete example when applicable
• Error messages follow the three-part template format
• Examples use realistic YAML/configuration syntax
• Error coverage increases from 8.2% to at least 40% overall
• All changes pass existing test suite

Code Region: pkg/workflow/*validation*.go

Review and enhance error messages in these high-impact validation files:
- pkg/workflow/repository_features_validation.go (328 lines)
- pkg/workflow/runtime_validation.go (283 lines)  
- pkg/workflow/agent_validation.go (273 lines)
- pkg/workflow/mcp_config_validation.go (258 lines)
- pkg/workflow/expression_validation.go (226 lines)

For each fmt.Errorf call that lacks an example:
1. Identify what valid input would look like
2. Add "Example:" followed by concrete YAML/configuration showing the correct format
3. Ensure the example is realistic and helpful
4. Follow the patterns from time_delta.go and manual_approval.go

Reference the Error Message Style Guide at .github/instructions/error-messages.instructions.md

Prioritize errors that users are most likely to encounter (required fields, format errors, type mismatches).

---

Task 2: Create Error Message Quality Tests

Priority: High
Estimated Effort: Medium
Focus Area: Workflow Compilation Error Messages

Description:
Add automated tests that verify error messages follow the style guide. This prevents regression and ensures new errors include examples.

Acceptance Criteria:

• Test validates error messages contain "Example:" when appropriate
• Test checks error messages follow the three-part format
• Test runs as part of standard test suite (make test-unit)
• Test provides clear failure messages when standards aren't met
• Test excludes wrapped errors (%w) which inherit context

Code Region: pkg/workflow/error_message_quality_test.go

Create a new test file pkg/workflow/error_message_quality_test.go that:

1. Scans all pkg/workflow/*.go files for fmt.Errorf calls
2. Extracts error message strings
3. Validates that messages follow quality standards:
   - Format/syntax errors include "Expected:" or "Valid:"
   - Type mismatches include "Example:"
   - Enum/choice errors list valid options
4. Excludes wrapped errors (those using %w) from example requirement
5. Reports specific line numbers for violations

The test should be similar in style to existing tests in the pkg/workflow directory.

Add a allowlist mechanism for legitimate exceptions (like wrapped errors).

Goal: Ensure all future error messages include examples automatically through CI checks.

---

Task 3: Standardize MCP Configuration Error Messages

Priority: High
Estimated Effort: Medium
Focus Area: Workflow Compilation Error Messages

Description:
MCP (Model Context Protocol) configuration is complex and error-prone. Standardize error messages in MCP-related files to provide clear examples of valid configurations.

Acceptance Criteria:

• All MCP configuration errors include concrete examples
• Examples show both "local" and "http" MCP server formats
• Tool configuration errors demonstrate correct syntax
• Network permission errors show valid domain patterns
• Errors distinguish between required and optional fields

Code Region: pkg/workflow/*mcp*.go

Focus on MCP-related validation files:
- pkg/workflow/mcp_config_validation.go
- pkg/workflow/mcp_config.go
- pkg/workflow/mcp_servers.go
- Any other files matching *mcp*.go pattern

Enhance error messages to include examples like:

For invalid MCP server config:
Example: mcp-servers:\n  my-server:\n    command: "npx"\n    args: ["`@my/server`"]

For HTTP MCP errors:
Example: mcp-servers:\n  http-server:\n    type: http\n    url: "(redacted)"

For tool allowlist errors:
Example: tools:\n  github:\n    allowed:\n      - get_repository\n      - create_issue

Reference similar patterns in the codebase and ensure consistency with the Error Message Style Guide.

---

Task 4: Improve Parser Error Messages

Priority: Medium
Estimated Effort: Medium
Focus Area: Workflow Compilation Error Messages

Description:
The parser package (pkg/parser) contains 103 error generation sites. Enhance these to provide better guidance when frontmatter parsing fails.

Acceptance Criteria:

• YAML parsing errors include examples of correct frontmatter structure
• Schema validation errors list valid field names
• Import resolution errors show correct import syntax
• Frontmatter extraction errors explain expected format
• Examples use realistic workflow configurations

Code Region: pkg/parser/*.go

Review and enhance error messages in pkg/parser directory:
- Focus on frontmatter parsing errors
- Focus on schema validation errors
- Focus on import/include resolution errors

For each error, add examples showing:
- Correct YAML frontmatter structure
- Valid field names and types
- Proper import syntax
- Common patterns from actual workflows in .github/workflows/

Example improvements:
- "Invalid frontmatter" → "Invalid frontmatter. Example:\n---\non: issues\nengine: copilot\n---"
- "Unknown field" → "Unknown field '%s'. Valid fields are: on, engine, tools, permissions. Example: engine: copilot"
- "Import failed" → "Import failed for '%s'. Example import syntax:\nimports:\n  - shared/tool-setup.md"

Ensure error messages help users quickly identify and fix frontmatter issues.

---

Task 5: Document Error Message Patterns in Developer Guide

Priority: Low
Estimated Effort: Small
Focus Area: Workflow Compilation Error Messages

Description:
Update DEVGUIDE.md to include a section on error message patterns with examples from the codebase. This helps contributors write quality error messages from the start.

Acceptance Criteria:

• DEVGUIDE.md includes "Error Message Guidelines" section
• Section references the Error Message Style Guide
• Includes 3-5 real examples from the codebase
• Shows both good and bad patterns
• Links to relevant test files
• Explains when examples are required vs optional

Code Region: DEVGUIDE.md

Add a new section to DEVGUIDE.md titled "Writing Quality Error Messages"

Include:
1. Link to .github/instructions/error-messages.instructions.md
2. The three-part template: [what's wrong] + [what's expected] + [example]
3. Real examples from time_delta.go (good) and areas needing improvement
4. When to include examples (always for format/enum/type errors)
5. When examples are optional (wrapped errors with %w)
6. How to test error message quality
7. Reference to error_message_quality_test.go (once created)

Make this section actionable and easy to follow for contributors.

Place this section under "Development Guidelines" or similar area in DEVGUIDE.md.

---

📊 Historical Context

Previous Focus Areas

Date	Focus Area	Type	Custom	Key Outcomes
2025-11-19	Workflow Compilation Error Messages	Custom	Y	First quality improvement run - established baseline for custom focus area diversity

---

🎯 Recommendations

Immediate Actions (This Week)

1. Task 1: Enhance Top 5 Validation Files - Priority: High - These files contain the majority of user-facing errors
2. Task 2: Create Error Message Quality Tests - Priority: High - Prevent regression and enforce standards
3. Task 3: Standardize MCP Configuration Errors - Priority: High - MCP is complex and needs clear examples

Short-term Actions (This Month)

1. Task 4: Improve Parser Error Messages - Priority: Medium - Enhance frontmatter parsing feedback
2. Complete error message audit - Review remaining files in pkg/workflow for error quality
3. Monitor error message coverage - Track improvement from 8.2% baseline

Long-term Actions (This Quarter)

1. Task 5: Document Error Patterns - Priority: Low - Establish contributor guidelines
2. Create error message linter - Consider static analysis tool to enforce standards
3. User feedback loop - Collect data on which errors users encounter most frequently

---

📈 Success Metrics

Track these metrics to measure improvement in Workflow Compilation Error Messages:

• Example Coverage: 8.2% → 40% (near-term), → 80% (long-term)
• High-Impact File Coverage: 0% → 100% (top 5 validation files include examples)
• User Support Requests: Track compilation-related questions in issues/discussions
• First-Time Workflow Success Rate: Measure how quickly new users create working workflows
• Error Message Length: Ensure messages remain concise while adding examples (target: <150 chars for simple errors)

---

Next Steps

1. Review and prioritize the tasks above
2. Assign tasks to Copilot agent via planner agent
3. Track progress on improvement items using the metrics above
4. Re-evaluate this focus area in 30 days to measure impact
5. Next analysis: 2025-11-20 - Focus area will be selected based on diversity algorithm (60% custom, 30% standard, 10% reuse)

---

Generated by Repository Quality Improvement Agent
Next scheduled analysis: 2025-11-20
Focus area selection strategy: 60% custom repository-specific areas, 30% standard categories, 10% reuse for consistency

AI generated by Repository Quality Improvement Agent