[Schema Consistency] 🔍 Schema Consistency Check - November 20, 2025

[github-actions[bot]]

🔍 Schema Consistency Check - November 20, 2025

This analysis validates consistency between the JSON schema, parser implementation, documentation, and real workflows using Strategy-003: Required Field Enforcement + Error Message Documentation.

Executive Summary

Strategy Used: Required Field Enforcement + Error Message Documentation (last used: 2025-11-12)
Inconsistencies Found: 1 critical (persistent issue)
New Findings: None
Effectiveness: Medium (no new issues discovered)

The analysis confirms that previously identified issues persist, but no new critical inconsistencies have been introduced. The codebase maintains excellent error documentation and proper nested field validation.

Full Analysis Details

Critical Issues

1. Required Field "on" Not Enforced in Compiler

Location: pkg/parser/schemas/main_workflow_schema.json:4 vs pkg/workflow/compiler.go

Issue: The schema declares "on" as a required field, but the compiler does not enforce this requirement. Instead, it auto-generates default trigger configurations when "on" is missing.

Evidence:

• Schema: "required": ["on"] (line 4 in main_workflow_schema.json)
• Compiler: applyDefaults() function generates default triggers when data.On == ""
• Behavior: Workflows compile successfully without "on" field

Impact:

• External validators (IDEs, linters) that use the JSON schema will incorrectly flag valid workflows as errors
• Schema doesn't accurately reflect actual compiler behavior
• Users may be confused about whether "on" is truly required

Recommendation: Either:

1. Remove "on" from required array and document auto-generation in description
2. Add compiler validation to enforce "on" requirement (breaking change)

Status: ⚠️ PERSISTENT (confirmed in multiple analyses since 2025-11-12)

---

Positive Findings

1. Excellent Error Message Documentation ✅

Location: docs/src/content/docs/troubleshooting/errors.md

The error documentation is comprehensive and well-maintained:

• Schema validation errors fully documented
• Frontmatter parsing errors with examples
• Compilation errors with solutions
• Runtime errors with troubleshooting steps
• "Did you mean" typo suggestions documented

Quality: EXEMPLARY - serves as a model for error documentation

2. Proper Nested Required Field Validation ✅

All conditionally required fields are properly validated:

• Cache configurations: key and path required together
• Schedule triggers: cron field required
• Safe outputs: Configuration-specific required fields enforced
• MCP servers: Type-specific requirements validated
• Steps: Either uses or run required (oneOf pattern)

Evidence: 20+ required field constraints across nested objects all properly enforced

3. Consistent Default Value Handling ✅

The schema uses two approaches for default values:

Explicit default fields (13 total):

• engine: "copilot"
• reaction: "eyes"
• Safe-outputs max values (1, 1024, 10240, etc.)
• threat-detection.enabled: true
• Safe-jobs input defaults

Description-documented defaults (10+ fields):

• name: defaults to filename without extension
• command: defaults to workflow filename
• concurrency: defaults to single job per engine
• github-token: defaults to GH_AW_GITHUB_TOKEN
• Various timeouts with documented defaults

Pattern: Core feature defaults use explicit default field, behavioral defaults documented in descriptions

---

Schema Required Fields Audit

Top-Level Required Fields

• Main schema: on only (enforcement issue noted above)
• MCP config schema: none
• Included file schema: none

Nested Required Fields (all properly enforced)

Context	Required Fields	Validation Location
Schedule trigger	cron	parseOnSection()
Cache configuration	key, path	cache.go validation
Step definition	uses OR run (oneOf)	Schema oneOf constraint
Concurrency object	group	Schema validation
Environment object	name	Schema validation
Container	image	Schema validation
Safe-outputs configs	Varies by type	Individual safe_*.go files

Total oneOf patterns: 80 (all with proper mutual exclusion)

---

Default Values Analysis

Explicitly Defined Defaults (13 total)

{
  "engine": "copilot",
  "reaction": "eyes",
  "safe-outputs.create-issue.max": 1,
  "safe-outputs.create-agent-task.max": 1,
  "safe-outputs.create-discussion.max": 1,
  "safe-outputs.add-comment.max": 1,
  "safe-outputs.create-pull-request-review-comment.max": 1,
  "safe-outputs.add-labels.max": 3,
  "safe-outputs.assign-milestone.max": 1,
  "safe-outputs.create-pull-request.max-patch-size": 1024,
  "safe-outputs.create-code-scanning-alert.max-size": 10240,
  "safe-outputs.noop.max": 1,
  "safe-outputs.threat-detection.enabled": true
}

Description-Documented Defaults (10+ fields)

Examples:

• name: "defaults to the filename without extension"
• timeout: "Default varies by engine (Claude: 60s, Codex: 120s)"
• startup-timeout: "Default: 120 seconds"
• cancel-in-progress: "Default: false"
• create-pull-request.draft: "defaults to true"
• push-to-pull-request-branch.branch: "defaults to 'triggering'"

Pattern Observation: Explicit default fields used for numeric limits and feature flags, while description-documented defaults used for complex/dynamic behavior.

---

Comparison to Previous Run (2025-11-12)

Finding	2025-11-12	2025-11-20	Change
"on" field enforcement gap	⚠️ Critical	⚠️ Critical	No change
Error documentation quality	✅ Excellent	✅ Excellent	No change
Nested required field validation	✅ Proper	✅ Proper	No change
Default value consistency	✅ Good	✅ Good	No change
New critical issues	None	None	No new issues

Conclusion: The codebase has not introduced new inconsistencies. The single critical issue (on field enforcement) remains unresolved but is well-understood.

---

Recommendations

Priority 1: Resolve "on" Field Requirement Ambiguity

Option A (Preferred - Non-Breaking):

{
  "required": [],
  "properties": {
    "on": {
      "description": "Workflow triggers... If not specified, defaults are auto-generated based on workflow configuration (command triggers get issue_comment/pull_request, others get workflow_dispatch)."
    }
  }
}

Option B (Breaking Change):
Add validation in ParseWorkflowFile() before calling parseOnSection():

if _, exists := frontmatter["on"]; !exists {
    return nil, fmt.Errorf("'on' field is required in workflow frontmatter")
}

Priority 2: Formalize Dynamic Defaults

Consider adding $comment fields to document dynamic defaults that can't be represented as static default values:

{
  "timeout": {
    "type": "integer",
    "$comment": "Runtime default: 60s (Claude), 120s (Codex), 180s (Custom)"
  }
}

Priority 3: Maintain Excellence

Continue current practices:

• ✅ Keep error documentation comprehensive and up-to-date
• ✅ Maintain proper oneOf patterns for mutual exclusion
• ✅ Continue enforcing nested required fields
• ✅ Document behavioral defaults in descriptions

---

Strategy Performance

Strategy-003 Effectiveness: MEDIUM for this run

• Findings: 1 critical (known), 0 new, 4 positive
• Thoroughness: Covered all 4 analysis areas
• Value: Confirms no regression, but no new discoveries
• Recommendation: Use every 6-8 weeks for regression checks

Next Strategy Recommendation:

• Use Strategy-005 (Conditional Logic Analysis) or Strategy-017 (Type Behavior) for different perspective
• Strategy-003 has reached diminishing returns on this codebase

---

Validation Scope

Areas Analyzed

✅ Schema required field definitions
✅ Compiler enforcement of required fields
✅ Default value declarations (explicit and documented)
✅ Error message documentation coverage
✅ Nested required field validation
✅ Real workflow compliance

Files Examined

• pkg/parser/schemas/main_workflow_schema.json (3527 lines)
• pkg/parser/schemas/mcp_config_schema.json
• pkg/parser/schemas/included_file_schema.json
• pkg/workflow/compiler.go (required field enforcement)
• pkg/workflow/tools.go (applyDefaults function)
• pkg/workflow/*_validation.go (multiple validation files)
• docs/src/content/docs/troubleshooting/errors.md
• .github/workflows/*.md (all 78 production workflows)

Validation Results

• Workflows checked: 78 (all have "on" field despite schema issue)
• Required field violations: 0 in actual workflows
• Schema accuracy: 99% (1 known issue out of 100+ constraints)

---

Next Steps

1. Decision Required: Address "on" field requirement ambiguity (see Priority 1 recommendation)
2. Continue Monitoring: Run Strategy-003 again in 6-8 weeks to check for regressions
3. Explore Alternative Strategies: Use Strategy-005 or Strategy-017 for next analysis to discover different issue categories

---

Analysis Date: 2025-11-20
Strategy: 003 (Required Field Enforcement + Error Message Documentation)
Analyzer: Automated Schema Consistency Checker
Cache: /tmp/gh-aw/cache-memory/strategies.json

AI generated by Schema Consistency Checker