refactor(action.yml, README.md, workflows, scripts): transition from …

[jbiskur]

…Gemini to ForgeCode integration

• Updated action.yml to replace Gemini model references with ForgeCode and added support for multiple LLM providers (OpenRouter, OpenAI, Anthropic).
• Modified README.md to reflect the new AI provider options and updated features.
• Introduced new scripts for provider setup and validation, ensuring compatibility with the ForgeCode CLI.
• Enhanced workflows to utilize the new provider and model inputs, streamlining the validation process.
• Added local testing script for easier validation before pushing changes to GitHub Actions.

[github-actions]

🤖 Documentation Validation

PR Validation Report

Summary

Documentation refactoring from Gemini to ForgeCode integration reveals several incomplete updates. The README and action.yml files contain outdated references to removed inputs and deprecated secrets, creating inconsistency between documented and actual implementation.

Override Applied 🔄

A deviation from standards has been approved and documented:

• Deviation: N/A
• Justification: N/A

This deviation has been recorded in the knowledge base for future reference.

Critical Violations ❌

• File: action.yml:355
  • Issue: Reference to undefined input 'mcp-server-url' in PR comment template
  • Why: The 'mcp-server-url' input was removed from the action inputs (lines 30-80) but is still referenced in the PR comment generation step. This will cause the comment to display an empty value or error.
  • Fix: Replace '${{ inputs.mcp-server-url }}' with a hardcoded value like 'https://usable.dev/api/mcp' or remove the line from the comment template since it's not a configurable input anymore.
  • Code:

  '- **Standards Source**: ${{ inputs.mcp-server-url }}\n' +
  

Important Issues ⚠️

• File: README.md:405-406

  • Issue: Release-Please integration example uses deprecated GEMINI_SERVICE_ACCOUNT_KEY secret
  • Why: The example in the Release-Please Integration section still references the old Gemini authentication method which has been replaced with LLM provider keys (OPENROUTER_API_KEY, ANTHROPIC_API_KEY, OPENAI_API_KEY).
  • Fix: Update the example to use 'OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }}' or show multiple provider options, consistent with other examples in the document.
  • Code:

  GEMINI_SERVICE_ACCOUNT_KEY: ${{ secrets.GEMINI_SERVICE_ACCOUNT_KEY }}
  

• File: README.md:486,497

  • Issue: Usable Integration Setup section references deprecated secret and authentication method
  • Why: The 'Add GitHub Secrets' example shows 'GEMINI_SERVICE_ACCOUNT_KEY=base64_encoded_key_here' and the workflow configuration example uses the old secret name, which are no longer valid in the refactored action.
  • Fix: Update both examples to show modern API key setup (OPENROUTER_API_KEY, ANTHROPIC_API_KEY, or OPENAI_API_KEY) and remove the base64 encoding instructions which are specific to Gemini service accounts.
  • Code:

  GEMINI_SERVICE_ACCOUNT_KEY=base64_encoded_key_here
  

• File: README.md:567,571

  • Issue: Comment-triggered revalidation example has old gemini-model input and secret references
  • Why: The example shows both a commented-out 'gemini-model' input that no longer exists (replaced with 'provider' and 'model') and uses the deprecated 'GEMINI_SERVICE_ACCOUNT_KEY' secret.
  • Fix: Update the example to comment out the new 'model' input instead and change the secrets section to use 'OPENROUTER_API_KEY' or show all three provider options.
  • Code:

  # gemini-model: 'gemini-2.5-flash'
  GEMINI_SERVICE_ACCOUNT_KEY: ${{ secrets.GEMINI_SERVICE_ACCOUNT_KEY }}
  

• File: README.md:501

  • Issue: Documentation references non-existent 'mcp-server-url' input as customizable
  • Why: The note claims users can 'customize the server URL with the mcp-server-url input' but this input has been removed from action.yml. The MCP URL is now hardcoded or configurable via environment variables only.
  • Fix: Remove the phrase about customizing mcp-server-url input, or clarify that it can only be customized via environment variables (USABLE_URL) if needed.
  • Code:

  You can customize the server URL with the `mcp-server-url` input if needed.
  

• File: README.md:695

  • Issue: Act test command references deprecated GEMINI_SERVICE_ACCOUNT_KEY
  • Why: The development setup example for running local tests with 'act' uses the old Gemini service account key instead of modern API keys.
  • Fix: Update to use OPENROUTER_API_KEY, ANTHROPIC_API_KEY, or OPENAI_API_KEY depending on the provider being tested.
  • Code:

  act pull_request -s GEMINI_SERVICE_ACCOUNT_KEY="$(cat key.json | base64)"
  

• File: README.md:701-730

  • Issue: Troubleshooting section contains references to Gemini CLI and Google Gemini
  • Why: The 'No Output from Gemini CLI' troubleshooting section references Google Gemini, Gemini CLI, and specific Gemini models that are no longer used. This will confuse users using ForgeCode with other providers.
  • Fix: Rename section to 'No Output from ForgeCode CLI' or 'No Validation Output', and update the log examples and error descriptions to be provider-agnostic. Reference ForgeCode CLI instead of Gemini CLI.
  • Code:

  ### No Output from Gemini CLI
  **Symptom**: GitHub Action runs but shows no Gemini output or errors
  - Gemini CLI failing silently
  

• File: README.md:848,882

  • Issue: Additional troubleshooting sections contain outdated references to Gemini
  • Why: The 'Validation Fails Immediately' section mentions Gemini as the potential cause, and the 'API Rate Limit Errors' section references 'Gemini API' specifically.
  • Fix: Update to reference the LLM provider being used (ForgeCode, OpenRouter, etc.) in a provider-agnostic way.
  • Code:

  - Prompt file not found
  - Missing required secrets
  - Invalid MCP configuration
  
  **Solution**: Check prompt file exists... Verify secrets are set... To Google Cloud quotas
  

• File: README.md:910-980

  • Issue: Pricing section shows only Google Gemini pricing, missing ForgeCode provider information
  • Why: The pricing section and acknowledgments focus exclusively on 'Google Gemini' but the action now supports multiple providers (OpenRouter, Anthropic, OpenAI). This is misleading documentation.
  • Fix: Either update the pricing section to show costs for multiple providers, or create separate pricing subsections for each supported provider. Update acknowledgments to reflect the new multi-provider architecture.
  • Code:

  ### Google Gemini Pricing (Vertex AI)
  | gemini-2.5-flash | $0.075 | $0.30 | $0.01-0.03 |
  - Google Gemini for AI capabilities
  

Suggestions ℹ️

• Add provider setup examples in prerequisites: The Prerequisites section (lines 27-34) could be enhanced with brief setup links for each provider (OpenRouter, Anthropic, OpenAI) similar to how they are shown in the main Quick Start. This would help users understand what they need before diving into the detailed examples.

  • File: README.md:27-34

• Create a migration guide for existing users: Since this is a major refactoring from Gemini to multi-provider support, consider adding a 'Migration Guide' section that clearly explains what changed and how existing users should update their workflows. This would help prevent confusion.

  • File: README.md

• Add cost comparison table for all providers: Replace or expand the current 'Cost Estimation' section to include pricing for multiple providers (OpenRouter, Anthropic, OpenAI) so users can make informed decisions about which provider to use.

  • File: README.md:908-920

• Update acknowledgments section: The acknowledgments section (line 980) still mentions 'Google Gemini for AI capabilities' which is no longer accurate. Consider updating it to acknowledge the multi-provider architecture and the libraries/services it now uses.

  • File: README.md:980

• Add provider-specific troubleshooting section: Consider adding provider-specific troubleshooting for each LLM provider (OpenRouter, Anthropic, OpenAI) with their common errors and solutions, rather than Gemini-specific guidance.

  • File: README.md:699-906

Validation Outcome

• Status: FAIL ❌
• Critical Issues: 1
• Important Issues: 7
• Suggestions: 5

Rationale: The PR contains incomplete migration from Gemini to ForgeCode. While action.yml was properly updated with new inputs, documentation still references removed inputs and deprecated secrets in multiple examples. The critical issue in action.yml:355 will cause runtime failures when posting PR comments. The 7 important issues create significant confusion for users following the README examples.

• Triggered by: @jbiskur

Standards Checked

• Documentation consistency with action.yml inputs
• Complete replacement of deprecated references
• Provider-agnostic language where applicable
• Example code accuracy and executability
• Secret naming conventions consistency

---

📊 Validation Statistics

• Provider: auto
• Model: anthropic/claude-haiku-4.5
• Standards Source:
• Commit: 053ee65
• Triggered by: @jbiskur

[github-actions]

🤖 Integration Test Validation

PR Validation Report

Summary

This PR (#12) refactors the MCP server configuration mechanism in the usable-pr-validator GitHub Action, transitioning from ForgeCode CLI commands to a direct .mcp.json file configuration approach. The changes improve reliability and security by eliminating unnecessary CLI calls and ensuring proper cleanup of sensitive authentication data. All modifications follow established shell scripting standards and maintain the expected behavior.

Override Applied 🔄

A deviation from standards has been approved and documented:

• Deviation: N/A
• Justification: N/A

This deviation has been recorded in the knowledge base for future reference.

Critical Violations ❌

None found.

Important Issues ⚠️

• File: scripts/setup-mcp.sh:25-35

  • Issue: Sensitive authentication token embedded in .mcp.json file via heredoc without explicit content validation
  • Why: While the token is only in memory/temp files that are cleaned up, embedding secrets in configuration files (even temporarily) creates a window of vulnerability. The variable $USABLE_API_TOKEN should be validated before embedding to prevent injection attacks.
  • Fix: Add validation before embedding: Check that $USABLE_API_TOKEN contains only valid characters (base64-compatible) before embedding in JSON. Consider using printf-style escaping to safely handle special characters in the token value.
  • Code:

  # Add validation before EOF
  if ! [[ "$USABLE_API_TOKEN" =~ ^[A-Za-z0-9._-]+$ ]]; then
    echo "::error::Invalid USABLE_API_TOKEN format"
    exit 1
  fi
  

• File: scripts/setup-mcp.sh:42-45

  • Issue: Error message in cleanup uses non-standard error handler format
  • Why: The script uses set -euo pipefail but the error handling at line 47 uses '::error::' (GitHub Actions format) which is appropriate, but the verification logic could be more robust. The file check happens but doesn't provide detailed error context.
  • Fix: Add more context to the error message and consider checking directory permissions: if [ ! -f ".mcp.json" ] || [ ! -r ".mcp.json" ]; then echo "::error::Failed to create or read .mcp.json"; exit 1; fi
  • Code:

  if [ ! -f ".mcp.json" ] || [ ! -r ".mcp.json" ]; then
    echo "::error::Failed to create or read .mcp.json"
    exit 1
  fi
  

Suggestions ℹ️

• Add file permissions check for .mcp.json: The .mcp.json file contains authentication tokens. Consider explicitly setting restrictive permissions (600) to ensure only the current user can read it, even though it's cleaned up after use.

  • File: scripts/setup-mcp.sh

• Document the .mcp.json configuration format: Add a comment in setup-mcp.sh explaining the mcpServers JSON schema being used, as this is a non-standard format specific to ForgeCode. This helps future maintainers understand the structure.

  • File: scripts/setup-mcp.sh

• Add verification that .mcp.json is valid JSON: After creating .mcp.json, validate it's valid JSON using jq before proceeding. This catches syntax errors early: if ! jq empty .mcp.json 2>/dev/null; then echo "::error::Invalid JSON in .mcp.json"; exit 1; fi

  • File: scripts/setup-mcp.sh

• Ensure cleanup order in action.yml: The cleanup step in action.yml (line 413) removes .mcp.json after validation completes. This is correct, but consider documenting that this must happen before any file uploads/artifacts to prevent secrets leakage.

  • File: action.yml

• Add logging for cleanup success: The cleanup step could log which files were actually removed (rm -v) to provide better visibility in logs: rm -fv /tmp/forge-config.yaml /tmp/validation-*.md .mcp.json

  • File: action.yml:413

Validation Outcome

• Status: PASS ✅
• Critical Issues: 0
• Important Issues: 2
• Suggestions: 5

Rationale: The PR successfully refactors the MCP server setup mechanism with correct implementation of the new .mcp.json configuration approach. The changes eliminate unnecessary CLI calls and improve reliability. While there are two important issues related to token validation and error handling robustness, these do not prevent the PR from being merged. The changes maintain backward compatibility, follow shell scripting best practices with set -euo pipefail, and include proper cleanup mechanisms for sensitive files.

• Triggered by: PR refactor(action.yml, README.md, workflows, scripts): transition from … #12: refactor(action.yml, README.md, workflows, scripts): transition from Gemini to ForgeCode integration

Standards Checked

• Shell script best practices (set -euo pipefail usage)
• Security: sensitive data handling and cleanup
• GitHub Actions action.yml format and structure
• Error handling in bash scripts
• Temporary file management
• Configuration file validation
• YAML syntax correctness

---

📊 Validation Statistics

• Provider: auto
• Model: anthropic/claude-haiku-4.5
• Standards Source:
• Commit: 053ee65
• Triggered by: @jbiskur