feat(action): Add reusable GitHub action

README.md

@@ -62,6 +62,7 @@ ai-sdks-test/
 │       └── test-results/          # Generated test reports
 ├── .env                           # Environment variables (gitignored)
 ├── .env.example                   # Template for API keys
+├── action.yml                     # GitHub Action to run the tests for a specific language on CI
 └── package.json                   # Root package.json for CLI alias
 ```
 
@@ -182,6 +183,53 @@ npm run cli run lang -- --case 1-simple         # Run 1-simple on lang* SDKs
 npm run cli list
 ```
 
+## Using as a GitHub Action
+
+This repository can be used as a reusable GitHub Action in SDK repositories (e.g., `sentry-javascript`, `sentry-python`) to run AI integration tests on a schedule.
+
+### Setup in SDK Repositories
+
+1. **Create a workflow** in your SDK repo (e.g., `.github/workflows/ai-integration-tests.yml`):
+
+```yaml
+name: AI Integration Tests
+
+on:
+  schedule:
+    - cron: "0 9 * * 1" # Weekly on Monday at 9am UTC
+  workflow_dispatch: # Allow manual trigger
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout SDK repository
+        uses: actions/checkout@v4
+
+      - name: Run AI Integration Tests
+        uses: getsentry/testing-ai-sdk-integrations@v1
+        with:
+          language: js # or 'python'
+          github-token: ${{ secrets.GITHUB_TOKEN }}
+          sentry-dsn: ${{ secrets.SENTRY_DSN }}
+          openai-api-key: ${{ secrets.OPENAI_API_KEY }}
+          anthropic-api-key: ${{ secrets.ANTHROPIC_API_KEY }}
+          google-api-key: ${{ secrets.GOOGLE_API_KEY }}
+```
+
+2. **Add secrets** to your SDK repository:
+   - `SENTRY_DSN` - Sentry project DSN
+   - `OPENAI_API_KEY` - OpenAI API key
+   - `ANTHROPIC_API_KEY` - Anthropic API key
+   - `GOOGLE_API_KEY` - Google API key for GenAI
+
+### How It Works
+
+- The action runs tests for the specified language (js or python)
+- On failure, it automatically creates or updates an issue in the **calling repository** (not this repo)
+- Issues are labeled with `ai-integration-test-failure` for easy tracking
+- Test results are included in the issue body in JSON format
+
 ## Test Scenarios
 
 Each SDK implementation includes these scenarios (where supported by the SDK):

action.yml

@@ -0,0 +1,146 @@
+name: 'Run AI SDK Integration Tests'
+description: 'Run AI SDK integration tests for a specific language (js or python) and report failures'
+author: 'Sentry Telemetry Experience Team'
+
+branding:
+  icon: 'check-circle'
+  color: 'purple'
+
+inputs:
+  language:
+    description: 'SDK language to test (js or python)'
+    required: true
+  github-token:
+    description: 'GitHub token for creating issues in the calling repo'
+    required: true
+  sentry-dsn:
+    description: 'Sentry DSN for error tracking'
+    required: true
+  openai-api-key:
+    description: 'OpenAI API key'
+    required: true
+  anthropic-api-key:
+    description: 'Anthropic API key'
+    required: true
+  google-api-key:
+    description: 'Google API key for GenAI'
+    required: true
+
+outputs:
+  success:
+    description: 'Whether all tests passed'
+    value: ${{ steps.run-tests.outputs.success }}
+
+runs:
+  using: 'composite'
+  steps:
+    - name: Setup Node.js
+      uses: actions/setup-node@v4
+      with:
+        node-version: '20'
+
+    - name: Setup Python
+      uses: actions/setup-python@v5
+      with:
+        python-version: '3.11'
+
+    - name: Install orchestration dependencies
+      shell: bash
+      working-directory: ${{ github.action_path }}/shared/orchestration
+      run: npm install
+
+    - name: Setup SDK dependencies
+      shell: bash
+      working-directory: ${{ github.action_path }}
+      run: npm run cli setup
+
+    - name: Run tests
+      id: run-tests
+      shell: bash
+      working-directory: ${{ github.action_path }}
+      env:
+        SENTRY_DSN: ${{ inputs.sentry-dsn }}
+        OPENAI_API_KEY: ${{ inputs.openai-api-key }}
+        ANTHROPIC_API_KEY: ${{ inputs.anthropic-api-key }}
+        GOOGLE_API_KEY: ${{ inputs.google-api-key }}
+      run: |
+        set +e
+        npm run cli run ${{ inputs.language }} --reports ctrf
+        EXIT_CODE=$?
+        set -e
+
+        if [ $EXIT_CODE -eq 0 ]; then
+          echo "success=true" >> $GITHUB_OUTPUT
+        else
+          echo "success=false" >> $GITHUB_OUTPUT
+        fi
+
+        exit $EXIT_CODE
+
+    - name: Create or update issue on failure
+      if: failure()
+      uses: actions/github-script@v7
+      with:
+        github-token: ${{ inputs.github-token }}
+        script: |
+          const fs = require('fs');
+          const path = require('path');
+          const language = '${{ inputs.language }}';
+          const runUrl = `${context.serverUrl}/${context.repo.owner}/${context.repo.repo}/actions/runs/${context.runId}`;
+
+          // Try to read test results from CTRF report
+          let resultsContent = 'Test execution failed. Check the [workflow run](' + runUrl + ') for details.';
+          const ctrfPath = '${{ github.action_path }}/shared/orchestration/test-results/ctrf-report.json';
+
+          if (fs.existsSync(ctrfPath)) {
+            try {
+              const results = JSON.parse(fs.readFileSync(ctrfPath, 'utf8'));
+              resultsContent = '### Test Results\n\n```json\n' + JSON.stringify(results, null, 2) + '\n```';
+            } catch (e) {
+              console.log('Could not parse test results:', e.message);
+            }
+          }
+
+          const issueLabel = 'ai-integration-test-failure';
+          const today = new Date().toISOString().split('T')[0];
+
+          const body = `## AI Integration Test Failure
+
+          **Language**: ${language}
+          **Date**: ${new Date().toISOString()}
+          **Workflow Run**: ${runUrl}
+
+          ${resultsContent}
+
+          ---
+          *This issue was automatically created by the [AI Integration Testing framework](https://github.com/getsentry/testing-ai-sdk-integrations).*
+          `;
+
+          // Check for existing open issues with the label
+          const issues = await github.rest.issues.listForRepo({
+            owner: context.repo.owner,
+            repo: context.repo.repo,
+            labels: issueLabel,
+            state: 'open'
+          });
+
+          if (issues.data.length > 0) {
+            // Update existing issue
+            await github.rest.issues.update({
+              owner: context.repo.owner,
+              repo: context.repo.repo,
+              issue_number: issues.data[0].number,
+              body: body
+            });
+            console.log(`Updated existing issue #${issues.data[0].number}`);
+          } else {
+            // Create new issue
+            const newIssue = await github.rest.issues.create({
+              owner: context.repo.owner,
+              repo: context.repo.repo,
+              title: `AI Integration Tests Failed (${language}) - ${today}`,
+              body: body,
+              labels: [issueLabel, 'automated']
+            });
+            console.log(`Created new issue #${newIssue.data.number}`);
+          }