Auto cherry pick by amanstep · Pull Request #32 · step-security/eval-action

amanstep · 2026-10-06T12:11:34Z

braintrustdata/eval-action@v2.0.0...v2.1.0

Move the project to Node 24 and pnpm 10 across local tool versions, GitHub workflows, and the packaged action runtime. Update Turbo and the lockfile for the newer package manager metadata. This is a breaking change as the github action now uses Node 24 instead of Node 20.

Replace the split Jest, ESLint, Prettier, esbuild, and Super Linter setup with Vite+ commands and configuration. Move packaging to vp pack with ESM output for the GitHub Action bundle, remove stale root tests, and update CI to run Vite+ checks, tests, and builds.

resolves braintrustdata/eval-action#88 Support Go eval workflows. Example action configuration: ```yaml - name: Run Evals uses: braintrustdata/eval-action@v1 with: api_key: ${{ secrets.BRAINTRUST_API_KEY }} runtime: go root: my_eval_dir paths: . ```

amanstep · 2026-10-06T12:12:13Z

📦 Target Release Version: v2.1.0
📋 Previous Release Version: v2.0.0

github-actions · 2026-10-06T12:13:17Z

🔍 Cherry-Pick Verification Report

📦 Upstream Changes: v2.0.0...v2.1.0

📋 File-by-File Analysis:

`.github/workflows/check-dist.yml`

Upstream has changes: ✅ Yes
File exists in PR: ✅ Yes
Changes match: ✅ Yes
Status: 🟢 Perfect - ✅ All changes applied correctly (+1 -1)

`.github/workflows/ci.yml`

Upstream has changes: ✅ Yes
File exists in PR: ✅ Yes
Changes match: ✅ Yes
Status: 🟢 Perfect - ✅ All changes applied correctly (+1 -1)

`.github/workflows/codeql-analysis.yml`

Upstream has changes: ✅ Yes
File exists in PR: ❌ No
Status: 🔴 Missing - File missing in PR (upstream has 4 additions, 4 deletions)

`.github/workflows/eval-py-uv.yml`

Upstream has changes: ✅ Yes
File exists in PR: ❌ No
Status: 🔴 Missing - File missing in PR (upstream has 4 additions, 3 deletions)

`.github/workflows/eval-py.yml`

Upstream has changes: ✅ Yes
File exists in PR: ❌ No
Status: 🔴 Missing - File missing in PR (upstream has 4 additions, 3 deletions)

`.github/workflows/eval-single.yaml`

Upstream has changes: ✅ Yes
File exists in PR: ❌ No
Status: 🔴 Missing - File missing in PR (upstream has 4 additions, 3 deletions)

`.github/workflows/eval.yml`

Upstream has changes: ✅ Yes
File exists in PR: ❌ No
Status: 🔴 Missing - File missing in PR (upstream has 4 additions, 3 deletions)

`.github/workflows/linter.yml`

Upstream has changes: ✅ Yes
File exists in PR: ✅ Yes
Changes match: ✅ Yes
Status: 🟢 Perfect - ✅ All changes applied correctly (+1 -1)

`README.md`

Upstream has changes: ✅ Yes
File exists in PR: ✅ Yes
Changes match: ❌ No
Status: 🟡 Partial - ❌ Cherry-pick incomplete (+108 -71) | Missing 7 additions | Missing 5 deletions

`action.yml`

Upstream has changes: ✅ Yes
File exists in PR: ✅ Yes
Changes match: ✅ Yes
Status: 🟢 Perfect - ✅ All changes applied correctly (+12 -0)

`eval/dist/index.js`

Upstream has changes: ✅ Yes
File exists in PR: ❌ No
Status: 🔴 Missing - ❌ No PR patch available (+9 -8)

`eval/src/main.test.ts`

Upstream has changes: ✅ Yes
File exists in PR: ✅ Yes
Changes match: ❌ No
Status: 🟡 Partial - ❌ Cherry-pick incomplete (+104 -0) | Missing 10 additions

`eval/src/main.ts`

Upstream has changes: ✅ Yes
File exists in PR: ✅ Yes
Changes match: ❌ No
Status: 🟡 Partial - ❌ Cherry-pick incomplete (+110 -38) | Missing 17 additions | Missing 3 deletions

`vite.config.ts`

Upstream has changes: ✅ Yes
File exists in PR: ❌ No
Status: 🔴 Missing - File missing in PR (upstream has 6 additions, 1 deletions)

📊 Summary:

Total files changed upstream: 14
Files present in PR: 7/14
Files with matching changes: 4/14

❌ Overall Status: 🔴 INCOMPLETE - Missing files or changes

github-actions · 2026-10-06T12:14:44Z

🔍 Cherry-Pick Verification Report

📦 Upstream Changes: v2.0.0...v2.1.0

📋 File-by-File Analysis:

`.github/workflows/check-dist.yml`

Upstream has changes: ✅ Yes
File exists in PR: ✅ Yes
Changes match: ✅ Yes
Status: 🟢 Perfect - ✅ All changes applied correctly (+1 -1)

`.github/workflows/ci.yml`

Upstream has changes: ✅ Yes
File exists in PR: ✅ Yes
Changes match: ✅ Yes
Status: 🟢 Perfect - ✅ All changes applied correctly (+1 -1)

`.github/workflows/codeql-analysis.yml`

Upstream has changes: ✅ Yes
File exists in PR: ❌ No
Status: 🔴 Missing - File missing in PR (upstream has 4 additions, 4 deletions)

`.github/workflows/eval-py-uv.yml`

Upstream has changes: ✅ Yes
File exists in PR: ❌ No
Status: 🔴 Missing - File missing in PR (upstream has 4 additions, 3 deletions)

`.github/workflows/eval-py.yml`

Upstream has changes: ✅ Yes
File exists in PR: ❌ No
Status: 🔴 Missing - File missing in PR (upstream has 4 additions, 3 deletions)

`.github/workflows/eval-single.yaml`

Upstream has changes: ✅ Yes
File exists in PR: ❌ No
Status: 🔴 Missing - File missing in PR (upstream has 4 additions, 3 deletions)

`.github/workflows/eval.yml`

Upstream has changes: ✅ Yes
File exists in PR: ❌ No
Status: 🔴 Missing - File missing in PR (upstream has 4 additions, 3 deletions)

`.github/workflows/linter.yml`

Upstream has changes: ✅ Yes
File exists in PR: ✅ Yes
Changes match: ✅ Yes
Status: 🟢 Perfect - ✅ All changes applied correctly (+1 -1)

`README.md`

Upstream has changes: ✅ Yes
File exists in PR: ✅ Yes
Changes match: ❌ No
Status: 🟡 Partial - ❌ Cherry-pick incomplete (+108 -71) | Missing 7 additions | Missing 5 deletions

`action.yml`

Upstream has changes: ✅ Yes
File exists in PR: ✅ Yes
Changes match: ✅ Yes
Status: 🟢 Perfect - ✅ All changes applied correctly (+12 -0)

`eval/dist/index.js`

Upstream has changes: ✅ Yes
File exists in PR: ✅ Yes
Changes match: ❌ No
Status: 🟡 Partial - ❌ Cherry-pick incomplete (+9 -8) | Missing 7 additions | Missing 6 deletions

`eval/src/main.test.ts`

Upstream has changes: ✅ Yes
File exists in PR: ✅ Yes
Changes match: ❌ No
Status: 🟡 Partial - ❌ Cherry-pick incomplete (+104 -0) | Missing 10 additions

`eval/src/main.ts`

Upstream has changes: ✅ Yes
File exists in PR: ✅ Yes
Changes match: ❌ No
Status: 🟡 Partial - ❌ Cherry-pick incomplete (+110 -38) | Missing 17 additions | Missing 3 deletions

`vite.config.ts`

Upstream has changes: ✅ Yes
File exists in PR: ❌ No
Status: 🔴 Missing - File missing in PR (upstream has 6 additions, 1 deletions)

📊 Summary:

Total files changed upstream: 14
Files present in PR: 8/14
Files with matching changes: 4/14

❌ Overall Status: 🔴 INCOMPLETE - Missing files or changes

github-actions · 2026-10-06T12:16:19Z

🔍 Cherry-Pick Verification Report

📦 Upstream Changes: v2.0.0...v2.1.0

📋 File-by-File Analysis:

`.github/workflows/check-dist.yml`

Upstream has changes: ✅ Yes
File exists in PR: ✅ Yes
Changes match: ✅ Yes
Status: 🟢 Perfect - ✅ All changes applied correctly (+1 -1)

`.github/workflows/ci.yml`

Upstream has changes: ✅ Yes
File exists in PR: ✅ Yes
Changes match: ✅ Yes
Status: 🟢 Perfect - ✅ All changes applied correctly (+1 -1)

`.github/workflows/codeql-analysis.yml`

Upstream has changes: ✅ Yes
File exists in PR: ❌ No
Status: 🔴 Missing - File missing in PR (upstream has 4 additions, 4 deletions)

`.github/workflows/eval-py-uv.yml`

Upstream has changes: ✅ Yes
File exists in PR: ❌ No
Status: 🔴 Missing - File missing in PR (upstream has 4 additions, 3 deletions)

`.github/workflows/eval-py.yml`

Upstream has changes: ✅ Yes
File exists in PR: ❌ No
Status: 🔴 Missing - File missing in PR (upstream has 4 additions, 3 deletions)

`.github/workflows/eval-single.yaml`

Upstream has changes: ✅ Yes
File exists in PR: ❌ No
Status: 🔴 Missing - File missing in PR (upstream has 4 additions, 3 deletions)

`.github/workflows/eval.yml`

Upstream has changes: ✅ Yes
File exists in PR: ❌ No
Status: 🔴 Missing - File missing in PR (upstream has 4 additions, 3 deletions)

`.github/workflows/linter.yml`

Upstream has changes: ✅ Yes
File exists in PR: ✅ Yes
Changes match: ✅ Yes
Status: 🟢 Perfect - ✅ All changes applied correctly (+1 -1)

`README.md`

Upstream has changes: ✅ Yes
File exists in PR: ✅ Yes
Changes match: ❌ No
Status: 🟡 Partial - ❌ Cherry-pick incomplete (+108 -71) | Missing 7 additions | Missing 5 deletions

`action.yml`

Upstream has changes: ✅ Yes
File exists in PR: ✅ Yes
Changes match: ✅ Yes
Status: 🟢 Perfect - ✅ All changes applied correctly (+12 -0)

`eval/dist/index.js`

Upstream has changes: ✅ Yes
File exists in PR: ✅ Yes
Changes match: ❌ No
Status: 🟡 Partial - ❌ Cherry-pick incomplete (+9 -8) | Missing 7 additions | Missing 6 deletions

`eval/src/main.test.ts`

Upstream has changes: ✅ Yes
File exists in PR: ✅ Yes
Changes match: ❌ No
Status: 🟡 Partial - ❌ Cherry-pick incomplete (+104 -0) | Missing 10 additions

`eval/src/main.ts`

Upstream has changes: ✅ Yes
File exists in PR: ✅ Yes
Changes match: ❌ No
Status: 🟡 Partial - ❌ Cherry-pick incomplete (+110 -38) | Missing 17 additions | Missing 3 deletions

`vite.config.ts`

Upstream has changes: ✅ Yes
File exists in PR: ❌ No
Status: 🔴 Missing - File missing in PR (upstream has 6 additions, 1 deletions)

📊 Summary:

Total files changed upstream: 14
Files present in PR: 8/14
Files with matching changes: 4/14

❌ Overall Status: 🔴 INCOMPLETE - Missing files or changes

github-actions · 2026-10-06T12:18:11Z

PR Review complete - see details below

github-actions · 2026-10-06T12:18:57Z

PR Review

Action Type

Node-based action - entry at eval/dist/index.js via node24.

Passed Checks

License: MIT with Braintrust Data Inc. and StepSecurity copyrights.
action.yml author: step-security.
SECURITY.md present.
No FUNDING.yml, renovate.json, PULL_REQUEST.md, ISSUE_TEMPLATE, CHANGELOG.md, .vscode.
Required workflows present: auto_cherry_pick.yml and actions_release.yml.
README uses major-only version tags (@v2, @v7, @v5) - no full semver.
StepSecurity banner present in README.
Subscription check URL correct (maintained-actions-subscription endpoint).
upstream variable braintrustdata/eval-action matches auto_cherry_pick.yml.
package.json: author step-security, repository contains step-security.
All dependencies used (@actions/core, @actions/github, axios, braintrust, zod).
dist folder present (eval/dist/index.js).
build script present in package.json.
script input present in audit_package.yml and actions_release.yml.

Failed Checks

.tool-versions duplicate entries: nodejs 24.15.0 and pnpm 10.33.0 each listed twice. Cherry-pick artifact - must fix before merging.

Warnings

paths SHELL_METACHARACTERS allows tilde (~) which bash expands with shell:true.
use_proxy marked required:true but has default - redundant.

Security Findings

Command injection mitigated: paths validated via regex; root used as cwd only.
API key masked with core.setSecret() and scoped to subprocess env only.
No critical vulnerabilities found.

Summary

Well-structured cherry-pick adding report_scores, report_metrics, github_token, step_key inputs plus README/checkout updates. Single blocker: duplicate .tool-versions entries.

github-actions · 2026-10-06T12:22:42Z

PR Review

Action Type: Node-based (node24 runtime, eval/dist/index.js)

PASSED CHECKS:

License: MIT with both Braintrust Data Inc. and StepSecurity copyright
action.yml author: step-security
SECURITY.md: Present
FUNDING.yml: Not present
renovate.json: Not present
PULL_REQUEST.md: Not present
ISSUE_TEMPLATE folder: Not present
CHANGELOG.md: Not present
.vscode folder: Not present
.github/workflows: Has auto_cherry_pick.yml and actions_release.yml
README banner: StepSecurity banner present
README semver: eval-action uses major-only tags (@v2)
Subscription URL: Correct endpoint called
Upstream variable: braintrustdata/eval-action matches auto_cherry_pick.yml
package.json author: step-security, repository URL has step-security
dist folder: Present at eval/dist/
Build script: build field present in scripts
Dependencies: All used

FAILED CHECKS:

.tool-versions duplicate entries: nodejs 24.15.0 and pnpm 10.33.0 each appear twice. Cherry-pick artifact.
examples/go/go.yml broken: Contains two complete workflow definitions both named 'Run Go evals', making invalid YAML. First block references braintrustdata/eval-action@v1 and actions/checkout@v4 instead of step-security equivalents. Cherry-pick artifact.

WARNINGS:

validateSubscription() parses GITHUB_EVENT_PATH JSON without try-catch. Malformed file causes unhandled crash.
README examples use unpinned action references (checkout@v7, setup-node@v7) without SHA pinning.

SECURITY FINDINGS:

paths input is protected against shell injection via SHELL_METACHARACTERS regex validation.
No other security issues found.

SUMMARY:
report_scores/report_metrics filtering is correctly implemented with proper parsing and tests. Two cherry-pick artifacts must be fixed before merge: duplicate .tool-versions and broken examples/go/go.yml (invalid YAML referencing upstream action).

github-actions · 2026-10-06T12:24:46Z

PR Review

Action Type

Node-based action (runs on node24, entry point: eval/dist/index.js)

Passed Checks

License - MIT license present with copyright for both original author (Braintrust Data Inc.) and StepSecurity.
action.yml author - author: 'step-security' is set correctly.
SECURITY.md - Present at repo root.
No FUNDING.yml - Not present.
Workflow files - .github/workflows/ contains both auto_cherry_pick.yml and actions_release.yml.
No renovate.json - Not present.
No PULL_REQUEST.md - Not present.
No ISSUE_TEMPLATE folder - Not present.
No CHANGELOG.md - Not present.
No .vscode folder - Not present.
README banner - StepSecurity Maintained Action banner present at line 1.
README examples use major version only - All usage examples use major versions only (e.g., @v2, @v7, @v5). No full semver tags.
Subscription check - validateSubscription() is present and calls the correct URL (https://agent.api.stepsecurity.io/v1/github/${GITHUB_REPOSITORY}/actions/maintained-actions-subscription).
Upstream variable - upstream = "braintrustdata/eval-action" correctly matches original-owner: "braintrustdata" and repo-name: "eval-action" from auto_cherry_pick.yml.
package.json author - "author": "step-security" is correct.
package.json repository - URL contains step-security.
dist folder - eval/dist/index.js and eval/dist/index.js.map are present.
Build script - "build": "vp run package" present in package.json.
actions_release.yml has script input - Present.

Failed Checks

github_token required mismatch - action.yml declares github_token with required: true, but the README table (updated in this PR) lists it as No (optional) and describes it as defaulting to ${{ github.token }}. Since a default is set, required in action.yml should be changed to false to match the documentation. See inline comment on action.yml:58.

Warnings

github_token description not updated in action.yml - The PR updates the README to describe github_token as "GitHub token used to create or update PR comments", but action.yml still has the generic description "Your GitHub token". Consider syncing these for consistency.
Example workflows use non-pinned action refs - Example workflows reference actions with major version tags (actions/checkout@v7, actions/setup-go@v5) rather than pinned commit SHAs. This meets the review criteria (major version is required), but SHA pinning is better for supply-chain security in examples that users copy directly.

Security Findings

Potential command injection via root input - The paths input is validated against SHELL_METACHARACTERS with an explicit note that the eval command runs with shell: true. However, the root input has no such validation (root: z.string()). If root is interpolated into the shell command string rather than used solely as a cwd option, a crafted root value could inject arbitrary shell commands. Recommend applying the same metacharacter refine to root, or confirming that root is only ever passed as cwd and never concatenated into a command string. See inline comment on eval/src/main.ts:92.

Summary

The PR cherry-picks upstream improvements (new report_scores/report_metrics filter inputs, improved README, and Go example fix) and is well-structured overall. One documentation inconsistency (github_token required: true vs README "No") must be resolved before merge, and the unvalidated root shell input warrants a security review.

github-actions · 2026-10-06T12:24:52Z

Follow-up note: The .tool-versions duplicate entry issue appears to have already been addressed in commit cae5a23 ('fix: remove duplicate entry') pushed after the initial cherry-pick. The examples/go/go.yml issue (duplicate workflow blocks + braintrustdata/eval-action@v1 reference) still requires attention — see the inline comment on that file.

github-actions · 2026-10-06T12:26:01Z

Review Update: Two corrections to the initial review comment above.

CORRECTION - .tool-versions: The duplicate-entry issue reported in the initial review is a local merge-commit artifact in this test environment and is NOT part of this PR actual diff. Please disregard that Failed Check.

NEW CRITICAL FINDING - examples/go/go.yml: This file has its entire YAML content duplicated (file doubled from 42 to 84 lines). The first 42 lines contain the old upstream workflow using braintrustdata/eval-action@v1 and actions/checkout@v4. The second 42 lines contain the correct step-security workflow using step-security/eval-action@v2 and actions/checkout@v7. This is a cherry-pick merge artifact. The old upstream content (lines 1-42) must be removed before merging - the file has duplicate top-level YAML keys and the braintrustdata action reference must not appear in the step-security fork examples.

An inline comment has been added on the diff to mark the affected lines.

github-actions Bot and others added 4 commits October 5, 2026 10:22

chore: Cherry-picked changes from upstream

4cdd947

amanstep added the review-required Request Claude AI code review on the PR label Oct 6, 2026

chore: dist updated

f9bd671

fix: remove duplicate entry

cae5a23

Raj-StepSecurity approved these changes Oct 6, 2026

View reviewed changes