You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit d3de647
Browse filesBrowse the repository at this point in the historyBrowse files
| `api_key` | Yes | Your [Braintrust API key](https://www.braintrust.dev/app/settings/api-keys). |
49
+
| `runtime` | Yes | The runtime to use: `node`, `python`, or `go`. |
50
+
| `root` | No | Root directory containing your evals. Defaults to `.`. |
51
+
| `paths` | No | Paths or glob patterns, relative to `root`, containing evals to run. Defaults to `.`. |
52
+
| `package_manager` | No | `npm` or `pnpm` for Node; `pip` or `uv` for Python; `go` for Go. Can be omitted for the default package manager. |
53
+
| `use_proxy` | No | Set to `true` to use the Braintrust proxy at `https://braintrustproxy.com/v1`, which can cache repetitive LLM calls and speed up evals. Defaults to `true`. |
54
+
| `terminate_on_failure` | No | Set to `true` to stop the eval process when an error occurs. Defaults to `false`. Ignored for Go evals. |
55
+
| `report_scores` | No | Comma- or newline-separated score names to include in the PR comment. Defaults to all available scores. |
56
+
| `report_metrics` | No | Comma- or newline-separated metric names to include in the PR comment. Defaults to all available metrics. |
57
+
| `github_token` | No | GitHub token used to create or update PR comments. Defaults to `${{ github.token }}`. |
34
58
35
59
## Full example
36
60
37
61
```yaml
38
62
name: Run pnpm evals
39
63
40
64
on:
65
+
pull_request:
41
66
push:
42
-
# Uncomment to run only when files in the 'evals' directory change
43
-
# - paths:
44
-
# - "evals/**"
67
+
# Uncomment to run only when files in the 'evals' directory change.
68
+
# paths:
69
+
# - "evals/**"
45
70
46
71
permissions:
47
-
pull-requests: write
48
72
contents: read
73
+
pull-requests: write
49
74
50
75
jobs:
51
76
eval:
@@ -54,74 +79,85 @@ jobs:
54
79
55
80
steps:
56
81
- name: Checkout
57
-
id: checkout
58
82
uses: actions/checkout@v7
59
83
with:
60
84
fetch-depth: 0
61
85
62
86
- name: Setup Node.js
63
-
id: setup-node
64
87
uses: actions/setup-node@v7
65
88
with:
66
-
node-version: 20
89
+
node-version: 24
67
90
68
91
- uses: step-security/action-setup@v5
69
92
with:
70
-
version: 8
93
+
version: 10
71
94
72
-
- name: Install Dependencies
73
-
id: install
74
-
run: pnpm install
95
+
- name: Install dependencies
96
+
run: pnpm install --frozen-lockfile
75
97
76
-
- name: Run Evals
98
+
- name: Run evals
77
99
uses: step-security/eval-action@v2
78
100
with:
79
101
api_key: ${{ secrets.BRAINTRUST_API_KEY }}
80
102
runtime: node
103
+
package_manager: pnpm
81
104
root: my_eval_dir
82
105
```
83
106
84
-
> [!IMPORTANT] You must specify `permissions` for the action to leave comments
85
-
> on your PR. Without these permissions, you'll see Github API errors.
86
-
87
-
To see examples of fully configured templates, see the `examples` directory:
107
+
For more fully configured workflows, see the `examples` directory:
88
108
89
109
- [`node with npm`](examples/node/npm.yml)
90
110
- [`node with pnpm`](examples/node/pnpm.yml)
91
111
- [`python with pip`](examples/python/pip.yml)
92
112
- [`python with uv`](examples/python/uv.yml)
93
113
- [`go`](examples/go/go.yml)
94
114
95
-
## Go evals
96
-
97
-
The Go runtime executes `go run ${paths}`. To include Go eval results in the PR
98
-
comment, print each `ExperimentSummary` as one JSON line after calling
99
-
`result.Summarize(ctx)`:
100
-
101
-
```go
102
-
summary, err := result.Summarize(ctx)
103
-
if err != nil {
104
-
log.Fatal(err)
105
-
}
106
-
b, err := json.Marshal(summary)
107
-
if err != nil {
108
-
log.Fatal(err)
109
-
}
110
-
fmt.Println(string(b))
111
-
```
115
+
## Runtime behavior
116
+
117
+
- **Node and Python:** the action runs `braintrust eval --jsonl` from `root` and
118
+
collects the emitted experiment summaries.
119
+
- **Go:** the action runs `go run ${paths}` from `root`. To include Go eval
120
+
results in the PR comment, print each `ExperimentSummary` as one JSON line
121
+
after calling `result.Summarize(ctx)`:
122
+
123
+
```go
124
+
summary, err := result.Summarize(ctx)
125
+
if err != nil {
126
+
log.Fatal(err)
127
+
}
128
+
b, err := json.Marshal(summary)
129
+
if err != nil {
130
+
log.Fatal(err)
131
+
}
132
+
fmt.Println(string(b))
133
+
```
134
+
135
+
The action creates or updates a single PR comment with a Braintrust link and a
136
+
result table with score and metric sections. To show only selected results, set
137
+
`report_scores` and `report_metrics` to their exact names:
112
138
113
-
## How it works
139
+
```yaml
140
+
- uses: step-security/eval-action@v2
141
+
with:
142
+
api_key: ${{ secrets.BRAINTRUST_API_KEY }}
143
+
runtime: node
144
+
report_scores: Levenshtein, Factuality
145
+
report_metrics: |
146
+
Duration
147
+
Cost
148
+
```
114
149
115
-
For Node and Python, the action runs `braintrust eval`. For Go, the action runs
116
-
`go run`on `paths` from `root`. It collects experiment results emitted as JSONL
117
-
and posts them as a comment in the PR alongside a link to Braintrust. For
118
-
example:
150
+
Each input accepts comma- or newline-separated names and filters its category
151
+
independently. When an input is omitted or empty, all results in that category
152
+
are included. For example:
119
153
120
-
### Example braintrust eval report
154
+
### Example Braintrust eval report
121
155
122
-
**Say Hi Bot (HEAD-1714341466)**
156
+
**[Say Hi Bot (HEAD-1714341466)](https://www.braintrustdata.com/app/braintrustdata.com/p/Say%20Hi%20Bot/experiments/HEAD-1714341466)**
reason = "esbuild@0.28.0 is a transitive dependency of braintrust; ExperimentSummary is a type-only import erased at bundle time, so braintrust's esbuild is never exercised"
2
+
id = "GHSA-82fw-gwwq-j7x9"
3
+
reason = "dev dependency (@vitest/mocker, vitest) — not bundled into the action runtime"
4
4
5
5
[[IgnoredVulns]]
6
-
id = "GHSA-w5hq-g745-h8pq"
7
-
reason = "uuid@9.0.1 is a transitive dependency of @braintrust/core; only uuid.parse/stringify are called (not the vulnerable generators), and this action only uses capitalize() which has no uuid code path"
6
+
id = "GHSA-5gmw-xhrv-c9v3"
7
+
reason = "dev dependency (tinypool) — not bundled into the action runtime"
8
+
9
+
[[IgnoredVulns]]
10
+
id = "GHSA-85c8-ppgw-ccpr"
11
+
reason = "dev dependency (tinypool) — not bundled into the action runtime"
0 commit comments