You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Per-scenario overrides (`resource_group`, `cluster_name`, `kubeconfig`, `test_env_vars`) still apply. Use `--skip-setup` or `--skip-cleanup` to bypass hooks. Expect the test to log iteration progress, classifier scores, and (on the final iteration) a Braintrust link when uploads are enabled.
Install the Braintrust client once networking is available:
76
-
77
-
```bash
78
-
poetry add braintrust --group dev
79
-
```
80
-
81
-
For a one-shot setup that installs all dev dependencies (Braintrust, openai, autoevals, pytest plugins), run:
82
-
83
-
```bash
84
-
poetry install --with dev
85
-
```
86
-
87
92
Each iteration logs to Braintrust; the console prints a clickable link (for aware terminals) when uploads succeed.
88
93
89
94
**Tips**
@@ -96,11 +101,7 @@ Each iteration logs to Braintrust; the console prints a clickable link (for awar
96
101
97
102
- Enabled by default; set `ENABLE_CLASSIFIER=false` to opt out.
98
103
- Requires Azure OpenAI credentials: `AZURE_API_BASE`, `AZURE_API_KEY`, `AZURE_API_VERSION`, and a classifier deployment specified via `CLASSIFIER_MODEL` (e.g. `azure/<deployment>`). Defaults to the same deployment as `MODEL` when not provided.
99
-
- Install classifier dependencies when online:
100
-
101
-
```bash
102
-
poetry add openai autoevals --group dev
103
-
```
104
+
- Install classifier dependencies when online (see Environment Setup above if not already installed).
104
105
105
106
- Scenarios can override the grading style by adding:
106
107
@@ -115,7 +116,7 @@ Classifier scores and rationales are attached to Braintrust uploads and printed
115
116
**Debugging classifiers**
116
117
117
118
```
118
-
poetry run pytest ... -o log_cli=true -o log_cli_level=DEBUG -s
0 commit comments