Skip to content

Commit 7b2e3c9

Browse files
committed
test: select backing services by profile instead of credential tags
Tests were gated on whether an API key happened to be present, so a test skipped on pull requests was a test nobody ran -- and fork PRs can never read repository secrets, so deferring them to a labelled run never covered the case it existed for. Replace that axis with service profiles (tests/profiles.yml): a run declares which backing services are real, and stubs replay cassettes recorded from real providers and keyed by audio sha256, so the same content assertions hold with or without credentials. Removes the credential-gated workflows and their runner script, which the profile-driven robot-tests workflow now covers.
1 parent 5b43bbf commit 7b2e3c9

47 files changed

Lines changed: 22768 additions & 1642 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/workflows/README.md‎

Lines changed: 51 additions & 146 deletions
Original file line numberDiff line numberDiff line change
@@ -14,168 +14,65 @@ Chronicle uses **three separate test workflows** to balance fast PR feedback wit
1414
|----------|---------|---------------|----------|---------|
1515
| `python-tests.yml` | Relevant PRs, dev/main | Root, backend, and ASR pytest lanes | Not required | Unit tests and branch coverage reports |
1616
| `robot-tests.yml` | All PRs | No-API Robot subset | Not required | Fast PR validation |
17-
| `full-tests-with-api.yml` | Push to dev/main | Full Robot selection | Required | Comprehensive validation |
18-
| `pr-tests-with-api.yml` | PR label trigger | Full Robot selection | Required | Pre-merge API testing |
1917

2018
## Workflow Details
2119

22-
### 1. `robot-tests.yml` - PR Tests (No API Keys)
20+
### 1. `robot-tests.yml` - Robot Framework Tests
2321

2422
**File**: `.github/workflows/robot-tests.yml`
2523

26-
**Trigger**:
27-
```yaml
28-
on:
29-
pull_request:
30-
paths:
31-
- 'tests/**/*.robot'
32-
- 'tests/**/*.py'
33-
- 'backends/advanced/src/**'
34-
```
24+
One workflow, one test suite. What varies per run is the **service profile**
25+
(`tests/profiles.yml`) -- which backing services are real:
3526

36-
**Characteristics**:
37-
- **No secrets required** - Works for external contributors
38-
- **Excludes**: Tests tagged with `requires-api-keys`
39-
- **Config**: `tests/configs/mock-services.yml`
40-
- **Makefile Target**: `make test-no-api OUTPUTDIR=results-no-api`
41-
- **Results**: `results-no-api/`
42-
- **Time**: ~10-15 minutes
43-
- **Selection**: Robot cases that do not require secrets, GPUs, or excluded slow/SDK environments
27+
| Trigger | Profiles | Secrets |
28+
|---------|----------|---------|
29+
| `pull_request` | `mock` | none needed |
30+
| `push` to `dev`/`main` | `mock`, `deepgram-openai` | Deepgram + OpenAI |
31+
| `workflow_dispatch` | chosen from a dropdown | as the profile requires |
4432

45-
**Benefits**:
46-
- Fast feedback on PRs
47-
- No API costs for every PR
48-
- External contributors can run full CI
49-
- Most development workflows covered
33+
**There is no API-key test split.** Every test runs in every profile. The `mock`
34+
profile replays recorded real provider responses from `tests/cassettes/`, so
35+
assertions about transcript content hold identically with or without
36+
credentials.
5037

51-
**What's Tested**:
52-
- API endpoints (auth, CRUD, permissions)
53-
- Infrastructure (workers, queues, health)
54-
- Basic integration (non-transcription)
38+
Two consequences worth knowing:
5539

56-
**What's Skipped**:
57-
- Audio upload with transcription
58-
- Memory operations requiring LLM
59-
- Audio streaming with STT
60-
- Full E2E pipeline tests
40+
- Fork pull requests work unchanged, because the PR profile needs no secrets.
41+
- The old `test-with-api-keys` label is gone. It could never have worked for the
42+
fork case it was added for: a `pull_request` run from a fork gets no secrets
43+
regardless of labels. Supporting that would require `pull_request_target`,
44+
which executes untrusted code with secrets in scope.
6145

62-
### 2. `full-tests-with-api.yml` - Dev/Main Tests (Full Suite)
46+
To validate against real providers before merging, either push to `dev`, or run
47+
the workflow manually:
6348

64-
**File**: `.github/workflows/full-tests-with-api.yml`
65-
66-
**Trigger**:
67-
```yaml
68-
on:
69-
push:
70-
branches: [dev, main]
71-
paths:
72-
- 'tests/**'
73-
- 'backends/advanced/src/**'
74-
workflow_dispatch: # Manual trigger available
49+
```bash
50+
gh workflow run robot-tests.yml --ref <branch> -f profile=deepgram-openai
7551
```
7652

77-
**Characteristics**:
78-
- **Requires secrets**: `DEEPGRAM_API_KEY`, `OPENAI_API_KEY`, `HF_TOKEN`
79-
- **Includes**: All tests (including `requires-api-keys`)
80-
- **Config**: `tests/configs/deepgram-openai.yml`
81-
- **Test Script**: `./run-robot-tests.sh`
82-
- **Results**: `results/`
83-
- **Time**: ~20-30 minutes
84-
- **Coverage**: 100% of test suite
85-
86-
**Benefits**:
87-
- Full validation before deployment
88-
- Catches API integration issues
89-
- Validates real transcription and memory processing
90-
- Comprehensive E2E coverage
91-
92-
**What's Tested**:
93-
- Everything from `robot-tests.yml` PLUS:
94-
- Audio upload with real transcription
95-
- Memory extraction with LLM
96-
- Audio streaming with STT
97-
- Full E2E pipeline validation
98-
99-
### 3. `pr-tests-with-api.yml` - Label-Triggered PR Tests
53+
## Usage Guide
10054

101-
**File**: `.github/workflows/pr-tests-with-api.yml`
55+
### For Contributors
10256

103-
**Trigger**:
104-
```yaml
105-
on:
106-
pull_request:
107-
types: [labeled, synchronize]
108-
```
57+
Run the same thing CI runs on your PR:
10958

110-
**Condition**:
111-
```yaml
112-
if: contains(github.event.pull_request.labels.*.name, 'test-with-api-keys')
59+
```bash
60+
cd tests
61+
make test # profile: mock -- free, no credentials
11362
```
11463

115-
**Characteristics**:
116-
- **Requires**: PR labeled with `test-with-api-keys`
117-
- **Requires secrets**: `DEEPGRAM_API_KEY`, `OPENAI_API_KEY`, `HF_TOKEN`
118-
- **Includes**: All tests (same as full-tests-with-api.yml)
119-
- **Config**: `tests/configs/deepgram-openai.yml`
120-
- **Time**: ~20-30 minutes
121-
- **Re-runs**: On new commits while label present
122-
123-
**Benefits**:
124-
- Test API integrations before merging
125-
- Useful for PRs modifying transcription/LLM code
126-
- Maintainers can trigger on trusted PRs
127-
- Catches issues before they reach dev/main
128-
129-
**Use Cases**:
130-
- PRs that modify transcription logic
131-
- PRs that change memory extraction
132-
- PRs that affect audio processing pipeline
133-
- Before merging large feature branches
134-
135-
## Usage Guide
136-
137-
### For Contributors
138-
139-
**Normal PR Workflow**:
140-
1. Push your branch
141-
2. Create PR
142-
3. `robot-tests.yml` runs the no-API Robot selection automatically
143-
4. Fix any failures
144-
5. Merge when tests pass
145-
146-
**Testing API Integrations**:
147-
1. Push your branch
148-
2. Create PR
149-
3. Ask maintainer to add `test-with-api-keys` label
150-
4. `pr-tests-with-api.yml` runs the full Robot selection
151-
5. Fix any failures
152-
6. Merge when tests pass
64+
If it passes locally it should pass on the PR: same suite, same profile.
15365

15466
### For Maintainers
15567

156-
**Adding the Label**:
15768
```bash
158-
# Via GitHub UI
159-
1. Go to PR
160-
2. Click "Labels" on right sidebar
161-
3. Select "test-with-api-keys"
69+
# Same suite against real providers
70+
cd tests && make test PROFILE=deepgram-openai
16271

163-
# Via GitHub CLI
164-
gh pr edit <pr-number> --add-label "test-with-api-keys"
72+
# Refresh recorded provider responses (the only step needing credentials)
73+
cd tests && make record-cassettes PROFILE=deepgram-openai
16574
```
16675

167-
**When to Use Label**:
168-
- PR modifies audio processing or transcription
169-
- PR changes memory extraction logic
170-
- PR affects LLM integration
171-
- Before merging large features
172-
- When in doubt about API changes
173-
174-
**Removing the Label**:
175-
- Label is automatically retained on new commits
176-
- Remove manually if no longer needed
177-
- Saves API costs if changes don't affect APIs
178-
17976
## Test Results
18077

18178
### PR Comments
@@ -220,8 +117,8 @@ Downloadable artifacts for deeper analysis:
220117
Must be configured in GitHub repository settings:
221118

222119
```bash
223-
DEEPGRAM_API_KEY # Required for full-tests-with-api.yml
224-
OPENAI_API_KEY # Required for full-tests-with-api.yml
120+
DEEPGRAM_API_KEY # Required for the deepgram-openai profile (dev/main pushes)
121+
OPENAI_API_KEY # Required for the deepgram-openai profile (dev/main pushes)
225122
HF_TOKEN # Optional (speaker recognition)
226123
```
227124

@@ -256,7 +153,7 @@ Workflows validate secrets before running tests:
256153
- **Monthly**: Potentially hundreds of runs
257154
- **Savings**: Significant with external contributors
258155

259-
**Full Tests** (`full-tests-with-api.yml`, `pr-tests-with-api.yml`):
156+
**Real-provider runs** (`deepgram-openai` profile, dev/main pushes):
260157
- **Transcription**: ~$0.10-0.30 per run (Deepgram)
261158
- **LLM**: ~$0.05-0.15 per run (OpenAI)
262159
- **Total**: ~$0.15-0.45 per run
@@ -365,14 +262,22 @@ runs-on: ubuntu-latest
365262
- Look for timing issues (increase timeouts)
366263
- Check Docker resource limits in CI
367264

368-
### Label Workflow Not Running
265+
### Want a real-provider run on a PR
369266

370-
**Problem**: Added label but workflow doesn't trigger
371-
**Solutions**:
372-
- Verify label name is exactly `test-with-api-keys`
373-
- Check workflow trigger includes `types: [labeled]`
374-
- Try removing and re-adding label
375-
- Push new commit to trigger synchronize event
267+
There is no label for this, and there cannot usefully be one: a `pull_request`
268+
run from a fork cannot read repository secrets. Instead:
269+
270+
```bash
271+
gh workflow run robot-tests.yml --ref <branch> -f profile=deepgram-openai
272+
```
273+
274+
Or merge to `dev`, where the `deepgram-openai` profile runs automatically.
275+
276+
### A profile fails immediately with a credential error
277+
278+
That is the harness refusing to start a stack it cannot configure. The message
279+
names the missing variable and how to set it. Run `make test PROFILE=mock` if
280+
you do not need real providers.
376281

377282
## Maintenance
378283

0 commit comments

Comments
 (0)