@@ -14,168 +14,65 @@ Chronicle uses **three separate test workflows** to balance fast PR feedback wit
1414| ----------| ---------| ---------------| ----------| ---------|
1515| ` python-tests.yml ` | Relevant PRs, dev/main | Root, backend, and ASR pytest lanes | Not required | Unit tests and branch coverage reports |
1616| ` robot-tests.yml ` | All PRs | No-API Robot subset | Not required | Fast PR validation |
17- | ` full-tests-with-api.yml ` | Push to dev/main | Full Robot selection | Required | Comprehensive validation |
18- | ` pr-tests-with-api.yml ` | PR label trigger | Full Robot selection | Required | Pre-merge API testing |
1917
2018## Workflow Details
2119
22- ### 1. ` robot-tests.yml ` - PR Tests (No API Keys)
20+ ### 1. ` robot-tests.yml ` - Robot Framework Tests
2321
2422** File** : ` .github/workflows/robot-tests.yml `
2523
26- ** Trigger** :
27- ``` yaml
28- on :
29- pull_request :
30- paths :
31- - ' tests/**/*.robot'
32- - ' tests/**/*.py'
33- - ' backends/advanced/src/**'
34- ` ` `
24+ One workflow, one test suite. What varies per run is the ** service profile**
25+ (` tests/profiles.yml ` ) -- which backing services are real:
3526
36- **Characteristics**:
37- - **No secrets required** - Works for external contributors
38- - **Excludes**: Tests tagged with ` requires-api-keys`
39- - **Config**: `tests/configs/mock-services.yml`
40- - **Makefile Target**: `make test-no-api OUTPUTDIR=results-no-api`
41- - **Results**: `results-no-api/`
42- - **Time**: ~10-15 minutes
43- - **Selection**: Robot cases that do not require secrets, GPUs, or excluded slow/SDK environments
27+ | Trigger | Profiles | Secrets |
28+ | ---------| ----------| ---------|
29+ | ` pull_request ` | ` mock ` | none needed |
30+ | ` push ` to ` dev ` /` main ` | ` mock ` , ` deepgram-openai ` | Deepgram + OpenAI |
31+ | ` workflow_dispatch ` | chosen from a dropdown | as the profile requires |
4432
45- **Benefits**:
46- - Fast feedback on PRs
47- - No API costs for every PR
48- - External contributors can run full CI
49- - Most development workflows covered
33+ ** There is no API-key test split.** Every test runs in every profile. The ` mock `
34+ profile replays recorded real provider responses from ` tests/cassettes/ ` , so
35+ assertions about transcript content hold identically with or without
36+ credentials.
5037
51- **What's Tested**:
52- - API endpoints (auth, CRUD, permissions)
53- - Infrastructure (workers, queues, health)
54- - Basic integration (non-transcription)
38+ Two consequences worth knowing:
5539
56- **What's Skipped**:
57- - Audio upload with transcription
58- - Memory operations requiring LLM
59- - Audio streaming with STT
60- - Full E2E pipeline tests
40+ - Fork pull requests work unchanged, because the PR profile needs no secrets.
41+ - The old ` test- with-api-keys ` label is gone. It could never have worked for the
42+ fork case it was added for: a ` pull_request ` run from a fork gets no secrets
43+ regardless of labels. Supporting that would require ` pull_request_target ` ,
44+ which executes untrusted code with secrets in scope.
6145
62- # ## 2. `full-tests-with-api.yml` - Dev/Main Tests (Full Suite)
46+ To validate against real providers before merging, either push to ` dev ` , or run
47+ the workflow manually:
6348
64- **File**: `.github/workflows/full-tests-with-api.yml`
65-
66- **Trigger**:
67- ` ` ` yaml
68- on:
69- push:
70- branches: [dev, main]
71- paths:
72- - 'tests/**'
73- - 'backends/advanced/src/**'
74- workflow_dispatch: # Manual trigger available
49+ ``` bash
50+ gh workflow run robot-tests.yml --ref < branch> -f profile=deepgram-openai
7551```
7652
77- **Characteristics**:
78- - **Requires secrets**: `DEEPGRAM_API_KEY`, `OPENAI_API_KEY`, `HF_TOKEN`
79- - **Includes**: All tests (including `requires-api-keys`)
80- - **Config**: `tests/configs/deepgram-openai.yml`
81- - **Test Script**: `./run-robot-tests.sh`
82- - **Results**: `results/`
83- - **Time**: ~20-30 minutes
84- - **Coverage**: 100% of test suite
85-
86- **Benefits**:
87- - Full validation before deployment
88- - Catches API integration issues
89- - Validates real transcription and memory processing
90- - Comprehensive E2E coverage
91-
92- **What's Tested**:
93- - Everything from `robot-tests.yml` PLUS :
94- - Audio upload with real transcription
95- - Memory extraction with LLM
96- - Audio streaming with STT
97- - Full E2E pipeline validation
98-
99- # ## 3. `pr-tests-with-api.yml` - Label-Triggered PR Tests
53+ ## Usage Guide
10054
101- **File**: `.github/workflows/pr-tests-with-api.yml`
55+ ### For Contributors
10256
103- **Trigger**:
104- ` ` ` yaml
105- on:
106- pull_request:
107- types: [labeled, synchronize]
108- ` ` `
57+ Run the same thing CI runs on your PR:
10958
110- **Condition**:
111- ` ` ` yaml
112- if: contains(github.event.pull_request.labels.*.name, 'test-with-api-keys')
59+ ``` bash
60+ cd tests
61+ make test # profile: mock -- free, no credentials
11362```
11463
115- **Characteristics**:
116- - **Requires**: PR labeled with `test-with-api-keys`
117- - **Requires secrets**: `DEEPGRAM_API_KEY`, `OPENAI_API_KEY`, `HF_TOKEN`
118- - **Includes**: All tests (same as full-tests-with-api.yml)
119- - **Config**: `tests/configs/deepgram-openai.yml`
120- - **Time**: ~20-30 minutes
121- - **Re-runs**: On new commits while label present
122-
123- **Benefits**:
124- - Test API integrations before merging
125- - Useful for PRs modifying transcription/LLM code
126- - Maintainers can trigger on trusted PRs
127- - Catches issues before they reach dev/main
128-
129- **Use Cases**:
130- - PRs that modify transcription logic
131- - PRs that change memory extraction
132- - PRs that affect audio processing pipeline
133- - Before merging large feature branches
134-
135- # # Usage Guide
136-
137- # ## For Contributors
138-
139- **Normal PR Workflow**:
140- 1. Push your branch
141- 2. Create PR
142- 3. `robot-tests.yml` runs the no-API Robot selection automatically
143- 4. Fix any failures
144- 5. Merge when tests pass
145-
146- **Testing API Integrations**:
147- 1. Push your branch
148- 2. Create PR
149- 3. Ask maintainer to add `test-with-api-keys` label
150- 4. `pr-tests-with-api.yml` runs the full Robot selection
151- 5. Fix any failures
152- 6. Merge when tests pass
64+ If it passes locally it should pass on the PR: same suite, same profile.
15365
15466### For Maintainers
15567
156- **Adding the Label**:
15768``` bash
158- # Via GitHub UI
159- 1. Go to PR
160- 2. Click "Labels" on right sidebar
161- 3. Select "test-with-api-keys"
69+ # Same suite against real providers
70+ cd tests && make test PROFILE=deepgram-openai
16271
163- # Via GitHub CLI
164- gh pr edit <pr-number> --add-label "test-with-api-keys"
72+ # Refresh recorded provider responses (the only step needing credentials)
73+ cd tests && make record-cassettes PROFILE=deepgram-openai
16574```
16675
167- **When to Use Label**:
168- - PR modifies audio processing or transcription
169- - PR changes memory extraction logic
170- - PR affects LLM integration
171- - Before merging large features
172- - When in doubt about API changes
173-
174- **Removing the Label**:
175- - Label is automatically retained on new commits
176- - Remove manually if no longer needed
177- - Saves API costs if changes don't affect APIs
178-
17976## Test Results
18077
18178### PR Comments
@@ -220,8 +117,8 @@ Downloadable artifacts for deeper analysis:
220117Must be configured in GitHub repository settings:
221118
222119``` bash
223- DEEPGRAM_API_KEY # Required for full-tests-with-api.yml
224- OPENAI_API_KEY # Required for full-tests-with-api.yml
120+ DEEPGRAM_API_KEY # Required for the deepgram-openai profile (dev/main pushes)
121+ OPENAI_API_KEY # Required for the deepgram-openai profile (dev/main pushes)
225122HF_TOKEN # Optional (speaker recognition)
226123```
227124
@@ -256,7 +153,7 @@ Workflows validate secrets before running tests:
256153- **Monthly**: Potentially hundreds of runs
257154- **Savings**: Significant with external contributors
258155
259- **Full Tests ** (`full-tests-with-api.yml`, `pr-tests-with-api.yml` ):
156+ **Real-provider runs ** (`deepgram-openai` profile, dev/main pushes ):
260157- **Transcription**: ~$0.10-0.30 per run (Deepgram)
261158- **LLM**: ~$0.05-0.15 per run (OpenAI)
262159- **Total**: ~$0.15-0.45 per run
@@ -365,14 +262,22 @@ runs-on: ubuntu-latest
365262- Look for timing issues (increase timeouts)
366263- Check Docker resource limits in CI
367264
368- # ## Label Workflow Not Running
265+ # ## Want a real-provider run on a PR
369266
370- **Problem**: Added label but workflow doesn't trigger
371- **Solutions**:
372- - Verify label name is exactly `test-with-api-keys`
373- - Check workflow trigger includes `types : [labeled]`
374- - Try removing and re-adding label
375- - Push new commit to trigger synchronize event
267+ There is no label for this, and there cannot usefully be one : a `pull_request`
268+ run from a fork cannot read repository secrets. Instead :
269+
270+ ` ` ` bash
271+ gh workflow run robot-tests.yml --ref <branch> -f profile=deepgram-openai
272+ ` ` `
273+
274+ Or merge to `dev`, where the `deepgram-openai` profile runs automatically.
275+
276+ # ## A profile fails immediately with a credential error
277+
278+ That is the harness refusing to start a stack it cannot configure. The message
279+ names the missing variable and how to set it. Run `make test PROFILE=mock` if
280+ you do not need real providers.
376281
377282# # Maintenance
378283
0 commit comments