Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
4e149b3
feat(e2e): add org pool for parallel test runs
ralphbean May 19, 2026
2e20e80
docs: add ADR 0039 for org pool parallel e2e tests
ralphbean May 19, 2026
f5ca637
fix(e2e): harden org pool locking and add acquireOrg tests
ralphbean May 19, 2026
a4e1257
fix(e2e): address review findings for org pool
ralphbean May 19, 2026
8ab00a0
feat(e2e): use shared mint and fullsend-ai app set
ralphbean May 20, 2026
cff31bc
feat(e2e): replace programmatic install/uninstall with CLI subprocess
ralphbean May 20, 2026
806eca8
ci(e2e): add E2E_GCP_PROJECT_ID secret to workflow
ralphbean May 20, 2026
567b1ed
ci(e2e): increase job timeout to 30 minutes
ralphbean May 20, 2026
c924c10
chore(e2e): restrict org pool to halfsend-01 (only enrolled org)
ralphbean May 20, 2026
e99cc66
fix(e2e): set CLI subprocess working directory to module root
ralphbean May 20, 2026
af6d689
fix(e2e): assert roles instead of agents in config.yaml
ralphbean May 20, 2026
d6a44ee
fix(e2e): update defaultRoles to match current defaults
ralphbean May 20, 2026
574b5e8
feat(e2e): add setup script for new e2e pool orgs
ralphbean May 20, 2026
4ce4bb5
fix(e2e): expand workflow trigger paths and add halfsend-02 to org pool
ralphbean May 20, 2026
ea62202
fix(e2e): increase default lock timeout to 20 minutes
ralphbean May 20, 2026
d693556
feat(e2e): add keyless GCP auth via Workload Identity Federation
ralphbean May 20, 2026
9a6d507
fix(e2e): move service account to secret reference
ralphbean May 21, 2026
4e2968e
fix(e2e): address review findings for org pool
ralphbean May 21, 2026
74963c0
feat(e2e): download triage run artifacts on failure
ralphbean May 21, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 33 additions & 8 deletions .github/workflows/e2e.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,24 +2,40 @@ name: E2E Tests

permissions:
contents: read
id-token: write

on:
push:
branches: [main]
paths: ['**/*.go', 'go.mod', 'go.sum', 'e2e/**']
paths:
- '**/*.go'
- 'go.mod'
- 'go.sum'
- 'e2e/**'
- 'internal/scaffold/fullsend-repo/**'
- 'internal/security/hooks/**'
- 'internal/dispatch/gcf/mintsrc/**'
- 'internal/sentencetoken/english.json'
- 'Makefile'
- '.github/workflows/e2e.yml'
pull_request:
paths: ['**/*.go', 'go.mod', 'go.sum', 'e2e/**']
paths:
- '**/*.go'
- 'go.mod'
- 'go.sum'
- 'e2e/**'
- 'internal/scaffold/fullsend-repo/**'
- 'internal/security/hooks/**'
- 'internal/dispatch/gcf/mintsrc/**'
- 'internal/sentencetoken/english.json'
- 'Makefile'
- '.github/workflows/e2e.yml'
workflow_dispatch:

concurrency:
group: e2e-halfsend
cancel-in-progress: false
queue: max

jobs:
e2e:
runs-on: ubuntu-latest
timeout-minutes: 15
timeout-minutes: 30
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[LOW] No workflow concurrency control after removing the global group

With only 2 active orgs, 3+ concurrent workflow runs will waste CI resources waiting for locks (up to 20 min each). Consider adding a concurrency group matching the pool size:

concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true

This cancels superseded commits on the same branch/PR while still allowing parallel runs across different PRs.

Flagged by 4/9 review agents

steps:
- uses: actions/checkout@v4

Expand Down Expand Up @@ -51,13 +67,22 @@ jobs:
env:
E2E_GITHUB_SESSION_B64: ${{ secrets.E2E_GITHUB_SESSION }}

- name: Authenticate to GCP
if: steps.secrets-check.outputs.available == 'true'
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ secrets.E2E_GCP_WIF_PROVIDER }}
service_account: ${{ secrets.E2E_GCP_SERVICE_ACCOUNT }}

- name: Run e2e tests
if: steps.secrets-check.outputs.available == 'true'
run: make e2e-test
env:
E2E_SCREENSHOT_DIR: ${{ runner.temp }}/e2e-screenshots
E2E_GITHUB_PASSWORD: ${{ secrets.E2E_GITHUB_PASSWORD }}
E2E_GITHUB_TOTP_SECRET: ${{ secrets.E2E_GITHUB_TOTP_SECRET }}
E2E_MINT_URL: ${{ secrets.E2E_MINT_URL }}
E2E_GCP_PROJECT_ID: ${{ secrets.E2E_GCP_PROJECT_ID }}

- name: Upload debug screenshots
if: always() && steps.secrets-check.outputs.available == 'true'
Expand Down
71 changes: 71 additions & 0 deletions docs/ADRs/0040-org-pool-for-parallel-e2e-tests.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
---
title: "40. Org pool for parallel e2e tests"
status: Accepted
relates_to:
- testing-agents
topics:
- e2e
- ci
- parallelism
---

# 40. Org pool for parallel e2e tests

Date: 2026-05-19

## Status

Accepted

## Context

The e2e tests exercise the full admin install/uninstall flow against a live
GitHub org using Playwright browser automation
([ADR 0010](0010-stored-session-for-e2e-browser-auth.md)). Each run creates
GitHub Apps, repos, secrets, variables, and enrollment PRs — then tears them
all down. These operations are destructive and non-reentrant: two concurrent
runs targeting the same org will collide on shared resources (`.fullsend` repo,
org secrets, app slugs) and fail unpredictably.

With a single test org, CI runs are serialized. A push to `main` and an
in-flight PR both trigger e2e, but only one can proceed; the other waits or
fails. As the contributor count grows this becomes a bottleneck.

## Decision

Maintain a pool of identically-configured GitHub orgs (currently `halfsend-01`
through `halfsend-06`). Each e2e run acquires exclusive access to one org
before proceeding, using a lightweight distributed lock implemented as a
purpose-built repo (`e2e-lock`) within each org.

**Acquisition:** The test runner scans the pool in order, attempting to create
the `e2e-lock` repo in each org. Repo creation is atomic on GitHub — if it
succeeds, the caller holds the lock. A `README.md` in the lock repo contains
the run's UUID for ownership verification. If all orgs are locked, the runner
falls back to polling with a configurable timeout (`E2E_LOCK_TIMEOUT`,
default 2 minutes).

**Release:** On test completion (pass or fail), the runner deletes the
`e2e-lock` repo, but only after verifying the UUID matches — preventing a
run from releasing another run's lock.

**Staleness:** If a runner crashes without releasing its lock, the lock repo's
`created_at` timestamp provides an age signal. A fresh lock (under 1 minute
old) resets the wait timer; an old lock is assumed stale and can be
force-acquired by a subsequent run.

Adding new orgs to the pool requires only provisioning the GitHub org (with
the shared test account as owner) and appending its name to the `orgPool`
slice in the test code. No architectural changes are needed.

## Consequences

- Up to N e2e runs execute in parallel, where N is the pool size.
- Each org must be pre-provisioned with the test account as owner and a
`test-repo` for enrollment testing.
- A crashed run leaves a stale lock that self-heals via the age-based
staleness check.
- The single `botsend` test account and its stored browser session are shared
across all orgs; session export and PAT creation remain per-run.
- Pool expansion is an operational task (provision org, update one slice
literal), not an architectural change.
Loading
Loading