-
Notifications
You must be signed in to change notification settings - Fork 10
Add e2e workflow for [QE] story test automation #28
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 1 commit
Commits
Show all changes
5 commits
Select commit
Hold shift + click to select a range
b8e2cc0
Add e2e workflow and fix shared issues in implement workflow
adalton f72730a
Address PR review feedback: fix phase reference and add feature-defec…
adalton 2c4783e
Make e2e workflow framework-agnostic
adalton e9542bb
Add parallelism discovery, lifecycle skeletons, and anti-pattern exam…
adalton cbed24a
Address PR review feedback: e2e scope clarity and clean-tree checks
adalton File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,158 @@ | ||
| # E2E Test Workflow | ||
|
|
||
| A story-to-tests workflow that takes a Jira [QE] Story, discovers the project's e2e testing infrastructure, plans test scenarios mapped to acceptance criteria, writes e2e test code following the project's patterns, validates against anti-patterns and scenario coverage, and manages review via GitHub PRs. | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| | Tool | Required | Purpose | | ||
| |------|----------|---------| | ||
| | Jira access (MCP or CLI) | For `/ingest` | Fetch [QE] Story issue details | | ||
| | GitHub CLI (`gh`) | For `/publish`, `/respond` | Create PRs, post review comments | | ||
| | Git | Yes | Branch management, commits | | ||
| | Project e2e test tooling | Yes | Discovered during `/ingest` from project's AGENTS.md, Makefile, CI workflows | | ||
| | Docs repo (local clone) | For `/ingest` | Read PRD and design document for upstream context | | ||
|
|
||
| ## Phases | ||
|
|
||
| | Phase | Command | Purpose | Artifact(s) | | ||
| |-------|---------|---------|-------------| | ||
| | Ingest | `/ingest` | Fetch [QE] story, verify [DEV] dependencies, explore e2e infrastructure | `01-context.md` | | ||
| | Plan | `/plan` | Map ACs to test scenarios, select reference suite | `02-plan.md` | | ||
| | Revise | `/revise` | Incorporate feedback into the test plan | Updated `02-plan.md` | | ||
| | Code | `/code` | Write e2e test code following discovered patterns | `03-test-report.md`, `04-impl-report.md` | | ||
| | Validate | `/validate` | Run tests, check anti-patterns, verify scenario coverage | `05-validation-report.md` | | ||
| | Publish | `/publish` | Push branch, create draft PR | `06-pr-description.md` | | ||
| | Respond | `/respond` | Address reviewer comments | `07-review-responses.md` | | ||
|
|
||
| ## Typical Flow | ||
|
|
||
| ```text | ||
| /ingest EDM-5678 | ||
| -> fetches [QE] story from Jira | ||
| -> verifies [DEV] dependencies are merged | ||
| -> loads design document and PRD context | ||
| -> explores e2e test infrastructure (framework, harness, patterns) | ||
| -> selects reference suite as pattern source | ||
| -> discovers validation profile (test execution, lint commands) | ||
| -> writes .artifacts/e2e/EDM-5678/01-context.md | ||
|
|
||
| /plan | ||
| -> maps each acceptance criterion to test scenarios | ||
| -> selects reference suite and documents patterns to follow | ||
| -> designs test file structure (suite file + test files) | ||
| -> plans harness method usage and auxiliary services | ||
| -> breaks work into ordered tasks (suite file first, then scenarios) | ||
| -> writes 02-plan.md | ||
|
|
||
| /revise (optional, repeatable) | ||
| -> user reviews plan, requests changes | ||
| -> plan updated, consistency maintained | ||
|
|
||
| /code | ||
| -> creates feature branch | ||
| -> for each task: read reference -> write test code -> run tests -> review -> commit | ||
| -> updates 02-plan.md with task completion status | ||
| -> writes 03-test-report.md, 04-impl-report.md | ||
|
|
||
| /validate | ||
| -> runs e2e tests (scoped to new suite) | ||
| -> checks for anti-patterns (hardcoded sleeps, missing cleanup, etc.) | ||
| -> verifies every AC has a passing test scenario | ||
| -> checks for regressions in adjacent suites | ||
| -> writes 05-validation-report.md | ||
|
|
||
| /publish | ||
| -> pushes feature branch | ||
| -> creates draft GitHub PR with Jira link | ||
| -> writes 06-pr-description.md | ||
|
|
||
| /respond (repeatable) | ||
| -> fetches PR review comments | ||
| -> proposes responses (user approves before posting) | ||
| -> applies code changes if needed | ||
| -> writes 07-review-responses.md | ||
| ``` | ||
|
|
||
| ## Artifacts | ||
|
|
||
| All artifacts are stored in `.artifacts/e2e/{jira-key}/`. | ||
|
|
||
| ```text | ||
| .artifacts/e2e/EDM-5678/ | ||
| 01-context.md (story context, e2e infrastructure, validation profile) | ||
| 02-plan.md (scenario breakdown, AC coverage -- updated as tasks complete) | ||
| 03-test-report.md (tests written, harness methods used) | ||
| 04-impl-report.md (changes, commits, deviations, discoveries) | ||
| 05-validation-report.md (check results, anti-patterns, regressions) | ||
| 06-pr-description.md (PR body) | ||
| 07-review-responses.md (review comment log) | ||
| publish-metadata.json (PR number, branch, URL) | ||
| ``` | ||
|
|
||
| ## Key Design Decisions | ||
|
|
||
| ### Discovery-Based Infrastructure | ||
|
|
||
| The workflow does not hardcode language-specific commands or framework assumptions. During `/ingest`, it discovers the project's e2e testing framework, harness, auxiliary services, execution commands, and conventions. This makes the workflow portable across projects using different testing stacks (Ginkgo, Playwright, pytest, Cypress, etc.). | ||
|
|
||
| ### Reference Suite Pattern | ||
|
|
||
| Before writing any test code, the workflow identifies the most similar existing e2e test suite in the project and extracts its patterns: imports, setup/teardown, harness usage, assertion style, labels, and cleanup. New tests follow these patterns exactly, ensuring consistency with the project's existing test base. | ||
|
|
||
| ### Scenario-Driven Planning | ||
|
|
||
| Unlike implementation planning (task-driven), e2e test planning is scenario-driven. Each acceptance criterion maps to one or more concrete test scenarios with specific Describe/Context/It nesting, steps, assertions, and labels. This ensures every AC is verifiably covered. | ||
|
|
||
| ### Anti-Pattern Detection | ||
|
|
||
| Validation checks for 10 common e2e test anti-patterns: hardcoded sleeps, brittle selectors, order-dependent tests, shared mutable state, missing cleanup, harness bypass, missing labels, hardcoded values, missing async polling, and missing failure diagnostics. Each detected anti-pattern is fixed during validation. | ||
|
|
||
| ### Feature Defects Are Not Test Bugs | ||
|
|
||
| If e2e tests reveal that the feature behaves differently than the acceptance criteria describe, that is a defect in the [DEV] implementation, not a test failure. The test is adjusted (xfail/skip) and the defect is noted in the implementation report. The e2e workflow does not fix feature code. | ||
|
|
||
| ### Incremental Commits | ||
|
|
||
| Each logical unit of work gets its own commit, following the project's commit format (discovered during `/ingest`). Each commit should be independently meaningful. | ||
|
|
||
| ### Plan as Living Document | ||
|
|
||
| `02-plan.md` is updated during `/code` as tasks are completed. On re-invocation (e.g., after context limits or interruptions), the plan shows which tasks are done and which remain. | ||
|
|
||
| ## Directory Structure | ||
|
|
||
| ```text | ||
| e2e/ | ||
| ├── SKILL.md # Workflow entry point | ||
| ├── guidelines.md # Behavioral rules and guardrails | ||
| ├── README.md # This file | ||
| ├── skills/ | ||
| │ ├── controller.md # Phase dispatcher and transitions | ||
| │ ├── ingest.md # Fetch story, explore e2e infrastructure | ||
| │ ├── plan.md # Map ACs to test scenarios | ||
| │ ├── revise.md # Incorporate plan feedback | ||
| │ ├── code.md # Write e2e test code | ||
| │ ├── validate.md # Run tests, check anti-patterns | ||
| │ ├── publish.md # Create GitHub PR | ||
| │ └── respond.md # Address review comments | ||
| └── commands/ | ||
| ├── ingest.md # /ingest command | ||
| ├── plan.md # /plan command | ||
| ├── revise.md # /revise command | ||
| ├── code.md # /code command | ||
| ├── validate.md # /validate command | ||
| ├── publish.md # /publish command | ||
| └── respond.md # /respond command | ||
| ``` | ||
|
|
||
| ## Getting Started | ||
|
|
||
| ```bash | ||
| # Install the workflow | ||
| ./install.sh claude --workflows e2e | ||
|
|
||
| # Or install all workflows | ||
| ./install.sh all | ||
| ``` | ||
|
|
||
| Then in your project, run the `e2e` workflow's `ingest` command for your [QE] Jira story (e.g., EDM-5678). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,25 @@ | ||
| --- | ||
| name: e2e | ||
| description: >- | ||
| Story-to-e2e-test workflow that takes a Jira [QE] Story, discovers the | ||
| project's e2e testing infrastructure, plans test scenarios, writes e2e | ||
| tests matching project conventions, validates them, and manages review | ||
| via GitHub PRs. Use when implementing [QE] stories produced by the | ||
| design workflow. | ||
| Activated by commands: /ingest, /plan, /revise, /code, /validate, /publish, /respond. | ||
| --- | ||
| # E2E Test Workflow Orchestrator | ||
|
|
||
| ## Quick Start | ||
|
|
||
| 1. If the user invoked a specific command (e.g., `/plan`, `/code`), read | ||
| `commands/{command}.md` and follow it. | ||
| 2. Otherwise, read `skills/controller.md` to load the workflow controller: | ||
| - If the user provided a Jira issue key or URL, execute the `/ingest` phase | ||
| - Otherwise, execute the first phase the user requests | ||
|
|
||
| If a step fails or produces unexpected output (e.g., Jira MCP errors, test | ||
| failures, build errors), stop and report the error to the user. Do not | ||
| advance to the next phase. Offer to retry the failed step or escalate. | ||
|
|
||
| For principles, hard limits, safety, quality, and escalation rules, see `guidelines.md`. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,11 @@ | ||
| --- | ||
| name: e2e:code | ||
| description: "Write e2e test code following discovered patterns, committing incrementally" | ||
| --- | ||
| # /code | ||
|
|
||
| Read `../skills/controller.md` and follow it. | ||
|
|
||
| Dispatch the **code** phase. Context: | ||
|
|
||
| $ARGUMENTS |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,11 @@ | ||
| --- | ||
| name: e2e:ingest | ||
| description: "Fetch [QE] story, verify dependencies, explore e2e infrastructure, build test-execution profile" | ||
| --- | ||
| # /ingest | ||
|
|
||
| Read `../skills/controller.md` and follow it. | ||
|
|
||
| Dispatch the **ingest** phase. Context: | ||
|
|
||
| $ARGUMENTS |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,11 @@ | ||
| --- | ||
| name: e2e:plan | ||
| description: "Map acceptance criteria to e2e test scenarios, select reference suite, design test structure" | ||
| --- | ||
| # /plan | ||
|
|
||
| Read `../skills/controller.md` and follow it. | ||
|
|
||
| Dispatch the **plan** phase. Context: | ||
|
|
||
| $ARGUMENTS |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,11 @@ | ||
| --- | ||
| name: e2e:publish | ||
| description: "Push feature branch and create draft PR for e2e tests" | ||
| --- | ||
| # /publish | ||
|
|
||
| Read `../skills/controller.md` and follow it. | ||
|
|
||
| Dispatch the **publish** phase. Context: | ||
|
|
||
| $ARGUMENTS |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,11 @@ | ||
| --- | ||
| name: e2e:respond | ||
| description: "Fetch and address PR reviewer comments on e2e test code" | ||
| --- | ||
| # /respond | ||
|
|
||
| Read `../skills/controller.md` and follow it. | ||
|
|
||
| Dispatch the **respond** phase. Context: | ||
|
|
||
| $ARGUMENTS |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,11 @@ | ||
| --- | ||
| name: e2e:revise | ||
| description: "Incorporate user feedback into the e2e test plan" | ||
| --- | ||
| # /revise | ||
|
|
||
| Read `../skills/controller.md` and follow it. | ||
|
|
||
| Dispatch the **revise** phase. Context: | ||
|
|
||
| $ARGUMENTS |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,11 @@ | ||
| --- | ||
| name: e2e:validate | ||
| description: "Run e2e tests, check for anti-patterns, verify scenario coverage, assess PR readiness" | ||
| --- | ||
| # /validate | ||
|
|
||
| Read `../skills/controller.md` and follow it. | ||
|
|
||
| Dispatch the **validate** phase. Context: | ||
|
|
||
| $ARGUMENTS |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.