Breeze: make breeze down discover and stop every compose project#66311
Merged
potiuk merged 1 commit intoapache:mainfrom May 3, 2026
Merged
Conversation
Discovers running docker compose projects via the `com.docker.compose.project` label and brings down every one matching a known breeze prefix (`breeze`, `prek`, `docker-compose`, `docs`, `db`, `providers`, plus `breeze-`, `airflow-test`, `constraints-`, `providers-`). One `breeze down` now leaves the host clean regardless of which breeze commands, prek hooks, or CI steps were running. Two new flags: `--all-projects` also catches compose projects that don't match any known breeze prefix (off by default to avoid wiping unrelated host projects); `--project-name <name>` restricts cleanup to a single project (useful in CI). Single-use cleanup gaps fixed: - `run_command_via_breeze_shell` (used by every breeze-shell-based prek hook) now wraps the subprocess in try/finally and passes `--volumes` to the down call, so prek runs no longer leak DB volumes between hooks and KeyboardInterrupt no longer skips cleanup. - `breeze registry extract-data` and `breeze registry backfill-data` added try/finally cleanup of their unique compose projects (previously leaked `breeze-registry-*` and `breeze-backfill-*` on every invocation). - `breeze doctor` now uses the same label-based cleanup.
6b82501 to
413ea6d
Compare
Contributor
Backport failed to create: v3-2-test. View the failure log Run detailsNote: As of Merging PRs targeted for Airflow 3.X In matter of doubt please ask in #release-management Slack channel.
You can attempt to backport this manually by running: cherry_picker 2635495 v3-2-testThis should apply the commit to the v3-2-test branch and leave the commit in conflict state marking After you have resolved the conflicts, you can continue the backport process by running: cherry_picker --continueIf you don't have cherry-picker installed, see the installation guide. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
breeze downonly ever stopped the defaultbreezecompose project — everyother project name (
prek,docs,db,airflow-test*,constraints-*,providers*,breeze-registry-*,breeze-backfill-*,*-run-*, the legacydocker-compose) was left running, so the host kept accumulating containersacross breeze invocations. CI workarounds papered over it by calling
breeze downrepeatedly between steps and hoping the env-var-overridden project namematched.
This change makes
breeze downdiscover every running docker composeproject on the host via the
com.docker.compose.projectlabel and bring downeach one whose name matches a known breeze prefix, in a single call. The
allowlist is centralized in
KNOWN_DOCKER_COMPOSE_PROJECT_NAMES/KNOWN_DOCKER_COMPOSE_PROJECT_PREFIXESinglobal_constants.pyso adding anew project_name pattern in any breeze command, prek hook, or CI step is a
one-line update with a comment that says so.
Two new flags:
--all-projects— also bring down compose projects whose names don't matchany known breeze prefix. Off by default to avoid wiping unrelated host
projects.
--project-name <name>— restrict cleanup to a single project, skippingdiscovery. Useful in CI steps that want to bring exactly one project down.
Single-use cleanup gaps fixed
scripts/ci/prek/common_prek_utils.py::run_command_via_breeze_shell(usedby every breeze-shell-based prek hook) now wraps the subprocess in
try/finallyand passes--volumesto its down call. Previously aKeyboardInterruptmid-hook skipped cleanup, and even successful runsleaked DB volumes between hooks.
breeze registry extract-dataandbreeze registry backfill-dataaddedtry/finallycleanup of their uniquebreeze-registry-*/breeze-backfill-*compose projects (previously leaked one perinvocation).
breeze doctornow uses the same label-based cleanup so it healsevery project, not just
breeze.Test plan
dev/breeze/tests/test_docker_command_utils.pycovering project-name matching, label parsing, the discovery → filter →
bring-down flow,
--all-projects,--project-name <name>, and--preserve-volumes. All 38 tests in the file pass.breeze down --helpandbreeze setup regenerate-command-imagesre-run; SVG / hash file regenerated and committed.
prek run ruff/ruff-formatclean on every changed file.Project selectiongroup in
--help.breeze downbetween steps; it should now catch every compose project regardless of
whether
COMPOSE_PROJECT_NAMEwas honored on the correspondingbreeze shellcall.closes: (no tracked issue — internal cleanup)
Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Opus 4.7) following the guidelines