Skip to content

Breeze: make breeze down discover and stop every compose project#66311

Merged
potiuk merged 1 commit intoapache:mainfrom
potiuk:breeze-down-discover-all-compose-projects
May 3, 2026
Merged

Breeze: make breeze down discover and stop every compose project#66311
potiuk merged 1 commit intoapache:mainfrom
potiuk:breeze-down-discover-all-compose-projects

Conversation

@potiuk
Copy link
Copy Markdown
Member

@potiuk potiuk commented May 3, 2026

Summary

breeze down only ever stopped the default breeze compose project — every
other project name (prek, docs, db, airflow-test*, constraints-*,
providers*, breeze-registry-*, breeze-backfill-*, *-run-*, the legacy
docker-compose) was left running, so the host kept accumulating containers
across breeze invocations. CI workarounds papered over it by calling breeze down repeatedly between steps and hoping the env-var-overridden project name
matched.

This change makes breeze down discover every running docker compose
project on the host via the com.docker.compose.project label and bring down
each one whose name matches a known breeze prefix, in a single call. The
allowlist is centralized in KNOWN_DOCKER_COMPOSE_PROJECT_NAMES /
KNOWN_DOCKER_COMPOSE_PROJECT_PREFIXES in global_constants.py so adding a
new project_name pattern in any breeze command, prek hook, or CI step is a
one-line update with a comment that says so.

Two new flags:

  • --all-projects — also bring down compose projects whose names don't match
    any known breeze prefix. Off by default to avoid wiping unrelated host
    projects.
  • --project-name <name> — restrict cleanup to a single project, skipping
    discovery. Useful in CI steps that want to bring exactly one project down.

Single-use cleanup gaps fixed

  • scripts/ci/prek/common_prek_utils.py::run_command_via_breeze_shell (used
    by every breeze-shell-based prek hook) now wraps the subprocess in
    try/finally and passes --volumes to its down call. Previously a
    KeyboardInterrupt mid-hook skipped cleanup, and even successful runs
    leaked DB volumes between hooks.
  • breeze registry extract-data and breeze registry backfill-data added
    try/finally cleanup of their unique breeze-registry-* /
    breeze-backfill-* compose projects (previously leaked one per
    invocation).
  • breeze doctor now uses the same label-based cleanup so it heals
    every project, not just breeze.

Test plan

  • 16 new unit tests in dev/breeze/tests/test_docker_command_utils.py
    covering project-name matching, label parsing, the discovery → filter →
    bring-down flow, --all-projects, --project-name <name>, and
    --preserve-volumes. All 38 tests in the file pass.
  • breeze down --help and breeze setup regenerate-command-images
    re-run; SVG / hash file regenerated and committed.
  • prek run ruff / ruff-format clean on every changed file.
  • Manually verified the new flags appear in the Project selection
    group in --help.
  • Smoke-test in CI: the migration_tests action calls breeze down
    between steps; it should now catch every compose project regardless of
    whether COMPOSE_PROJECT_NAME was honored on the corresponding breeze shell call.

closes: (no tracked issue — internal cleanup)


Was generative AI tooling used to co-author this PR?
  • Yes — Claude Code (Opus 4.7)

Generated-by: Claude Code (Opus 4.7) following the guidelines

Copy link
Copy Markdown
Contributor

@jscheffl jscheffl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool!

Discovers running docker compose projects via the
`com.docker.compose.project` label and brings down every one matching
a known breeze prefix (`breeze`, `prek`, `docker-compose`, `docs`, `db`,
`providers`, plus `breeze-`, `airflow-test`, `constraints-`, `providers-`).
One `breeze down` now leaves the host clean regardless of which breeze
commands, prek hooks, or CI steps were running.

Two new flags: `--all-projects` also catches compose projects that don't
match any known breeze prefix (off by default to avoid wiping unrelated
host projects); `--project-name <name>` restricts cleanup to a single
project (useful in CI).

Single-use cleanup gaps fixed:
- `run_command_via_breeze_shell` (used by every breeze-shell-based prek
  hook) now wraps the subprocess in try/finally and passes `--volumes`
  to the down call, so prek runs no longer leak DB volumes between hooks
  and KeyboardInterrupt no longer skips cleanup.
- `breeze registry extract-data` and `breeze registry backfill-data`
  added try/finally cleanup of their unique compose projects (previously
  leaked `breeze-registry-*` and `breeze-backfill-*` on every invocation).
- `breeze doctor` now uses the same label-based cleanup.
@potiuk potiuk force-pushed the breeze-down-discover-all-compose-projects branch from 6b82501 to 413ea6d Compare May 3, 2026 17:03
@potiuk potiuk merged commit 2635495 into apache:main May 3, 2026
8 checks passed
@potiuk potiuk deleted the breeze-down-discover-all-compose-projects branch May 3, 2026 17:04
@github-project-automation github-project-automation Bot moved this from Backlog to Done in Airflow Registry May 3, 2026
@potiuk potiuk added this to the Airflow 3.2.2 milestone May 3, 2026
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 3, 2026

Backport failed to create: v3-2-test. View the failure log Run details

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status Branch Result
v3-2-test Commit Link

You can attempt to backport this manually by running:

cherry_picker 2635495 v3-2-test

This should apply the commit to the v3-2-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.

After you have resolved the conflicts, you can continue the backport process by running:

cherry_picker --continue

If you don't have cherry-picker installed, see the installation guide.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:dev-tools area:registry backport-to-v3-2-test Mark PR with this label to backport to v3-2-test branch

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

2 participants