Skip to content

Lazy-load Alembic in db_manager imports#65655

Merged
potiuk merged 2 commits into
apache:mainfrom
hkc-8010:fix/reduce-alembic-import-noise
May 10, 2026
Merged

Lazy-load Alembic in db_manager imports#65655
potiuk merged 2 commits into
apache:mainfrom
hkc-8010:fix/reduce-alembic-import-noise

Conversation

@hkc-8010
Copy link
Copy Markdown
Contributor

@hkc-8010 hkc-8010 commented Apr 22, 2026

What this PR does

Avoids eagerly importing Alembic when airflow.utils.db / airflow.utils.db_manager are imported.

Today, airflow.utils.db_manager imports from alembic import command at module import time. Because airflow.utils.db imports RunDBManager, that path loads Alembic during a normal import and emits the noisy alembic.runtime.plugins INFO lines like:

setup plugin alembic.autogenerate.schemas
setup plugin alembic.autogenerate.tables
setup plugin alembic.autogenerate.types
setup plugin alembic.autogenerate.constraints
setup plugin alembic.autogenerate.defaults
setup plugin alembic.autogenerate.comments

This PR defers importing alembic.command until the DB migration operations actually need it.

Why

This keeps normal Airflow imports quiet without suppressing logs globally and without changing Alembic behavior during real migration operations.

To preserve existing test and patching behavior, airflow.utils.db_manager.command remains available via a lazy proxy.

Related issue

Closes #65652

Testing

  • ruff check airflow-core/src/airflow/utils/db_manager.py airflow-core/tests/unit/utils/test_db_manager.py
  • Breeze Python 3.10 container:
    • python -m pytest airflow-core/tests/unit/utils/test_db_manager.py -q
    • python -m pytest providers/edge3/tests/unit/edge3/models/test_db.py -q
  • Verified the original repro is clean in Breeze:
    • import airflow.utils.db_manager
    • import airflow.utils.db

Gen-AI disclosure

This PR was prepared with assistance from generative AI tools. I reviewed the generated changes, validated the behavior locally, and take responsibility for the final code and testing.

@hkc-8010
Copy link
Copy Markdown
Contributor Author

A bit of extra validation context from local repro/testing:

  • I reproduced the noisy alembic.runtime.plugins output in OSS Airflow 3.2.0, not just in Astro Runtime.
  • In OSS, plain import airflow stayed clean in my tests, but these import paths emitted the setup plugin alembic.autogenerate.* lines:
    • import airflow.utils.db
    • import airflow.utils.db_manager
    • from airflow.utils.db_manager import RunDBManager
  • I validated the fix in Breeze with Python 3.13 + Postgres:
    • python -m pytest tests/unit/utils/test_db_manager.py -q -> 15 passed
    • import airflow.utils.db_manager -> clean
    • import airflow.utils.db -> clean

So the issue appears to be the eager Alembic import on the DB manager import path, and the lazy import in this PR removes the noise without changing the actual migration call sites.

@potiuk potiuk added ready for maintainer review Set after triaging when all criteria pass. and removed ready for maintainer review Set after triaging when all criteria pass. labels Apr 22, 2026
@potiuk potiuk marked this pull request as draft April 23, 2026 01:09
@potiuk
Copy link
Copy Markdown
Member

potiuk commented Apr 23, 2026

@hkc-8010 Converting to draft — this PR doesn't yet meet our Pull Request quality criteria.

  • Failing CI: Special tests / Pendulum2 test: core / All-core:Pendulum2-Postgres:14:3.10:Core...Serialization, Special tests / Latest Boto test: providers / All-prov:LatestBoto-Postgres:14:3.10:-amazon,celer...standard, Special tests / Pendulum2 test: providers / All-prov:Pendulum2-Postgres:14:3.10:-amazon,celer...standard (+9 more). Please investigate and fix.

See the linked criteria for how to fix each item, then mark the PR "Ready for review". This is not a rejection — just an invitation to bring the PR up to standard. No rush.


Note: This comment was drafted by an AI-assisted triage tool and may contain mistakes. Once you have addressed the points above, an Apache Airflow maintainer — a real person — will take the next look at your PR. We use this two-stage triage process so that our maintainers' limited time is spent where it matters most: the conversation with you.

@hkc-8010
Copy link
Copy Markdown
Contributor Author

hkc-8010 commented May 2, 2026

Follow-up after investigating the draft conversion / CI failures:

  • The core failures in Pendulum2, LowestDeps, and the regular DB matrices were all the same two failures in airflow-core/tests/unit/utils/test_db_manager.py. The root cause was the new regression test mutating sys.modules in-process and only partially restoring state.
  • The providers failure (providers/edge3/tests/unit/edge3/models/test_db.py::TestEdgeDBManager::test_create_db_from_orm) was a separate compatibility issue: existing tests patch airflow.utils.db_manager.command, and my first lazy-import version removed that module attribute.

This follow-up push addresses both:

  • replaces the sys.modules mutation with a subprocess-based import assertion, so the regression test is hermetic
  • keeps airflow.utils.db_manager.command available through a lazy proxy, so patch-based tests keep working without reintroducing eager Alembic imports

Local validation on Python 3.10 in Breeze:

  • python -m pytest airflow-core/tests/unit/utils/test_db_manager.py -q
  • python -m pytest providers/edge3/tests/unit/edge3/models/test_db.py -q
  • import smoke check for airflow.utils.db_manager and airflow.utils.db stays clean (no Alembic plugin noise)

@hkc-8010 hkc-8010 marked this pull request as ready for review May 2, 2026 20:43
@hkc-8010
Copy link
Copy Markdown
Contributor Author

hkc-8010 commented May 7, 2026

@potiuk Following up on this one when you have a moment. The CI-related issues from your earlier draft-conversion note were investigated and addressed in the May 2 follow-up, and the PR is ready for another pass. I’d really appreciate a quick re-review when you get a chance. Thank you!

@potiuk potiuk force-pushed the fix/reduce-alembic-import-noise branch from 1e26d47 to 4de238c Compare May 10, 2026 03:22
@potiuk potiuk merged commit 27a539f into apache:main May 10, 2026
142 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Importing airflow.utils.db eagerly imports Alembic and emits noisy alembic.runtime.plugins INFO logs

2 participants