cli airflow db without dags and plugings

[raphaelauv]

Description

Currently If in my conf I have a xcom_backend or/and a secret_backend I can't run the cli airflow db without giving access to the dags and plugins

version: '3.8'

services:
  airflow_init_db:
    image: airflow_v2
    env_file:
      - ./conf/airflow.env
    volumes:
      # Link for custom_xcom_backend and custom_secret_backend
      - $AIRFLOW_DAGS_SRC_PATH/utils/backends/:/usr/local/airflow/dags/utils/backends/
    command: airflow db upgrade

without the volume mapping it fail with

[2022-06-06 17:50:03,697] {configuration.py:484} ERROR - No module named 'utils'
2022-06-06T17:50:03.705575364Z Traceback (most recent call last):
2022-06-06T17:50:03.705607597Z   File "/usr/local/lib/python3.8/site-packages/airflow/configuration.py", line 482, in getimport
2022-06-06T17:50:03.705616559Z     return import_string(full_qualified_path)
2022-06-06T17:50:03.705622858Z   File "/usr/local/lib/python3.8/site-packages/airflow/utils/module_loading.py", line 32, in import_string
2022-06-06T17:50:03.705628878Z     module = import_module(module_path)
2022-06-06T17:50:03.705634245Z   File "/usr/local/lib/python3.8/importlib/__init__.py", line 127, in import_module
2022-06-06T17:50:03.705640056Z     return _bootstrap._gcd_import(name[level:], package, level)
2022-06-06T17:50:03.705645481Z   File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
2022-06-06T17:50:03.705651321Z   File "<frozen importlib._bootstrap>", line 991, in _find_and_load
2022-06-06T17:50:03.705657152Z   File "<frozen importlib._bootstrap>", line 961, in _find_and_load_unlocked
2022-06-06T17:50:03.705662652Z   File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
2022-06-06T17:50:03.705668816Z   File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
2022-06-06T17:50:03.705674535Z   File "<frozen importlib._bootstrap>", line 991, in _find_and_load
2022-06-06T17:50:03.705680730Z   File "<frozen importlib._bootstrap>", line 961, in _find_and_load_unlocked
2022-06-06T17:50:03.705686719Z   File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
2022-06-06T17:50:03.705692431Z   File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
2022-06-06T17:50:03.705697930Z   File "<frozen importlib._bootstrap>", line 991, in _find_and_load
2022-06-06T17:50:03.705703442Z   File "<frozen importlib._bootstrap>", line 973, in _find_and_load_unlocked
2022-06-06T17:50:03.705708839Z ModuleNotFoundError: No module named 'utils'
2022-06-06T17:50:03.705713891Z 
2022-06-06T17:50:03.705718604Z During handling of the above exception, another exception occurred:
2022-06-06T17:50:03.705723909Z 
2022-06-06T17:50:03.705728911Z Traceback (most recent call last):
2022-06-06T17:50:03.705734002Z   File "/usr/local/bin/airflow", line 8, in <module>
2022-06-06T17:50:03.705739747Z     sys.exit(main())
2022-06-06T17:50:03.705744848Z   File "/usr/local/lib/python3.8/site-packages/airflow/__main__.py", line 48, in main
2022-06-06T17:50:03.705750643Z     args.func(args)
2022-06-06T17:50:03.705755795Z   File "/usr/local/lib/python3.8/site-packages/airflow/cli/cli_parser.py", line 47, in command
2022-06-06T17:50:03.705787110Z     func = import_string(import_path)
2022-06-06T17:50:03.705795008Z   File "/usr/local/lib/python3.8/site-packages/airflow/utils/module_loading.py", line 32, in import_string
2022-06-06T17:50:03.705801328Z     module = import_module(module_path)
2022-06-06T17:50:03.705806591Z   File "/usr/local/lib/python3.8/importlib/__init__.py", line 127, in import_module
2022-06-06T17:50:03.705812365Z     return _bootstrap._gcd_import(name[level:], package, level)
2022-06-06T17:50:03.705819554Z   File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
2022-06-06T17:50:03.705825591Z   File "<frozen importlib._bootstrap>", line 991, in _find_and_load
2022-06-06T17:50:03.705831388Z   File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
2022-06-06T17:50:03.705836771Z   File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
2022-06-06T17:50:03.705842707Z   File "<frozen importlib._bootstrap_external>", line 783, in exec_module
2022-06-06T17:50:03.705848644Z   File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
2022-06-06T17:50:03.705854751Z   File "/usr/local/lib/python3.8/site-packages/airflow/cli/commands/db_command.py", line 24, in <module>
2022-06-06T17:50:03.705861189Z     from airflow.utils import cli as cli_utils, db
2022-06-06T17:50:03.705866663Z   File "/usr/local/lib/python3.8/site-packages/airflow/utils/db.py", line 28, in <module>
2022-06-06T17:50:03.705872575Z     from airflow.jobs.base_job import BaseJob  # noqa: F401
2022-06-06T17:50:03.705878078Z   File "/usr/local/lib/python3.8/site-packages/airflow/jobs/__init__.py", line 19, in <module>
2022-06-06T17:50:03.705883975Z     import airflow.jobs.backfill_job
2022-06-06T17:50:03.705889540Z   File "/usr/local/lib/python3.8/site-packages/airflow/jobs/backfill_job.py", line 28, in <module>
2022-06-06T17:50:03.705895379Z     from airflow import models
2022-06-06T17:50:03.705900376Z   File "/usr/local/lib/python3.8/site-packages/airflow/models/__init__.py", line 20, in <module>
2022-06-06T17:50:03.705906053Z     from airflow.models.baseoperator import BaseOperator, BaseOperatorLink
2022-06-06T17:50:03.705911136Z   File "/usr/local/lib/python3.8/site-packages/airflow/models/baseoperator.py", line 61, in <module>
2022-06-06T17:50:03.705916510Z     from airflow.models.taskinstance import Context, TaskInstance, clear_task_instances
2022-06-06T17:50:03.705921856Z   File "/usr/local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 81, in <module>
2022-06-06T17:50:03.705927088Z     from airflow.models.xcom import XCOM_RETURN_KEY, XCom
2022-06-06T17:50:03.705932440Z   File "/usr/local/lib/python3.8/site-packages/airflow/models/xcom.py", line 496, in <module>
2022-06-06T17:50:03.705938338Z     XCom = resolve_xcom_backend()
2022-06-06T17:50:03.705959642Z   File "/usr/local/lib/python3.8/site-packages/airflow/models/xcom.py", line 483, in resolve_xcom_backend
2022-06-06T17:50:03.705966771Z     clazz = conf.getimport("core", "xcom_backend", fallback=f"airflow.models.xcom.{BaseXCom.__name__}")
2022-06-06T17:50:03.705972908Z   File "/usr/local/lib/python3.8/site-packages/airflow/configuration.py", line 485, in getimport
2022-06-06T17:50:03.705978934Z     raise AirflowConfigException(
2022-06-06T17:50:03.705985412Z airflow.exceptions.AirflowConfigException: The object could not be loaded. Please check "xcom_backend" key in "core" section. Current value: "utils.backends.custom_xcom_backend.XComBackend".

Use case/motivation

It would allow running the cli airflow db ... without access to dags or plugins

( my main motivation is that I'm using gcp composer and the db update K8S job do not sync the dags and plugins ( from gcs ) before running the cmd airflow db update . So I can't update my composer env )

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[potiuk]

You can override the configuration that currently causes the plugins to be loaded to not do it. AIRFLOW__CORE__..... speciflically to run the CLI.

Would that solve the problem @raphaelauv ?

[raphaelauv]

yes it work that way.

But I can't custom this conf only for the airflow-db K8S job in gcp composer and also can't patch the K8S job direclty

the namespace "composer-system" is managed and the request's verb "patch" is denied

That mean if I want update my gcp composer env , I need to turn off all my dags , disable my conf , then update composer , then reactivate my conf , then reactivate all my dags.

No very HA 😓

thanks for answer @potiuk

[potiuk]

But isn't that something that should be raised to the composer team? I do not think we can do much about it, really. In Our Helm Chart we have the possibility of using "base" airflow image for DB jobs - maybe that should be an option for Composer, to make "vanilla" airflow (including custom config) to run DB ?

[potiuk]

I also converted it to a discussion as it's likeley where it belongs.