Scheduler and Webserver shutdown after being up briefly when going from Airflow 2.2.2 to Airflow 2.6.3

[bilindhajer]

Environment:

Airflow Version: 2.6.3
Python Version: 3.8.17
MySQL Version: 5.7 (each instance in Aurora RDS)

I am installing Airflow with pip within a DockerFile using base image python:3.8.17-bullseye.

Providers:
crypto,postgres,hive,jdbc,mysql,statsd,amazon,salesforce,s3

My Airflow Cluster is running on AWS ECS, a task with a container for the webserver, and a container for the scheduler. The scheduler is using LocalExecutor. Both have been given 8GB of memory reservation.

Airflow Configurations which I modified to reflect new properties available in Airflow 2.6.3: (Note I am setting some of these via environment variables).

[core]
dags_folder = /usr/local/airflow/dags
hostname_callable = socket:gethostname
might_contain_dag_callable = airflow.utils.file.might_contain_dag_via_default_heuristic
default_timezone = utc
executor = LocalExecutor
parallelism = 32 
max_active_tasks_per_dag = 16
dags_are_paused_at_creation = True
max_active_runs_per_dag = 16 
load_examples = False
plugins_folder = /usr/local/airflow/plugins
execute_tasks_new_python_interpreter = False
fernet_key = 
donot_pickle = True
dagbag_import_timeout = 30.0
dagbag_import_error_tracebacks = True
dagbag_import_error_traceback_depth = 2
dag_file_processor_timeout = 50
task_runner = StandardTaskRunner
default_impersonation = 
security = 
unit_test_mode = False
enable_xcom_pickling = False
allowed_deserialization_classes = airflow\..*
killed_task_cleanup_time = 60
dag_run_conf_overrides_params = True
dag_discovery_safe_mode = True
dag_ignore_file_syntax = regexp
default_task_retries = 0
default_task_retry_delay = 300
max_task_retry_delay = 86400
default_task_weight_rule = downstream
default_task_execution_timeout = 
min_serialized_dag_update_interval = 30
compress_serialized_dags = False
min_serialized_dag_fetch_interval = 10
max_num_rendered_ti_fields_per_task = 30
check_slas = True
xcom_backend = airflow.models.xcom.BaseXCom
lazy_load_plugins = True
lazy_discover_providers = True
hide_sensitive_var_conn_fields = True
sensitive_var_conn_names = 
default_pool_task_slot_count = 128
max_map_length = 1024
daemon_umask = 0o077

[database]
sql_alchemy_conn = 
sql_engine_encoding = utf-8
sql_alchemy_pool_enabled = True
sql_alchemy_pool_size = 5
sql_alchemy_max_overflow = 10
sql_alchemy_spool_recycle = 1800
sql_alchemy_pool_pre_ping = True
sql_alchemy_schema = 
load_default_connections = False
max_db_retries = 3
check_migrations = True

[logging]
base_log_folder = /usr/local/airflow/logs
remote_logging = False
remote_log_conn_id = 
delete_local_logs = False
google_key_path = 
remote_base_log_folder = 
remote_task_handler_kwargs = 
encrypt_s3_logs = False
logging_level = INFO
celery_logging_level = 
fab_logging_level = WARNING
logging_config_class = log_config.DEFAULT_LOGGING_CONFIG
colored_console_log = True
colored_log_format = [%%(blue)s%%(asctime)s%%(reset)s] {{%%(blue)s%%(filename)s:%%(reset)s%%(lineno)d}} %%(log_color)s%%(levelname)s%%(reset)s - %%(log_color)s%%(message)s%%(reset)s
colored_formatter_class = airflow.utils.log.colored_log.CustomTTYColoredFormatter
log_format = [%%(asctime)s] {{%%(filename)s:%%(lineno)d}} %%(levelname)s - %%(message)s
simple_log_format = %%(asctime)s %%(levelname)s - %%(message)s
dag_processor_log_target = file
dag_processor_log_format = [%(asctime)s] [SOURCE:DAG_PROCESSOR] {%(filename)s:%(lineno)d} %(levelname)s - %(message)s
log_formatter_class = airflow.utils.log.timezone_aware.TimezoneAware
secret_mask_adapter = 
task_log_prefix_template = 
log_filename_template = {{ ti.dag_id }}/{{ ti.task_id }}/{{ ts }}/{{ try_number }}.log
log_processor_filename_template = {{ filename }}.log
dag_processor_manager_log_location = {AIRFLOW_HOME}/logs/dag_processor_manager/dag_processor_manager.log
task_log_reader = task
extra_logger_names = 
worker_log_server_port = 8793
trigger_log_server_port = 8794
file_task_handler_new_folder_permissions = 0o775
file_task_handler_new_file_permissions = 0o664

[metrics]
metrics_allow_list = 
metrics_block_list = 
statsd_on = False
statsd_host = localhost
statsd_port = 8125
statsd_prefix = airflow
stat_name_handler = 
statsd_datadog_enabled = False
statsd_datadog_tags = 
statsd_datadog_metrics_tags = True
statsd_disabled_tags = job_id,run_id
statsd_influxdb_enabled = False
otel_on = False
otel_host = localhost
otel_port = 8889
otel_prefix = airflow
otel_interval_milliseconds = 60000

[secrets]
backend = 
backend_kwargs = 

[cli]
api_client = airflow.api.client.local_client
endpoint_url = http://localhost:8080

[debug]
fail_fast = False

[api]
enable_experimental_api = False
auth_backends = airflow.api.auth.backend.session
maximum_page_limit = 100
fallback_page_limit = 100
google_oauth2_audience = 
google_key_path = 
access_control_allow_headers = 
access_control_allow_methods = 
access_control_allow_origins = 

[lineage]
backend = 

[atlas]
sasl_enabled = False
host = 
port = 21000
username = 
password = 

[operators]
default_owner = airflow
default_cpus = 1
default_ram = 512
default_disk = 512
default_gpus = 0
default_queue = default
allow_illegal_arguments = False

[hive]
default_hive_mapred_queue = 

[webserver]
default_ui_timezone = UTC
web_server_host = 0.0.0.0
web_server_port = 8080
web_server_ssl_cert = 
web_server_ssl_key = 
session_backend = database
web_server_master_timeout = 120
web_server_worker_timeout = 120
worker_refresh_batch_size = 1
worker_refresh_interval = 6000
reload_on_plugin_change = False
workers = 4
worker_class = sync
access_logfile = -
error_logfile = -
access_logformat = 
expose_config = False
expose_hostname = False
expose_stacktrace = False
dag_default_view = grid
dag_orientation = LR
log_fetch_timeout_sec = 5
log_fetch_delay_sec = 2
log_auto_tailing_offset = 30
log_animation_speed = 1000
hide_paused_dags_by_default = False
page_size = 100
navbar_color = #fff
default_dag_run_display_number = 25
enable_proxy_fix = true
proxy_fix_x_for = 1
proxy_fix_x_proto = 1
proxy_fix_x_host = 1
proxy_fix_x_port = 1
proxy_fix_x_prefix = 1
cookie_secure = False
cookie_samesite = Lax
default_wrap = False
x_frame_enabled = True
show_recent_stats_for_completed_runs = True
update_fab_perms = True
session_lifetime_minutes = 43200
instance_name_has_markup = False
auto_refresh_interval = 3
warn_deployment_exposure = True
audit_view_excluded_events = gantt,landing_times,tries,duration,calendar,graph,grid,tree,tree_data
enable_swagger_ui = True
auth_rate_limited = True
auth_rate_limit = 5 per 40 second
caching_hash_method = md5

[email]
email_backend = airflow.utils.email.send_email_smtp
email_conn_id = smtp_default
default_email_on_retry = True
default_email_on_failure = True

[smtp]
smtp_host = localhost
smtp_starttls = True
smtp_ssl = False
smtp_port = 25
smtp_mail_from = [email protected]
smtp_timeout = 30
smtp_retry_limit = 5

[sentry]
sentry_on = False
sentry_dsn = 

[local_kubernetes_executor]
kubernetes_queue = kubernetes

[celery_kubernetes_executor]
kubernetes_queue = kubernetes

[celery]
celery_app_name = airflow.executors.celery_executor
worker_concurrency = 16
worker_prefetch_multiplier = 1
worker_enable_remote_control = True
broker_url = redis://redis:6379/0
result_backend_sqlalchemy_engine_options = 
flower_host = 0.0.0.0
flower_url_prefix = 
flower_port = 5555
flower_basic_auth = 
sync_parallelism = 0
celery_config_options = airflow.config_templates.default_celery.DEFAULT_CELERY_CONFIG
ssl_active = False
ssl_key = 
ssl_cert = 
ssl_cacert = 
pool = prefork
operation_timeout = 1.0
task_track_started = True
task_publish_max_retries = 3
worker_precheck = False

[celery_broker_transport_options]

[dask]
cluster_address = 127.0.0.1:8786
tls_ca = 
tls_cert = 
tls_key = 

[scheduler]
job_heartbeat_sec = 5
scheduler_heartbeat_sec = 5
num_runs = -1
scheduler_idle_sleep_time = 1
min_file_process_interval = 30
parsing_cleanup_interval = 60
stale_dag_threshold = 50
dag_dir_list_interval = 300
print_stats_interval = 30
pool_metrics_interval = 5.0
scheduler_health_check_threshold = 30
enable_health_check = False
scheduler_health_check_server_port = 8974
orphaned_tasks_check_interval = 300.0
child_process_log_directory = {AIRFLOW_HOME}/logs/scheduler
scheduler_zombie_task_threshold = 300
zombie_detection_interval = 10.0
catchup_by_default = False
ignore_first_depends_on_past_by_default = True
max_tis_per_query = 512
use_row_level_locking = True
max_dagruns_to_create_per_loop = 10
max_dagruns_per_loop_to_schedule = 20
schedule_after_task_execution = True
parsing_pre_import_modules = True
parsing_processes = 2
file_parsing_sort_mode = modified_time
standalone_dag_processor = False
max_callbacks_per_loop = 20
dag_stale_not_seen_duration = 600
use_job_schedule = True
allow_trigger_in_future = False
trigger_timeout_check_interval = 15
task_queued_timeout = 600.0
task_queued_timeout_check_interval = 120.0
allowed_run_id_pattern = ^[A-Za-z0-9_.~:+-]+$

[triggerer]
default_capacity = 1000
job_heartbeat_sec = 5

[kerberos]
ccache = /tmp/airflow_krb5_ccache
principal = airflow
reinit_frequency = 3600
kinit_path = kinit
keytab = airflow.keytab
forwardable = True
include_ip = True

[elasticsearch]
host = 
log_id_template = {dag_id}-{task_id}-{run_id}-{map_index}-{try_number}
end_of_log_mark = end_of_log
frontend = 
write_stdout = False
json_format = False
json_fields = asctime, filename, lineno, levelname, message
host_field = host
offset_field = offset
index_patterns = _all

[elasticsearch_configs]
use_ssl = False
verify_certs = True

[kubernetes_executor]
api_client_retry_configuration = 
pod_template_file = 
worker_container_repository = 
worker_container_tag = 
namespace = default
delete_worker_pods = True
delete_worker_pods_on_failure = False
worker_pods_creation_batch_size = 1
multi_namespace_mode = False
multi_namespace_mode_namespace_list = 
in_cluster = True
kube_client_request_args = 
delete_option_kwargs = 
enable_tcp_keepalive = True
tcp_keep_idle = 120
tcp_keep_intvl = 30
tcp_keep_cnt = 6
verify_ssl = True
worker_pods_queued_check_interval = 60
ssl_ca_cert = 

[sensors]
default_timeout = 604800

A very similar setup has no issues on Airflow 2.2.2, this happens when going from Airflow 2.2.2 (using Python 3.7) to Airflow 2.6.3 (using Python 3.8).

Description of Problem:

Both Webserver and Scheduler containers shutdown after being up for a brief period. Here are a snippet of the logs:

Webserver received sigterm after 4 gunicorn workers fail to start after 120 second timeout.

[�[34m2023-08-15T16:47:48.279+0000�[0m] {{�[34mwebserver_command.py:�[0m432}} INFO�[0m - Received signal: 15. Closing gunicorn.�[0m
--

Scheduler logs:

  ____________       _____________
 ____    |__( )_________  __/__  /________      __
____  /| |_  /__  ___/_  /_ __  /_  __ \_ | /| / /
___  ___ |  / _  /   _  __/ _  / / /_/ /_ |/ |/ /
 _/_/  |_/_/  /_/    /_/    /_/  \____/____/|__/
[�[34m2023-08-15T16:58:32.266+0000�[0m] {{�[34mexecutor_loader.py:�[0m114}} INFO�[0m - Loaded executor: LocalExecutor�[0m
[2023-08-15 16:58:32 +0000] [30] [INFO] Starting gunicorn 20.1.0
[2023-08-15 16:58:32 +0000] [30] [INFO] Listening at: http://[::]:8793 (30)
[2023-08-15 16:58:32 +0000] [30] [INFO] Using worker: sync
[2023-08-15 16:58:32 +0000] [31] [INFO] Booting worker with pid: 31
[�[34m2023-08-15T16:58:32.302+0000�[0m] {{�[34mscheduler_job_runner.py:�[0m788}} INFO�[0m - Starting the scheduler�[0m
[�[34m2023-08-15T16:58:32.302+0000�[0m] {{�[34mscheduler_job_runner.py:�[0m795}} INFO�[0m - Processing each file at most -1 times�[0m
[2023-08-15 16:58:32 +0000] [33] [INFO] Booting worker with pid: 33
[�[34m2023-08-15T16:58:32.377+0000�[0m] {{�[34mmanager.py:�[0m165}} INFO�[0m - Launched DagFileProcessorManager with pid: 168�[0m
[�[34m2023-08-15T16:58:32.378+0000�[0m] {{�[34mscheduler_job_runner.py:�[0m1553}} INFO�[0m - Resetting orphaned tasks for active dag runs�[0m
[�[34m2023-08-15T16:58:32.383+0000�[0m] {{�[34msettings.py:�[0m60}} INFO�[0m - Configured default timezone Timezone('UTC')�[0m
[�[34m2023-08-15T16:58:32.394+0000�[0m] {{�[34msettings.py:�[0m499}} INFO�[0m - Loaded airflow_local_settings from /usr/local/airflow/config/airflow_local_settings.py .�[0m
[�[34m2023-08-15T16:58:32.394+0000�[0m] {{�[34mlogging_config.py:�[0m47}} INFO�[0m - Successfully imported user-defined logging config from log_config.DEFAULT_LOGGING_CONFIG�[0m
[�[34m2023-08-15T16:58:32.753+0000�[0m] {{�[34mscheduler_job_runner.py:�[0m247}} INFO�[0m - Exiting gracefully upon receiving signal 15�[0m
[�[34m2023-08-15T16:58:33.756+0000�[0m] {{�[34mprocess_utils.py:�[0m131}} INFO�[0m - Sending Signals.SIGTERM to group 168. PIDs of all processes in the group: [168]�[0m
[�[34m2023-08-15T16:58:33.757+0000�[0m] {{�[34mprocess_utils.py:�[0m86}} INFO�[0m - Sending the signal Signals.SIGTERM to group 168�[0m
[�[34m2023-08-15T16:58:33.889+0000�[0m] {{�[34mprocess_utils.py:�[0m79}} INFO�[0m - Process psutil.Process(pid=168, status='terminated', exitcode=0, started='16:58:32') (168) terminated with exit code 0�[0m
[�[34m2023-08-15T16:58:33.889+0000�[0m] {{�[34mlocal_executor.py:�[0m400}} INFO�[0m - Shutting down LocalExecutor; waiting for running tasks to finish.  Signal again if you don't want to wait.�[0m
[�[34m2023-08-15T16:58:33.918+0000�[0m] {{�[34mprocess_utils.py:�[0m131}} INFO�[0m - Sending Signals.SIGTERM to group 168. PIDs of all processes in the group: []�[0m
[�[34m2023-08-15T16:58:33.918+0000�[0m] {{�[34mprocess_utils.py:�[0m86}} INFO�[0m - Sending the signal Signals.SIGTERM to group 168�[0m
[�[34m2023-08-15T16:58:33.918+0000�[0m] {{�[34mprocess_utils.py:�[0m100}} INFO�[0m - Sending the signal Signals.SIGTERM to process 168 as process group is missing.�[0m
[�[34m2023-08-15T16:58:33.918+0000�[0m] {{�[34mscheduler_job_runner.py:�[0m864}} INFO�[0m - Exited execute loop�[0m
[2023-08-15 16:58:33 +0000] [30] [INFO] Handling signal: term
[2023-08-15 16:58:33 +0000] [31] [INFO] Worker exiting (pid: 31)
[2023-08-15 16:58:33 +0000] [33] [INFO] Worker exiting (pid: 33)
[2023-08-15 16:58:34 +0000] [30] [INFO] Shutting down: Master____________       _____________
--

As mentioned a very similar setup has no issues on Airflow 2.2.2/Python 3.7. I am able to run this setup successfully locally. Any help would be appreciated. Thank you!

[bilindhajer]

Hi, still facing this issue. Things that I have tried but still result in the scheduler and webserver workers shutting down,

1. Python 3.7, 3.8, 3.9 in this Airflow setup still have this issue,
2. We removed the healthcheck we had in place in anticipation that it might be shutting down the containers preemptively.
3. This is not an issue with the database connection.
4. Does not appear to be an issue with resources even though we are seeing spikes CPU and memory utilization utilization below.

I attached more of the webserver and scheduler logs below.

Webserver logs

[�[34m2023-08-17T23:17:33.613+0000�[0m] {{�[34mmigration.py:�[0m213}} INFO�[0m - Context impl MySQLImpl.�[0m
--
[�[34m2023-08-17T23:17:33.614+0000�[0m] {{�[34mmigration.py:�[0m216}} INFO�[0m - Will assume non-transactional DDL.�[0m
[�[34m2023-08-17T23:17:33.770+0000�[0m] {{�[34mmigration.py:�[0m213}} INFO�[0m - Context impl MySQLImpl.�[0m
[�[34m2023-08-17T23:17:33.770+0000�[0m] {{�[34mmigration.py:�[0m216}} INFO�[0m - Will assume non-transactional DDL.�[0m
[�[34m2023-08-17T23:17:33.778+0000�[0m] {{�[34mdb.py:�[0m1591}} INFO�[0m - Creating tables�[0m
INFO  [alembic.runtime.migration] Context impl MySQLImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
Initialization done
Successfully Initialized Database
Upgrading database
/usr/local/lib/python3.8/site-packages/airflow/configuration.py:479 FutureWarning: The 'log_filename_template' setting in [logging] has the old default value of '{{ ti.dag_id }}/{{ ti.task_id }}/{{ ts }}/{{ try_number }}.log'. This value has been changed to 'dag_id={{ ti.dag_id }}/run_id={{ ti.run_id }}/task_id={{ ti.task_id }}/{% if ti.map_index >= 0 %}map_index={{ ti.map_index }}/{% endif %}attempt={{ try_number }}.log' in the running config, but please update your config before Apache Airflow 3.0.
DB: mysql://<redacted>@<redacted>
Performing upgrade with database mysql://<redacted>@<redacted>
[�[34m2023-08-17T23:17:36.026+0000�[0m] {{�[34mmigration.py:�[0m213}} INFO�[0m - Context impl MySQLImpl.�[0m
[�[34m2023-08-17T23:17:36.027+0000�[0m] {{�[34mmigration.py:�[0m216}} INFO�[0m - Will assume non-transactional DDL.�[0m
[�[34m2023-08-17T23:17:36.034+0000�[0m] {{�[34mdb.py:�[0m1591}} INFO�[0m - Creating tables�[0m
INFO  [alembic.runtime.migration] Context impl MySQLImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
Upgrades done
Successfully Upgraded Database
Setting variables
Cluster=staging
/usr/local/lib/python3.8/site-packages/airflow/configuration.py:479 FutureWarning: The 'log_filename_template' setting in [logging] has the old default value of '{{ ti.dag_id }}/{{ ti.task_id }}/{{ ts }}/{{ try_number }}.log'. This value has been changed to 'dag_id={{ ti.dag_id }}/run_id={{ ti.run_id }}/task_id={{ ti.task_id }}/{% if ti.map_index >= 0 %}map_index={{ ti.map_index }}/{% endif %}attempt={{ try_number }}.log' in the running config, but please update your config before Apache Airflow 3.0.
Variable Cluster created
/usr/local/lib/python3.8/site-packages/airflow/configuration.py:479 FutureWarning: The 'log_filename_template' setting in [logging] has the old default value of '{{ ti.dag_id }}/{{ ti.task_id }}/{{ ts }}/{{ try_number }}.log'. This value has been changed to 'dag_id={{ ti.dag_id }}/run_id={{ ti.run_id }}/task_id={{ ti.task_id }}/{% if ti.map_index >= 0 %}map_index={{ ti.map_index }}/{% endif %}attempt={{ try_number }}.log' in the running config, but please update your config before Apache Airflow 3.0.
Successfully deleted connection with `conn_id`=airflow_db
/usr/local/lib/python3.8/site-packages/airflow/configuration.py:479 FutureWarning: The 'log_filename_template' setting in [logging] has the old default value of '{{ ti.dag_id }}/{{ ti.task_id }}/{{ ts }}/{{ try_number }}.log'. This value has been changed to 'dag_id={{ ti.dag_id }}/run_id={{ ti.run_id }}/task_id={{ ti.task_id }}/{% if ti.map_index >= 0 %}map_index={{ ti.map_index }}/{% endif %}attempt={{ try_number }}.log' in the running config, but please update your config before Apache Airflow 3.0.
Successfully added `conn_id`=airflow_db : mysql://<redacted>@<redacted>
Successfully add MySql Connection for Airflow DB
/usr/local/lib/python3.8/site-packages/airflow/configuration.py:479 FutureWarning: The 'log_filename_template' setting in [logging] has the old default value of '{{ ti.dag_id }}/{{ ti.task_id }}/{{ ts }}/{{ try_number }}.log'. This value has been changed to 'dag_id={{ ti.dag_id }}/run_id={{ ti.run_id }}/task_id={{ ti.task_id }}/{% if ti.map_index >= 0 %}map_index={{ ti.map_index }}/{% endif %}attempt={{ try_number }}.log' in the running config, but please update your config before Apache Airflow 3.0.
admin already exist in the db
Successfully created admin user
/usr/local/lib/python3.8/site-packages/airflow/configuration.py:479 FutureWarning: The 'log_filename_template' setting in [logging] has the old default value of '{{ ti.dag_id }}/{{ ti.task_id }}/{{ ts }}/{{ try_number }}.log'. This value has been changed to 'dag_id={{ ti.dag_id }}/run_id={{ ti.run_id }}/task_id={{ ti.task_id }}/{% if ti.map_index >= 0 %}map_index={{ ti.map_index }}/{% endif %}attempt={{ try_number }}.log' in the running config, but please update your config before Apache Airflow 3.0.
Variable PagerDutyIntegrationKey created
Starting Airflow webserver
Starting webserver with airflow v2 --------------------------------------------------
/usr/local/lib/python3.8/site-packages/airflow/configuration.py:479 FutureWarning: The 'log_filename_template' setting in [logging] has the old default value of '{{ ti.dag_id }}/{{ ti.task_id }}/{{ ts }}/{{ try_number }}.log'. This value has been changed to 'dag_id={{ ti.dag_id }}/run_id={{ ti.run_id }}/task_id={{ ti.task_id }}/{% if ti.map_index >= 0 %}map_index={{ ti.map_index }}/{% endif %}attempt={{ try_number }}.log' in the running config, but please update your config before Apache Airflow 3.0.
[�[34m2023-08-17T23:17:51.227+0000�[0m] {{�[34mlogging_config.py:�[0m47}} INFO�[0m - Successfully imported user-defined logging config from log_config.DEFAULT_LOGGING_CONFIG�[0m
____________       _____________
____    \|__( )_________  __/__  /________      __
____  /\| \|_  /__  ___/_  /_ __  /_  __ \_ \| /\| / /
___  ___ \|  / _  /   _  __/ _  / / /_/ /_ \|/ \|/ /
_/_/  \|_/_/  /_/    /_/    /_/  \____/____/\|__/
Running the Gunicorn Server with:
Workers: 4 sync
Host: 0.0.0.0:8080
Timeout: 120
Logfiles: - -
Access Logformat:
=================================================================
[�[34m2023-08-17T23:17:53.068+0000�[0m] {{�[34mwebserver_command.py:�[0m432}} INFO�[0m - Received signal: 15. Closing gunicorn.�[0m

Scheduler Logs:

Starting Airflow scheduler
--
/usr/local/lib/python3.8/site-packages/airflow/configuration.py:479 FutureWarning: The 'log_filename_template' setting in [logging] has the old default value of '{{ ti.dag_id }}/{{ ti.task_id }}/{{ ts }}/{{ try_number }}.log'. This value has been changed to 'dag_id={{ ti.dag_id }}/run_id={{ ti.run_id }}/task_id={{ ti.task_id }}/{% if ti.map_index >= 0 %}map_index={{ ti.map_index }}/{% endif %}attempt={{ try_number }}.log' in the running config, but please update your config before Apache Airflow 3.0.
____________       _____________
____    \|__( )_________  __/__  /________      __
____  /\| \|_  /__  ___/_  /_ __  /_  __ \_ \| /\| / /
___  ___ \|  / _  /   _  __/ _  / / /_/ /_ \|/ \|/ /
_/_/  \|_/_/  /_/    /_/    /_/  \____/____/\|__/
[�[34m2023-08-17T23:21:20.495+0000�[0m] {{�[34mexecutor_loader.py:�[0m114}} INFO�[0m - Loaded executor: LocalExecutor�[0m
[2023-08-17 23:21:20 +0000] [30] [INFO] Starting gunicorn 20.1.0
[2023-08-17 23:21:20 +0000] [30] [INFO] Listening at: http://[::]:8793 (30)
[2023-08-17 23:21:20 +0000] [30] [INFO] Using worker: sync
[2023-08-17 23:21:20 +0000] [31] [INFO] Booting worker with pid: 31
[�[34m2023-08-17T23:21:20.535+0000�[0m] {{�[34mscheduler_job_runner.py:�[0m788}} INFO�[0m - Starting the scheduler�[0m
[�[34m2023-08-17T23:21:20.535+0000�[0m] {{�[34mscheduler_job_runner.py:�[0m795}} INFO�[0m - Processing each file at most -1 times�[0m
[�[34m2023-08-17T23:21:20.618+0000�[0m] {{�[34mmanager.py:�[0m165}} INFO�[0m - Launched DagFileProcessorManager with pid: 166�[0m
[�[34m2023-08-17T23:21:20.620+0000�[0m] {{�[34mscheduler_job_runner.py:�[0m1553}} INFO�[0m - Resetting orphaned tasks for active dag runs�[0m
[2023-08-17 23:21:20 +0000] [171] [INFO] Booting worker with pid: 171
[�[34m2023-08-17T23:21:20.626+0000�[0m] {{�[34msettings.py:�[0m60}} INFO�[0m - Configured default timezone Timezone('UTC')�[0m
[�[34m2023-08-17T23:21:20.637+0000�[0m] {{�[34mlogging_config.py:�[0m47}} INFO�[0m - Successfully imported user-defined logging config from log_config.DEFAULT_LOGGING_CONFIG�[0m
[�[34m2023-08-17T23:21:29.732+0000�[0m] {{�[34mscheduler_job_runner.py:�[0m247}} INFO�[0m - Exiting gracefully upon receiving signal 15�[0m
[�[34m2023-08-17T23:21:30.736+0000�[0m] {{�[34mprocess_utils.py:�[0m131}} INFO�[0m - Sending Signals.SIGTERM to group 166. PIDs of all processes in the group: [166]�[0m
[�[34m2023-08-17T23:21:30.737+0000�[0m] {{�[34mprocess_utils.py:�[0m86}} INFO�[0m - Sending the signal Signals.SIGTERM to group 166�[0m
[�[34m2023-08-17T23:21:30.869+0000�[0m] {{�[34mprocess_utils.py:�[0m79}} INFO�[0m - Process psutil.Process(pid=166, status='terminated', exitcode=0, started='23:21:20') (166) terminated with exit code 0�[0m
[�[34m2023-08-17T23:21:30.870+0000�[0m] {{�[34mlocal_executor.py:�[0m400}} INFO�[0m - Shutting down LocalExecutor; waiting for running tasks to finish.  Signal again if you don't want to wait.�[0m
[�[34m2023-08-17T23:21:30.910+0000�[0m] {{�[34mprocess_utils.py:�[0m131}} INFO�[0m - Sending Signals.SIGTERM to group 166. PIDs of all processes in the group: []�[0m
[�[34m2023-08-17T23:21:30.911+0000�[0m] {{�[34mprocess_utils.py:�[0m86}} INFO�[0m - Sending the signal Signals.SIGTERM to group 166�[0m
[�[34m2023-08-17T23:21:30.911+0000�[0m] {{�[34mprocess_utils.py:�[0m100}} INFO�[0m - Sending the signal Signals.SIGTERM to process 166 as process group is missing.�[0m
[�[34m2023-08-17T23:21:30.911+0000�[0m] {{�[34mscheduler_job_runner.py:�[0m864}} INFO�[0m - Exited execute loop�[0m
[2023-08-17 23:21:30 +0000] [30] [INFO] Handling signal: term
[2023-08-17 23:21:30 +0000] [31] [INFO] Worker exiting (pid: 31)
[2023-08-17 23:21:30 +0000] [171] [INFO] Worker exiting (pid: 171)
[2023-08-17 23:21:31 +0000] [30] [INFO] Shutting down: Master

Does anyone have any pointers they could please share? I cannot ssh to the containers either as they don't come up long enough.

Thank you.

[potiuk]

Well. Still does not help. Something in your environment is sendig SIGTERM to the processes. This seems awfully like something in the way you deploy airflow. There is some way you are starting and managing the processes - but how it's done you have not explained. Regardless it looks looks like that something sends a TERM signal to both - webserver and scheduler. Likely some kind of monitor in your instance (like the healthcheck you mentioned). Maybe you have not damonized the processes of yours when you run it and they got sigterm when whatever management solution you have finished starting it up and decided to close all non-daemonized processes for example - you should look in within your deployment mechanisms (this is wild guess, i have no idea what way you are starting and running and monitoring airflow containers there).

[potiuk]

So if you are using ECS - you need to look at your ECS logs and configuration, not at airflow configuration in short.

[bilindhajer]

You are absolutely right! Our telegraf container was failing causing the entire airflow service to come down. It is working now. Thanks a bunch!