Airflow triggerer memory is increasing continously

[deepak4babu]

Apache Airflow version

Other Airflow 2 version (please specify below)

If "Other Airflow 2 version" selected, which one?

2.8.3

What happened?

We started to use deferrable operators and sensors from 2 weeks back. As part of it, we started using triggerer. The triggerer container memory is increasing steadily, even when no triggers are running.

What you think should happen instead?

Memory consumption of idle triggerer should be less and does not increase

How to reproduce

Deploy triggerer in kubernetes with airflow version 2.8.3. Monitor for 24hrs

Operating System

kubernetes

Versions of Apache Airflow Providers

apache-airflow-providers-celery==3.6.1
apache-airflow-providers-common-io==1.4.0
apache-airflow-providers-common-sql==1.11.1
apache-airflow-providers-ftp==3.7.0
apache-airflow-providers-http==4.10.0
apache-airflow-providers-imap==3.5.0
apache-airflow-providers-smtp==1.8.0
apache-airflow-providers-sqlite==3.7.1

Deployment

Other

Deployment details

Airflow version: 2.8.3
Python version: 3.9.x
Database: Postgres 13

Kubernetes details:
Azure kubernetes service
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
k8s Server Version: v1.29.4

Dags are packaged as python packages and installed in the airflow base image. Airflow triggerer is deployed using our custom helm chart, which is just a modified version of official one.

Anything else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[gopidesupavan]

@deepak4babu there are some optimizations made around this area, it will be available in coming airflow release 2.10.3. FYI #40487.

[boris-tumbev]

@deepak4babu there are some optimizations made around this area, it will be available in coming airflow release 2.10.3. FYI #40487.

It does not fix the problem unfortunately, in our setup the memory still get higher and higher and the triggerer eventually restarts

[potiuk]

@deepak4babu there are some optimizations made around this area, it will be available in coming airflow release 2.10.3. FYI #40487.

It does not fix the problem unfortunately, in our setup the memory still get higher and higher and the triggerer eventually restarts

Can you do some memory check using https://github.com/bloomberg/memray for example and see what causes the memory leak?

[jasgok]

check the container "triggerer-log-groomer", its memory usage increases when log size increases