Error on webserver: worker was sent SIGKILL

[AlexandreGCastor]

Hello,

On airflow 2.6.2 deployed via helm chart 1.10.0, I have the following error on webserver log: [2023-08-02 07:45:10 +0000] [13] [ERROR] Worker (pid:23) was sent SIGKILL! Perhaps out of memory?

I have remote logging on gcs and for live logging the webserver request the airflow worker. Sometimes logs are slow to open, may timeout or 404 on worker (and bucket).

Airflow webserver config:
expose_config: "True"
worker_refresh_interval: 600
reload_on_plugin_change: "True"
expose_stacktrace: "True"
log_fetch_delay_sec: 5
log_fetch_timeout_sec: 20
web_server_worker_timeout: 300
worker_refresh_batch_size: 0
workers: 3

resources:
requests:
cpu: '1'
limits:
memory: 8Gi
ephemeral-storage: 512Mi

Airflow webserver memory consumption is below 25%

I would like to know how to fix this error.

Thanks in advance

Alexandre Gué,
Lead SRE, Castordoc

[potiuk]

I think you should look at the logs of your K8S. For some reason it's killing webserver. And you will not find it in the logs of airlfow because if teh deployment is killing it with SIGKILL, then the webserver has no chance to react.

You need to (as with any other k8s app) look in your k8s logs for reason why you get SIGKILL. It could be memory, other resources, maybe the liveness probe is doing it (then you need to check liveness probe logs). Generally this is usually indicatin of some resource problem. It's part of the deployment management to make sure the resources are fine-tuned for the particular deployement.

[AlexandreGCastor]

Thanks, my webserver is not killed, only a worker. I see no restart and pod still logs long time after.

But checking in previous logs, I find that:
[2023-08-02 09:23:23 +0000] [14] [CRITICAL] WORKER TIMEOUT (pid:23)

So the sigKill is a consequence of the timeout which is 5min !