How do I reload webserver_config.py without restarting the webserver

[tmvcode]

Hi, in my webserver_config.py I have the AUTH_LDAP_BIND_PASSWORD = readFileAsString(/etc/config/password.txt). The password.txt is mounted from a k8s secret and updated on a regular interval. It seems like the webserver_config.py is loaded only once and cached during start up, because it's not picking up the updated AUTH_LDAP_BIND_PASSWORD dynamically, even though my password.txt file is updated by the k8s volume mount. To get updated AUTH_LDAP_BIND_PASSWORD I have to do a rolling restart on the webserver pod. Is there a way I can trrigger a reload of webserver_config.py?

[potiuk]

You cant. You need to restart the webserver. Note that generally speaking your gunicorn servers will restart every now and then (I believe it is configurable - look it up in Airflow and gunicorn configuration) and they should generally pick up the new configuration if it is done dynamically (AIRFLOW__*_CMD prefix in variable or via Secrets Backend) - see https://airflow.apache.org/docs/apache-airflow/stable/howto/set-config.html. Also such gunicorn when restarted will parse webserver_config.py (not .yml) whenever such gunicorn process restarts. I do not know how you map and read your webserver_config.yml into webserver_config.py of Airlfow (this is done somethigng by your deployment) - but if you can have your webserver_config.py read and process the .yml file of yours, it should be used anew by every restarted gunicorn process.

But how you map it is something done in your deployment because airflow does not have a concept of webserver_config.yml

[tmvcode]

My understanding is that the webserver starts up gunicorn with the default 4 worker processes to handle the requests. Those workers are fed with the settings from the webserver_config.yml and the settings are cached unless the worker processes get restarted. I added the following debug line into my webserver_config.yml and saw that my password was being updated with the new value from the k8s secret file volume mount for every requests, but I still wasn't able to log into the webserver because the workers were still holding on to the old AUTH_LDAP_BIND_PASSWORD value. The only way I know to get the webserver to use the new value of AUTH_LDAP_BIND_PASSWORD is to do a rolling restart of the airflow webserver pod.

    @app.before_request
    def set_auth_ldap_bind_password () -> None:
        AUTH_LDAP_BIND_PASSWORD = readFileAsString(/etc/config/password.txt)
        print("setting AUTH_LDAP_BIND_PASSWORD before every request")

[potiuk]

Again. Airlfow (standard open-source) has no notion of webserver_config.yml. It has webserver-config.py. If you have some code that uses webserver_config.yml, then you need to check with whoever developed it - but it's not part of open-source airflow.

Maybe you are talking about something else - but there is no webserver_config.yml anywhere. There is webserver-configmap.yml but this is used to create webserver_config.py - and if you somehow make your dynamic code to change this file, then yes it will be updated when gunicorn restarts.

[tmvcode]

Sorry I meant changes were made to webserver_config.py, not webserver_config.yml. There is no webserver_config.yml. I also experimented with the webserver's 3 config properties below to force the gunicorn worker to refresh sooner. You can see from the log below the worker did restart in about ~20 seconds. The AUTH_LDAP_BIND_PASSWORD did refresh from reading a dynamically mounted volume password.txt file, but it had no effect. I was able to log into the UI even though the AUTH_LDAP_BIND_PASSWORD was set to some wrong value (wrong_password).

[webserver]
worker_refresh_batch_size: 1
worker_refresh_interval: 20
workers: 1

[2024-10-25 04:11:07 +0000] [15] [INFO] Handling signal: ttin
[2024-10-25 04:11:07 +0000] [140] [INFO] Booting worker with pid: 140
setting AUTH_LDAP_BIND_PASSWORD before every request: xxx
10.240.47.193 - - [25/Oct/2024:04:11:07 +0000] "GET /login/ HTTP/1.1" 200 17363 "https://10.240.32.31/home" "EF-Strati-HealthCheck-Client-Heka/2.0"
[2024-10-25 04:11:08 +0000] [15] [INFO] Handling signal: ttou
[2024-10-25 04:11:08 +0000] [139] [INFO] Worker exiting (pid: 139)
[2024-10-25 04:11:09 +0000] [15] [ERROR] Worker (pid:139) was sent SIGTERM!
setting AUTH_LDAP_BIND_PASSWORD before every request: xxx
...

setting AUTH_LDAP_BIND_PASSWORD before every request: wrong_password
10.240.32.126 - - [25/Oct/2024:04:11:28 +0000] "GET /login/ HTTP/1.1" 200 17363 "https://10.240.32.31/home" "EF-Strati-HealthCheck-Client-Heka/2.0"
[2024-10-25 04:11:28 +0000] [15] [INFO] Handling signal: ttou
[2024-10-25 04:11:28 +0000] [140] [INFO] Worker exiting (pid: 140)
[2024-10-25 04:11:29 +0000] [15] [ERROR] Worker (pid:140) was sent SIGTERM!
setting AUTH_LDAP_BIND_PASSWORD before every request: wrong_password

[potiuk]

So you need to check if your webserve_config.py is actually read at startup and whether it actually reads the file when gunicorn restart. It's as easy as adding some log staments in there and looking at the logs. I won't be able to debug it for you- you need to just debug it - knowing the details you already know about gunicorn restarts. Maybe - similar to .yml -> .py you have a typo somewhere. I believe webserver_config.py should be re-read with every restart of the gunicorn server - but don't take my word for it - check it.