On the Airflow web UI as admin user on Admin tab connections getting error

[vinayambaty]

Hello,
I have built Airflow version: 2.6.2 with helm chart on Kubernetes 1.26.1 version .
The error while try to access Admin tab connections on web console of Airflow UI Error as below
please guide me to fix this issue Thanks.

#######################################
Ooops!
Something bad has happened.

Airflow is used by many users, and it is very likely that others had similar problems and you can easily find
a solution to your problem.

Consider following these steps:

  * gather the relevant information (detailed logs with errors, reproduction steps, details of your deployment)

  * find similar issues using:
     * GitHub Discussions
     * GitHub Issues
     * Stack Overflow
     * the usual search engine you use on a daily basis

  * if you run Airflow on a Managed Service, consider opening an issue using the service support channels

  * if you tried and have difficulty with diagnosing and fixing the problem yourself, consider creating a bug report.
    Make sure however, to include all relevant details and results of your investigation so far.

Python version: 3.7.17
Airflow version: 2.6.2
Node: redact
-------------------------------------------------------------------------------
Error! Please contact server admin.
#########################################

[Taragolis]

Without webserver logs it is hard to say what is going wrong.

But let's me guess that it is related to:

• Users Error #34970
• Under security tab there is Ooops! Something bad has happened. errors. #35064

[vinayambaty]

Please find the error logs of webserver

[2023-10-25T09:11:58.294+0000] {app.py:1745} ERROR - Exception on /connection/list/ [GET]
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.7/site-packages/flask/app.py", line 2529, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/airflow/.local/lib/python3.7/site-packages/flask/app.py", line 1825, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/airflow/.local/lib/python3.7/site-packages/flask/app.py", line 1823, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/airflow/.local/lib/python3.7/site-packages/flask/app.py", line 1799, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
  File "/home/airflow/.local/lib/python3.7/site-packages/flask_appbuilder/security/decorators.py", line 139, in wraps
    return f(self, *args, **kwargs)
  File "/home/airflow/.local/lib/python3.7/site-packages/flask_appbuilder/views.py", line 554, in list
    widgets = self._list()
  File "/home/airflow/.local/lib/python3.7/site-packages/flask_appbuilder/baseviews.py", line 1182, in _list
    page_size=page_size,
  File "/home/airflow/.local/lib/python3.7/site-packages/flask_appbuilder/baseviews.py", line 1081, in _get_list_widget
    page_size=page_size,
  File "/home/airflow/.local/lib/python3.7/site-packages/flask_appbuilder/models/sqla/interface.py", line 500, in query
    query_results = query.all()
  File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 2773, in all
    return self._iter().all()
  File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/result.py", line 1476, in all
    return self._allrows()
  File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/result.py", line 401, in _allrows
    rows = self._fetchall_impl()
  File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/result.py", line 1389, in _fetchall_impl
    return self._real_result._fetchall_impl()
  File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/result.py", line 1813, in _fetchall_impl
    return list(self.iterator)
  File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/orm/loading.py", line 151, in chunks
    rows = [proc(row) for row in fetch]
  File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/orm/loading.py", line 151, in <listcomp>
    rows = [proc(row) for row in fetch]
  File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/orm/loading.py", line 984, in _instance
    state.manager.dispatch.load(state, context)
  File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/event/attr.py", line 334, in __call__
    fn(*args, **kw)
  File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/orm/mapper.py", line 3702, in _event_on_load
    instrumenting_mapper._reconstructor(state.obj())
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/connection.py", line 170, in on_db_load
    if self.password:
  File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/orm/attributes.py", line 606, in __get__
    retval = self.descriptor.__get__(instance, owner)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/connection.py", line 267, in get_password
    return fernet.decrypt(bytes(self._password, "utf-8")).decode()
  File "/home/airflow/.local/lib/python3.7/site-packages/cryptography/fernet.py", line 210, in decrypt
    raise InvalidToken
cryptography.fernet.InvalidToken

[Taragolis]

cryptography.fernet.InvalidToken

Yeah, that is the different one.

Did you setup [core] fernet_key? For the additional reference, you could have a look:

• Security / Secrets / Fernet
• Airflow Helm Chart: Parameters reference

[vinayambaty]

Dear Team,

We tried to uninstall airflow, deleted the secrets referencing the airlfow and tried to re-install the same. However, we are encourntering the same error.

I believe during the installation fernet key should get generated/encrypted by itself. Could you please investigate further and suggest us next steps.

We are seeing the same log trace as posted earlier.

Thanks

[jscheffl]

Depending on your setup, have you resetted the DB as well via airflow db reset CLI?

[vinayambaty]

hi jens
Thanks for the Update
I had done "airflow db reset" after that restarted webserver now not able to login to Airflow UI and changes needs to done after reset ?
Thanks

[jscheffl]

I was thinking about if your setup is broken and you did a clean install, then the DB might be the last thing that keeps (a potential broken) leftover.
If this did not help, then something in your setup is "strange" or you did a error during install.

[vinayambaty]

Hi Jens,
After helm upgrade and install the airflow service able to login to web ui with admin but the above issue still same as above connection error.

Thanks

[jscheffl]

Then something in your setup is severely broken. If the stack trace is really the same, then something is generally broken but nothing that can be diagnosed from far away.
Maybe you need to drop the full db instance and start over from scratch.

[potiuk]

Clear your Connection table (you can do it manually or with airflow db clean I guess). My best guess is that you added some connection when you did not have Fernet Key or when you had a different one - maybe that was bad configuration, maybe temporary setting. You should generally use "rotation' feature described in the documentation (see the CLI docs) to change key.

[potiuk]

Byt yes. likely Jens advice about starting from scratch is good - you might start having other problems if your Fernet key was not properly rotated.

[zhangruilin2020]

i met the same problem, 'airflow db reset' and 'airflow db init' did not work. Finally, I delete all table from pg metadata, it worked after re-install airflow.

[vinayambaty]

Hi @Taragolis @jens-scheffler-bosch @potiuk

Thanks for the Support Now my issue got fixed 👍
what I had done is moved db to different location and created new airflow setup it worked (previously when ever I install the Airflow its using the same DB from the PV db so the issue was the same)

Cheers,
Vinay Ambaty