Unable to configure azure blob for airflow logs on 2.0.0

[msinha251]

Hi All,

I am trying to configure the azure blob for airflow logs [Airflow deployed on AKS], but unable to do so.
I have used helm chart for deployment for airflow deployment on AKS and then followed this https://airflow.apache.org/docs/apache-airflow/1.10.3/howto/write-logs.html#write-logs-azure for configuring it.
But it's unable to connect with blob.... Keep getting below error, can someone please suggest or help on this.

("'NoneType' object is not callable",) Traceback (most recent call last): File "/home/airflow/.local/lib/python3.6/site-packages/celery/app/trace.py", line 412, in trace_task R = retval = fun(*args, **kwargs) File "/home/airflow/.local/lib/python3.6/site-packages/celery/app/trace.py", line 704, in __protected_call__ return self.run(*args, **kwargs) File "/home/airflow/.local/lib/python3.6/site-packages/airflow/executors/celery_executor.py", line 87, in execute_command _execute_in_fork(command_to_exec) File "/home/airflow/.local/lib/python3.6/site-packages/airflow/executors/celery_executor.py", line 120, in _execute_in_fork logging.shutdown() File "/usr/local/lib/python3.6/logging/__init__.py", line 1946, in shutdown h.close() File "/home/airflow/.local/lib/python3.6/site-packages/airflow/providers/microsoft/azure/log/wasb_task_handler.py", line 97, in close self.wasb_write(log, remote_loc, append=True) File "/home/airflow/.local/lib/python3.6/site-packages/airflow/providers/microsoft/azure/log/wasb_task_handler.py", line 181, in wasb_write self.hook.load_string( File "/home/airflow/.local/lib/python3.6/site-packages/cached_property.py", line 36, in __get__ value = obj.__dict__[self.func.__name__] = self.func(obj) File "/home/airflow/.local/lib/python3.6/site-packages/airflow/providers/microsoft/azure/log/wasb_task_handler.py", line 61, in hook return WasbHook(remote_conn_id) File "/home/airflow/.local/lib/python3.6/site-packages/airflow/providers/microsoft/azure/hooks/wasb.py", line 65, in __init__ self.connection = self.get_conn() File "/home/airflow/.local/lib/python3.6/site-packages/airflow/providers/microsoft/azure/hooks/wasb.py", line 71, in get_conn return BlockBlobService(account_name=conn.login, account_key=conn.password, **service_options) TypeError: 'NoneType' object is not callable

[ephraimbuddy]

Hi @msinha251 , for remote logging to work, you need to follow the documentation which is in https://airflow.apache.org/docs/apache-airflow-providers-microsoft-azure/stable/logging.html

In summary, you need to update the following in airflow configuration:

[logging]
remote_logging=True
remote_base_log_folder=wasb://container-name-you-desire
remote_log_conn_id=wasb_default

Create the wasb_default connection with the necessary information.
You can also add your logging configuration class:

logging_config_class=path.to.your.logging_config

[msinha251]

I have tried adding the sas token in extra field it didn't work. Other then I have also tried with upgraded version of apache-airflow-providers-microsoft-azure but it's same error.

[ephraimbuddy]

Run this:
pip uninstall azure-storage --yes && pip install azure-storage-blob apache-airflow-providers-microsoft-azure==1.1.0

then try the logging again.

[msinha251]

Tried this, but this leads to import issue:
from azure.storage.blob import BlobClient, BlobServiceClient, ContainerClient, StorageStreamDownloader ImportError: cannot import name 'BlobClient'

[ephraimbuddy]

It's a known issue and this is how you can resolve it:
pip uninstall azure-storage apache-airflow-providers-microsoft-azure
pip install azure-storage-blob apache-airflow-providers-microsoft-azure==1.1.0
Please run the commands one after the other

[ephraimbuddy]

Just make sure azure-storage is uninstalled and then install azure-storage-blob