Temporary Directory non-usable in Airflow/MWAA

[rajdallas]

We use Airflow 2.4.3 (hosted in AWS MWAA) & Python 3.10 version and intermittently we face the below issue (i.e) No usable Temporary directory available and particular task fails with below error message. It is not specific to any DAG or Task or follow any particular schedule. We do see other DAGs run during same time without any problem.

Please let us know if any one has encountered below issue.

Error Message :
INFO [Filling up the DagBag from /usr/local/airflow/dags/<DAG_Name>.py
/usr/local/airflow/.local/lib/python3.10/site-packages/airflow/configuration.py:545 DeprecationWarning: The processor_poll_interval option in [scheduler] has been renamed to scheduler_idle_sleep_time - the old setting has been used, but please update your config.
/usr/local/airflow/.local/lib/python3.10/site-packages/airflow/models/baseoperator.py:408 RemovedInAirflow3Warning: provide_context is deprecated as of 2.0 and is no longer required
base.py: INFO [Using connection ID 'aws_default' for task execution.
task_command.py: INFO [Running <TaskInstance: <DAG_Name>.<Task_Name>
Traceback (most recent call last):
File "/usr/local/airflow/.local/bin/airflow", line 8, in sys.exit(main())
File "/usr/local/airflow/.local/lib/python3.10/site-packages/airflow/main.py", line 39, in main args.func(args)
File "/usr/local/airflow/.local/lib/python3.10/site-packages/airflow/cli/cli_parser.py", line 52, in command return func(*args, **kwargs)
File "/usr/local/airflow/.local/lib/python3.10/site-packages/airflow/utils/cli.py", line 103, in wrapper return f(*args, **kwargs)
File "/usr/local/airflow/.local/lib/python3.10/site-packages/airflow/cli/commands/task_command.py", line 382, in task_run _run_task_by_selected_method(args, dag, ti)
File "/usr/local/airflow/.local/lib/python3.10/site-packages/airflow/cli/commands/task_command.py", line 189, in _run_task_by_selected_method _run_task_by_local_task_job(args, ti)
File "/usr/local/airflow/.local/lib/python3.10/site-packages/airflow/cli/commands/task_command.py", line 247, in _run_task_by_local_task_job run_job.run()
File "/usr/local/airflow/.local/lib/python3.10/site-packages/airflow/jobs/base_job.py", line 247, in run self._execute()
File "/usr/local/airflow/.local/lib/python3.10/site-packages/airflow/jobs/local_task_job.py", line 78, in _execute self.task_runner = get_task_runner(self)
File "/usr/local/airflow/.local/lib/python3.10/site-packages/airflow/task/task_runner/init.py", line 62, in get_task_runner task_runner = task_runner_class(local_task_job)
File "/usr/local/airflow/.local/lib/python3.10/site-packages/airflow/task/task_runner/standard_task_runner.py", line 37, in init super().init(local_task_job)
File "/usr/local/airflow/.local/lib/python3.10/site-packages/airflow/task/task_runner/base_task_runner.py", line 91, in init cfg_path tmp_configuration_copy(chmod=0o600,include_env=False, include_cmds=False)
File "/usr/local/airflow/.local/lib/python3.10/site-packages/airflow/utils/configuration.py", line 43, in tmp_configuration_copy temp_fd, cfg_path = mkstemp()
File "/usr/lib/python3.10/tempfile.py", line 334, in mkstemp prefix, suffix, dir, output_type = _sanitize_params(prefix, suffix, dir)
File "/usr/lib/python3.10/tempfile.py", line 126, in _sanitize_params dir = gettempdir()
File "/usr/lib/python3.10/tempfile.py", line 299, in gettempdir return _os.fsdecode(_gettempdir())
File "/usr/lib/python3.10/tempfile.py", line 292, in _gettempdir tempdir = _get_default_tempdir()
File "/usr/lib/python3.10/tempfile.py", line 223, in _get_default_tempdir raise FileNotFoundError(_errno.ENOENT,FileNotFoundError: [Errno 2] No usable temporary directory found in ['/tmp', '/var/tmp', '/usr/tmp', '/usr/local/airflow'

[potiuk]

CCertainly looks like MWAA issue.