SSHOperator sigterm in HA

[alokgogate]

Hi guys,

We were trying scheduler HA with SSHOperator in our dag, but as soon as we try to do test HA i.e. manually kill one of the schedulers the task running also gets killed. What we were trying was to connect to beeline on another node to run a hive job. Is this behaviour expected for SSHOperator ? Should we use the native operators available like hive operator, spark operator etc instead of SSHOperator

Error trace seen in logs of dag:

[2021-04-12 16:14:18,035] {taskinstance.py:1239} ERROR - Received SIGTERM. Terminating subprocesses.
[2021-04-12 16:14:18,035] {local_task_job.py:77} ERROR - Received SIGTERM. Terminating subprocesses
[2021-04-12 16:14:18,035] {taskinstance.py:570} DEBUG - Refreshing TaskInstance <TaskInstance: test_beeline_sshop.run_job 2021-04-12T10:42:57.584041+00:00 [running]> from DB
[2021-04-12 16:14:18,056] {taskinstance.py:605} DEBUG - Refreshed TaskInstance <TaskInstance: test_beeline_sshop.run_job 2021-04-12T10:42:57.584041+00:00 [running]>
[2021-04-12 16:14:18,058] {taskinstance.py:1455} ERROR - SSH operator error: Task received SIGTERM signal
Traceback (most recent call last):
File "/opt/anaconda3/lib/python3.7/site-packages/airflow/providers/ssh/operators/ssh.py", line 136, in execute
readq, _, _ = select([channel], [], [], self.timeout)
File "/opt/anaconda3/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1241, in signal_handler
raise AirflowException("Task received SIGTERM signal")
airflow.exceptions.AirflowException: Task received SIGTERM signal
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/anaconda3/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1112, in _run_raw_task
self._prepare_and_execute_task_with_callbacks(context, task)
File "/opt/anaconda3/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1285, in _prepare_and_execute_task_with_callbacks
result = self._execute_task(context, task_copy)
File "/opt/anaconda3/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1315, in _execute_task
result = task_copy.execute(context=context)
File "/opt/anaconda3/lib/python3.7/site-packages/airflow/providers/ssh/operators/ssh.py", line 171, in execute
raise AirflowException(f"SSH operator error: {str(e)}")
airflow.exceptions.AirflowException: SSH operator error: Task received SIGTERM signal
[2021-04-12 16:14:18,059] {taskinstance.py:1862} DEBUG - Task Duration set to 3.2764
[2021-04-12 16:14:18,060] {taskinstance.py:1503} INFO - Marking task as FAILED. dag_id=test_beeline_sshop, task_id=run_job, execution_date=20210412T104257, start_date=20210412T104414, end_date=20210412T104418
[2021-04-12 16:14:18,066] {process_utils.py:100} INFO - Sending Signals.SIGTERM to GPID 124305
[2021-04-12 16:14:18,070] {taskinstance.py:1239} ERROR - Received SIGTERM. Terminating subprocesses.
[2021-04-12 16:14:18,070] {cli_action_loggers.py:84} DEBUG - Calling callbacks: []
[2021-04-12 16:14:18,080] {process_utils.py:66} INFO - Process psutil.Process(pid=124305, status='terminated') (124305) terminated with exit code 1

[ashb]

Which executor are you using? If you are using the LocalExecuto, then yes, this is expected behaviour and tasks will not survive longer than the parent scheduler process being killed.

If you want that you should use the Celery executor.

[alokgogate]

Yes it's being tested with LocalExecutor
will try with Celery Executor