Issue with Airflow Composer Upgrade to 2.5.3

[chauthangdu]

Hello Airflow Community,

I hope you're all doing well. I'm currently facing an issue after upgrading my GCP Airflow Composer from version 2.2.4 to 2.5.3. I'm reaching out to seek your assistance in resolving this issue.

After the upgrade, I encountered the error message "Task exited with return code Negsignal.SIGKILL." I have checked the resource utilization, including worker CPU and memory, and confirmed that they are well below their respective limits, with usage around 60% to 70% of the maximum resources.

Additionally, due to the above error, I've noticed a related issue with the execution of data replication tasks that utilize the BigQueryExecuteQueryOperator for daily data append jobs. These tasks are being executed twice, leading to duplicated data. When reviewing the logs, some tasks in the first log report the error "Task exited with return code Negsignal.SIGKILL." For other tasks, the log ends with "Inserting job XXX," indicating that the job was sent to BigQuery and executed successfully. In the second log, all tasks show a status of "Success."

Furthermore, in the Composer monitoring dashboard, I have observed occurrences of task failures in the "Celery execute command failures" and "System terminated tasks" sections that were not present before the update.

And on the Airflow home screen after the upgrade, there is a warning message similar to this:

While upgrading the metadatabase, Airflow had to move some bad data in order to apply new constraints. The moved data can be found in the following tables:
Source table Table with moved rows
rendered_task_instance_fields _airflow_moved__2_3__dangling__rendered_task_instance_fields
task_fail _airflow_moved__2_3__dangling__task_fail
xcom _airflow_moved__2_3__dangling__xcom
Please inspect the moved data to decide whether you need to keep them, and manually drop the moved tables to dismiss this warning. Read more about it in Upgrading.

I've thoroughly reviewed the configurations, resource utilization, and task setup but haven't been able to pinpoint the root cause of these issues.

I kindly request your expertise and insights to help me resolve this matter. If you have encountered similar situations or have any suggestions, I would greatly appreciate your input.

[potiuk]

What does composer support say ?