RabbitMQOperator task stuck on queued when using CeleryExecutor as core executor #43351
-
Apache Airflow Provider(s)redis Versions of Apache Airflow Providersairflow-provider-rabbitmq 0.6.1 Apache Airflow version2.10.2 Operating Systemairflow bitnami image DeploymentOfficial Apache Airflow Helm Chart Deployment detailsNo response What happenedWhen using CeleryExecutor as the core executor for Airflow and utilizing the RabbitMQOperator task from rabbitmq_provider.operators.rabbitmq, the task remains stuck in the queued state and is not recognized by Celery workers. The issue arises when attempting to use RabbitMQOperator to publish messages to an external RabbitMQ queue, which is not related to the Airflow instance. The root cause of the conflict is that the RabbitMQOperator takes a queue field as a parameter, and this value is mistakenly treated as the name of the queue that the Celery workers will use to execute the task. This creates confusion, as the queue parameter is intended to represent the destination queue for message publishing, not the execution queue for Celery tasks. What you think should happen insteadNo response How to reproduceNA Anything elseNo response Are you willing to submit PR?
Code of Conduct
|
Beta Was this translation helpful? Give feedback.
Replies: 3 comments 1 reply
-
|
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval. |
Beta Was this translation helpful? Give feedback.
-
|
|
Beta Was this translation helpful? Give feedback.
-
|
Thank you so much, when using the sequential executor, the operator publishes to the correct external queue and does not get stuck. however with the celery executor, it gets stuck because it mistakenly assigns the queue name as the task executor, which caused the confusion. i removed the airflow-provider-rabbitmq package, created a custom operator, and used pika to handle my use case. |
Beta Was this translation helpful? Give feedback.
-
|
Yep., This question is to whoever provides the operator - not here. Converted to discussion |
Beta Was this translation helpful? Give feedback.
airflow-provider-rabbitmqis not official Airflow provider. There is noRabbitMQOperatorin the Apache-Airflow project.Unless you can explain what is the bug in Airflow code base I suggest to submit this report to the maintainers of
airflow-provider-rabbitmq