Airflow 1.10.10 TriggerDagRunOperator failure on Key already exists #18838
Unanswered
baraknielsen
asked this question in
Q&A
Replies: 1 comment
-
Thanks for opening your first issue here! Be sure to follow the issue template! |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Apache Airflow version:
1.10.10
What happened:
Hi All,
when trying to trigger my dag which has a two-level nested subdags from another dag
the task is failing with an error that one of the subdags key already exist although the dag is eventually get triggered,
from the airflow code, I saw that when calling main_dag.subdags it returns all the subdags inside including the ones that are nested on its subdags,
so when I run the TriggerDagRunOperator it tries to trigger the second level subdags twice due to this airflow code:
under airflow/api/common/experimental/trigger_dag.py:91
What you expected to happen:
the TriggerDagRunOperator should finish with success state
How to reproduce it:
Trigger Dag code:
Triggered Dag code:
Failed operator log:
[2021-04-20 20:11:18,501] {taskinstance.py:900} INFO - Executing <Task(TriggerDagRunOperator): trigger_dag> on 2021-04-19T08:30:00+00:00
[2021-04-20 20:11:18,503] {standard_task_runner.py:53} INFO - Started process 62669 to run task
[2021-04-20 20:11:18,633] {logging_mixin.py:112} INFO - Running %s on host %s <TaskInstance: TriggerExample.trigger_dag 2021-04-19T08:30:00+00:00 [running]> ip-192-168-1-10.ec2.internal
[2021-04-20 20:11:18,692] {logging_mixin.py:112} INFO - [2021-04-20 20:11:18,691] {dagbag.py:396} INFO - Filling up the DagBag from /*****/airflow/dags/triggeredDag.py
[2021-04-20 20:11:18,880] {taskinstance.py:1145} ERROR - (psycopg2.errors.UniqueViolation) duplicate key value violates unique constraint "dag_run_dag_id_run_id_key"
DETAIL: Key (dag_id, run_id)=(TriggeredDag.sub1.sub2, trig__2021-04-19) already exists.
[SQL: INSERT INTO dag_run (dag_id, execution_date, start_date, end_date, state, run_id, external_trigger, conf) VALUES (%(dag_id)s, %(execution_date)s, %(start_date)s, %(end_date)s, %(state)s, %(run_id)s, %(external_trigger)s, %(conf)s) RETURNING dag_run.id]
[parameters: {'dag_id': 'TriggeredDag.sub1.sub2', 'execution_date': <Pendulum [2021-04-19T00:00:00+00:00]>, 'start_date': datetime.datetime(2021, 4, 20, 17, 11, 18, 875681, tzinfo=<Timezone [UTC]>), 'end_date': None, 'state': 'running', 'run_id': 'trig__2021-04-19', 'external_trigger': True, 'conf': None}]
am I missing something? or is this a real bug?
Beta Was this translation helpful? Give feedback.
All reactions