Allow skipping some mapped tasks with downstream mapped tasks

[arroadie]

When a mapped task with downstream mapped task returns None or raises an exception, the whole next task is skipped instead of only the direct mapped task dependent on the original mapped task.
It would feel more natural if, just like a regular dag task instances, it would be mapped directly to its upstream an other tasks of the same level would still work.

[potiuk]

Please explain more details, show examples or screenshots if you want anything happens with it. The explanation of yours is not really clear

[arroadie]

Apologies, wrote it in a rush to avoid losing the context but you're right!
So, let me break it with an example, running airflow v2.3.2:

Dag with 3 tasks [A,B,C], where A returns a list and B and C are dynamically mapped tasks.
Task A runs and returns 5 elements, which would translate to 5 mapped tasks B and C.
Task B starts and has 5 mapped instances: B[0], B[1], B[2], B[3], B[4]. Some of the scenarios for failures are (disconsider retries):
a) No mapped task B[0-n]fails, all C tasks are executed
b) All mapped tasks B[0-n] fails, C is marked as skipped
c) If a runtime exception is raised in B[n], B as a whole is currently marked as a failure and C is skipped
d) If a AirflowSkipException is raised in B[n], B as a whole is skipped and C as well.

Scenarios "a" and "b" are the happy paths, but for "c" and "d", I would expect that task C[m] would be executed for B[m] where m are the mapped tasks from B that did not returned an error.

Let me know if that make more sense!

[potiuk]

This can be (afaik) implemented using task groups. See https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-42+Dynamic+Task+Mapping about "Depth first vs. Breadth first" execution. Not sure if this is implemented in current incarnation of Dynamic Task mapping or whether it's been deferred for later (you need to double-check).

[arroadie]

I'll check if using task groups would allow tasks to have a "depth first" approach

[arroadie]

Also, thanks for the link! Didn't have the chance to check it before and it looks great