Some tasks in mapped tasks fail before starting, logs are empty

[michaelimas]

Hi dear community,
Will appreciate your insight on this issue we are experiencing-

Once in a while, we experience failed task instances without any log, as if they never started but still marked as failed.

This is how the log of one of the tasks look like:

This happens only in mapped tasks and only for some of the tasks.
When clearing the DAG run it succeeds so it seems like a temporary issue.
It doesn't trigger the on_failure_callback so we don't even get an indication which makes it a very high severity issue.

Looking at the metrics in datadog i see that the scheduler pods have terminated around that time and 2 new pods were initialised ~5 mins beforehand, probably due to deployment.

My questions are:

1. Which component is responsible for this "fault"- is it the scheduler? the worker? on which level these tasks got stuck
2. What can cause that component to "drop" these tasks? maybe you have an idea on the relevant configuration that we can tune.

Any help will be very appriciated.
We run a Kubenetes executor and Airflow v2.9.3

[enchant3dmango]

Could you share the DAG code and the code you are triggering?

Also, try to check the related log in your S3 bucket.

[PowerToThePeople111]

I experienced the same. Without sharing the exact code, I can say, that my dag is just a chain of sequential virtualenv tasks. The first one is mapped and exhibits the same problem as the issue-starter: some mapped tasks are in failed state although they have no start or end date. Also they do not even try to be re-executed: although retries are possible, they failed finally already on their first try.

Another also strange thing I observed is, that initially there are 4 mapped tasks (1-4) queued. But instead of taking one of those initially mapped tasks as a second one to process, it will just take another one (the 5th) leaving the other ones waiting. Could it be, that this is causing the issue: like some kind of time out?

[PowerToThePeople111]

This is a screenshot of the run I do atm.

[PowerToThePeople111]

Ok, observing it even further, I found that exactly the tasks 1,3 and 4 which have been initially queued, but not been executed, are now marked as "up for retry" and will probably fail later on. My hunge is, that if your max_active_tis_per_dagrun is bigger than the number of workers that can be spawned (I guess infra started to limit me ;) ) to actually run in parallel, this can cause mapped tasks to fail.

My current approach is to set max_active_tis_per_dagrun to 1, forcing airflow to do things step by step.