Tasks with a newer logical_date are never scheduled to run when max_active_tis_per_dag is used

[franzwong]

Due to issues with the external data provider, newer data may be available before older data. The order of the data is not a concern for me, so I allow the tasks to keep retrying throughout the day. I have limited the number of task instances. However, tasks from newer DAG runs are never executed because Airflow always prioritizes tasks from older DAG runs.

I created a simple example to reproduce this. There are 6 active DAG runs. Airflow only executes tasks from the first 4 DAG runs and never executes tasks from the remaining 2 DAG runs. I tried using a pool with 2 slots and 6 DAGs, but the result is the same.

Any thoughts on this? Do I need to implement a custom scheduler? Thanks.

import pendulum
import time

from airflow import DAG
from airflow.providers.standard.operators.python import PythonOperator
from airflow.sdk.exceptions import AirflowException


def example(**context):
    print("Before sleep")
    time.sleep(30)
    print("After sleep")
    raise AirflowException("Testing")


with DAG(
    dag_id="example_dag",
    start_date=pendulum.datetime(2026, 5, 1, 0, 0, tz="UTC"),
    schedule="* * * * *",
    catchup=False,
    max_active_runs=6,
    is_paused_upon_creation=False,
    tags=["example"],
) as dag:
    PythonOperator(
        task_id="example",
        python_callable=example,
        retries=300,
        retry_delay=pendulum.duration(seconds=30),
        max_active_tis_per_dag=2,
    )