Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] Add run_after column to DagRun model #45732

Draft
wants to merge 4 commits into
base: main
Choose a base branch
from

Conversation

uranusjr
Copy link
Member

@uranusjr uranusjr commented Jan 17, 2025

See AIP-83 amendment to support classic Airflow authoring style and Option 2 clarification doc WIP (Question 3) for context.

This is one of the less controversial approaches to solve ordering and run ID generation for runs with a null logical date.

Depending on the run type, the value is set to:

  • BACKFILL_JOB: DagRunInfo.run_after
  • SCHEDULED: DagModel.next_run_create_after (this is the same as backfill, only calculated earlier in the scheduler loop and stored in db)
  • MANUAL: data_interval.end
  • ASSET_TRIGGERED: Creation time of the last triggering asset event

@boring-cyborg boring-cyborg bot added area:API Airflow's REST/HTTP API area:CLI area:db-migrations PRs with DB migration area:Scheduler including HA (high availability) scheduler kind:documentation labels Jan 17, 2025
@uranusjr uranusjr added the legacy api Whether legacy API changes should be allowed in PR label Jan 17, 2025
@uranusjr uranusjr force-pushed the dagrun-run-after branch 4 times, most recently from 5404c72 to 3d72e45 Compare January 17, 2025 10:06
Depending on the run type, the value is set to:

* BACKFILL_JOB: DagRunInfo.run_after
* SCHEDULED: DagModel.next_run_create_after (this is the same as the
  backfill variant, only calculated pre-emptively and stored in db)
* MANUAL: data_interval.end
* ASSET_TRIGGERED: Creation time of the last triggering asset event
@uranusjr uranusjr force-pushed the dagrun-run-after branch 2 times, most recently from 68ad56d to ad0510f Compare January 17, 2025 14:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:API Airflow's REST/HTTP API area:CLI area:db-migrations PRs with DB migration area:Scheduler including HA (high availability) scheduler kind:documentation legacy api Whether legacy API changes should be allowed in PR
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant