Skip to content

Conversation

@charlie-costanzo
Copy link
Member

Description

During this month's NTD ridership report generation, it was discovered that the most recent data wasn't available in the warehouse due to an order of operations conflict related to the scheduling of the sync_ntd_api dag – It was previously only running on the first of the month.

This PR modifies the schedule to run every Wednesday, as per @csuyat-dot's recommendation, so that data is more likely to be available when expected.

Resolves #4533

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation

How has this been tested?

n/a

Post-merge follow-ups

  • No action required

@charlie-costanzo charlie-costanzo self-assigned this Dec 5, 2025
@charlie-costanzo charlie-costanzo added the product: ntd reporting Items related to ntd reporting pipeline. Product owner is Christian. label Dec 5, 2025
@github-actions
Copy link

github-actions bot commented Dec 5, 2025

Terraform plan in iac/cal-itp-data-infra/composer/us

No changes. Your infrastructure matches the configuration.
No changes. Your infrastructure matches the configuration.

Terraform has compared your real infrastructure against your configuration
and found no differences, so no changes are needed.

📝 Plan generated in Plan Terraform for Warehouse and DAG changes #1157

@github-actions
Copy link

github-actions bot commented Dec 5, 2025

Terraform plan in iac/cal-itp-data-infra/airflow/us

Plan: 0 to add, 1 to change, 0 to destroy.
Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
!~  update in-place

Terraform will perform the following actions:

  # google_storage_bucket_object.calitp-composer["dags/sync_ntd_data_api/METADATA.yml"] will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-composer" {
!~      crc32c              = "dz7uzA==" -> (known after apply)
!~      detect_md5hash      = "BIiiuffNcTJ2hEwgHT9kFg==" -> "different hash"
!~      generation          = 1764100054823250 -> (known after apply)
        id                  = "calitp-composer-dags/sync_ntd_data_api/METADATA.yml"
!~      md5hash             = "BIiiuffNcTJ2hEwgHT9kFg==" -> (known after apply)
        name                = "dags/sync_ntd_data_api/METADATA.yml"
#        (17 unchanged attributes hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

📝 Plan generated in Plan Terraform for Warehouse and DAG changes #1157

@github-actions
Copy link

github-actions bot commented Dec 5, 2025

Terraform plan in iac/cal-itp-data-infra-staging/airflow/us

Plan: 0 to add, 1 to change, 0 to destroy.
Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
!~  update in-place

Terraform will perform the following actions:

  # google_storage_bucket_object.calitp-staging-composer["dags/sync_ntd_data_api/METADATA.yml"] will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-staging-composer" {
!~      crc32c              = "dz7uzA==" -> (known after apply)
!~      detect_md5hash      = "BIiiuffNcTJ2hEwgHT9kFg==" -> "different hash"
!~      generation          = 1764100045161066 -> (known after apply)
        id                  = "calitp-staging-composer-dags/sync_ntd_data_api/METADATA.yml"
!~      md5hash             = "BIiiuffNcTJ2hEwgHT9kFg==" -> (known after apply)
        name                = "dags/sync_ntd_data_api/METADATA.yml"
#        (17 unchanged attributes hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

📝 Plan generated in Plan Terraform for Warehouse and DAG changes #1157

@erikamov erikamov force-pushed the pipeline-run-ntd-api-dag-weekly branch from 8847f27 to b1b2bc5 Compare December 9, 2025 20:18
@erikamov erikamov merged commit 702c084 into main Dec 9, 2025
8 checks passed
@erikamov erikamov deleted the pipeline-run-ntd-api-dag-weekly branch December 9, 2025 20:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

product: ntd reporting Items related to ntd reporting pipeline. Product owner is Christian.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Change update cadence of Monthly NTD Ridership table to weekly

3 participants