Skip to content

Littlepay refund handling fix #4572

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
lauriemerrell wants to merge 5 commits into
base: main
Choose a base branch
from
Draft

Littlepay refund handling fix #4572

lauriemerrell wants to merge 5 commits into from

Conversation

@lauriemerrell
Copy link
Contributor

@lauriemerrell lauriemerrell commented Dec 9, 2025 •
edited
Loading

Description

Describe your changes and why you're making them. Please include the context, motivation, and relevant dependencies.

Resolves #3972

  • Updates refunds to handle a schema change in v3 feeds where refund_amount now means what you'd think it should -- one item out of Payments: Littlepay v3 breaking changes #4581
  • Fixes what I think is a bug in the existing refunds logic where we were probably deduping too aggressively by not including micropayment_id in the consideration for duplicates -- from looking at the data, seems like if two rows have non-null differing micropayment IDs, they should be considered actually distinct -- this results in an increase in count of rows being preserved
  • Refactors the model to use an outer join instead of a union so that rows that appear in both micropayments + refunds are handled correctly

TODO:

  • Pick columns that we want to keep from refunds vs. micropayments
  • Fully assess the impact of these changes by comparing resulting table with prod -- example query below
with stg as (
  SELECT * FROM `cal-itp-data-infra-staging.laurie_staging.int_payments__refunds_deduped`),


prd as (
  SELECT * FROM `cal-itp-data-infra.staging.int_payments__refunds_deduped`
),

compare as (

select 
  coalesce(stg.refunds_key, stg.micropayments_key) is null as missing_stg,
  prd._key is null as missing_prd,
  coalesce(stg.aggregation_id, prd.aggregation_id) as aggregation_id,
  coalesce(stg.micropayment_id, prd.micropayment_id) as micropayment_id,
  coalesce(stg.participant_id, prd.participant_id) as participant_id,
  coalesce(stg.proposed_amount, prd.proposed_amount) as proposed_amount,
  coalesce(stg.transaction_date, prd.transaction_date) as transaction_date,
  coalesce(stg.refunds_refund_amount, stg.micropayment_refund_amount, prd.refund_amount) as refund_amount,
  coalesce(stg.refund_id, prd.refund_id) as refund_id,
  coalesce(stg.coalesced_id, prd.coalesced_id) as coalesced_id,
  stg.source_table,
  prd._key as prod_key,
  stg.refunds_key,
  stg.micropayments_key

from stg
full outer join prd
on coalesce(stg.micropayments_key, stg.refunds_key) = prd._key
where coalesce(stg.micropayments_key, stg.refunds_key)  is null or prd._key is null
order by aggregation_id ),

-- this is an intentional change being made so ignore discrepancies from this
check_multiple_micropayments as (
  select 
    aggregation_id,
    count(distinct micropayment_id) as ct
  from `cal-itp-data-infra.staging.int_littlepay__unioned_refunds`
  group by 1
  having ct >1
)

select compare.*
from compare 
left join check_multiple_micropayments
on compare.aggregation_id = check_multiple_micropayments.aggregation_id 
where check_multiple_micropayments.aggregation_id is null and missing_prd
order by transaction_date desc

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation

How has this been tested?

Include commands/logs/screenshots as relevant.

If making changes to dbt models, make sure they were created or update on Staging. Please run the command poetry run dbt run -s CHANGED_MODEL --target staging and poetry run dbt test -s CHANGED_MODEL --target staging, then include the output in this section of the PR.

Post-merge follow-ups

Document any actions that must be taken post-merge to deploy or otherwise implement the changes in this PR (for example, running a full refresh of some incremental model in dbt). If these actions will take more than a few hours after the merge or if they will be completed by someone other than the PR author, please create a dedicated follow-up issue and link it here to track resolution.

  • No action required
  • Actions required (specified below)

@github-actions
Copy link

github-actions bot commented Dec 9, 2025 •
edited
Loading

Terraform plan in iac/cal-itp-data-infra/airflow/us

Plan: 0 to add, 5 to change, 0 to destroy. 
Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
!~  update in-place

Terraform will perform the following actions:

  # google_storage_bucket_object.calitp-composer-dags["models/intermediate/payments/_int_payments.yml"] will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-composer-dags" {
!~      crc32c              = "wODMcA==" -> (known after apply)
!~      detect_md5hash      = "tuYyCRja13CEhEblQn7u/A==" -> "different hash"
!~      generation          = 1751416667352298 -> (known after apply)
        id                  = "calitp-composer-data/warehouse/models/intermediate/payments/_int_payments.yml"
!~      md5hash             = "tuYyCRja13CEhEblQn7u/A==" -> (known after apply)
        name                = "data/warehouse/models/intermediate/payments/_int_payments.yml"
#        (17 unchanged attributes hidden)
    }

  # google_storage_bucket_object.calitp-composer-dags["models/intermediate/payments/int_payments__micropayments_adjustments_refunds_joined.sql"] will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-composer-dags" {
!~      crc32c              = "XSqIkw==" -> (known after apply)
!~      detect_md5hash      = "elr0bcpIW6u8mBzWySRNdw==" -> "different hash"
!~      generation          = 1751416665330335 -> (known after apply)
        id                  = "calitp-composer-data/warehouse/models/intermediate/payments/int_payments__micropayments_adjustments_refunds_joined.sql"
!~      md5hash             = "elr0bcpIW6u8mBzWySRNdw==" -> (known after apply)
        name                = "data/warehouse/models/intermediate/payments/int_payments__micropayments_adjustments_refunds_joined.sql"
#        (17 unchanged attributes hidden)
    }

  # google_storage_bucket_object.calitp-composer-dags["models/intermediate/payments/int_payments__refunds_deduped.sql"] will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-composer-dags" {
!~      crc32c              = "FP22ww==" -> (known after apply)
!~      detect_md5hash      = "M7ioJDOK4YrQB76AvfEJAw==" -> "different hash"
!~      generation          = 1751416666597245 -> (known after apply)
        id                  = "calitp-composer-data/warehouse/models/intermediate/payments/int_payments__refunds_deduped.sql"
!~      md5hash             = "M7ioJDOK4YrQB76AvfEJAw==" -> (known after apply)
        name                = "data/warehouse/models/intermediate/payments/int_payments__refunds_deduped.sql"
#        (17 unchanged attributes hidden)
    }

  # google_storage_bucket_object.calitp-composer-dags["models/intermediate/payments/int_payments__refunds_to_aggregations.sql"] will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-composer-dags" {
!~      crc32c              = "HxjT6g==" -> (known after apply)
!~      detect_md5hash      = "zZIja39dr1TtsxxCA/4kIg==" -> "different hash"
!~      generation          = 1751416670127544 -> (known after apply)
        id                  = "calitp-composer-data/warehouse/models/intermediate/payments/int_payments__refunds_to_aggregations.sql"
!~      md5hash             = "zZIja39dr1TtsxxCA/4kIg==" -> (known after apply)
        name                = "data/warehouse/models/intermediate/payments/int_payments__refunds_to_aggregations.sql"
#        (17 unchanged attributes hidden)
    }

  # google_storage_bucket_object.calitp-composer-dags["models/intermediate/payments/littlepay_feed_unions/int_littlepay__unioned_refunds.sql"] will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-composer-dags" {
!~      crc32c              = "L8ggEw==" -> (known after apply)
!~      detect_md5hash      = "glUiaPQIv7b7qMen0eNG5w==" -> "different hash"
!~      generation          = 1760473616986578 -> (known after apply)
        id                  = "calitp-composer-data/warehouse/models/intermediate/payments/littlepay_feed_unions/int_littlepay__unioned_refunds.sql"
!~      md5hash             = "glUiaPQIv7b7qMen0eNG5w==" -> (known after apply)
        name                = "data/warehouse/models/intermediate/payments/littlepay_feed_unions/int_littlepay__unioned_refunds.sql"
#        (17 unchanged attributes hidden)
    }

Plan: 0 to add, 5 to change, 0 to destroy.

📝 Plan generated in Plan Terraform for Warehouse and DAG changes #1191

@github-actions
Copy link

github-actions bot commented Dec 9, 2025 •
edited
Loading

Terraform plan in iac/cal-itp-data-infra-staging/airflow/us

Plan: 0 to add, 15 to change, 9 to destroy. 
Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
!~  update in-place
-   destroy

Terraform will perform the following actions:

  # google_storage_bucket_object.calitp-staging-composer["dags/README.md"] will be destroyed
  # (because key ["dags/README.md"] is not in for_each map)
-   resource "google_storage_bucket_object" "calitp-staging-composer" {
-       bucket              = "calitp-staging-composer" -> null
-       content_type        = "text/plain; charset=utf-8" -> null
-       crc32c              = "Uf9ILA==" -> null
-       detect_md5hash      = "7p0s8Je/yxIv8Fxq5ApeJA==" -> null
-       event_based_hold    = false -> null
-       generation          = 1765492313258132 -> null
-       id                  = "calitp-staging-composer-dags/README.md" -> null
-       md5hash             = "7p0s8Je/yxIv8Fxq5ApeJA==" -> null
-       md5hexhash          = "ee9d2cf097bfcb122ff05c6ae40a5e24" -> null
-       media_link          = "https://storage.googleapis.com/download/storage/v1/b/calitp-staging-composer/o/dags%2FREADME.md?generation=1765492313258132&alt=media" -> null
-       metadata            = {} -> null
-       name                = "dags/README.md" -> null
-       output_name         = "dags/README.md" -> null
-       self_link           = "https://www.googleapis.com/storage/v1/b/calitp-staging-composer/o/dags%2FREADME.md" -> null
-       source              = "../../../../airflow/dags/README.md" -> null
-       storage_class       = "STANDARD" -> null
-       temporary_hold      = false -> null
#        (6 unchanged attributes hidden)
    }

  # google_storage_bucket_object.calitp-staging-composer["dags/create_external_tables/METADATA.yml"] will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-staging-composer" {
!~      crc32c              = "yCFz/w==" -> (known after apply)
!~      detect_md5hash      = "lShsRtpkUR8bc1rXUwlqcA==" -> "different hash"
!~      generation          = 1765492313229307 -> (known after apply)
        id                  = "calitp-staging-composer-dags/create_external_tables/METADATA.yml"
!~      md5hash             = "lShsRtpkUR8bc1rXUwlqcA==" -> (known after apply)
        name                = "dags/create_external_tables/METADATA.yml"
#        (17 unchanged attributes hidden)
    }

  # google_storage_bucket_object.calitp-staging-composer["dags/download_parse_and_validate_gtfs.py"] will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-staging-composer" {
!~      crc32c              = "GvruJA==" -> (known after apply)
!~      detect_md5hash      = "EVCNHN6Hq0uodzYuK8Zg4A==" -> "different hash"
!~      generation          = 1765494172783784 -> (known after apply)
        id                  = "calitp-staging-composer-dags/download_parse_and_validate_gtfs.py"
!~      md5hash             = "EVCNHN6Hq0uodzYuK8Zg4A==" -> (known after apply)
        name                = "dags/download_parse_and_validate_gtfs.py"
#        (17 unchanged attributes hidden)
    }

  # google_storage_bucket_object.calitp-staging-composer["dags/sync_ntd_data_api/METADATA.yml"] will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-staging-composer" {
!~      crc32c              = "W2xk2w==" -> (known after apply)
!~      detect_md5hash      = "MKjOm+rmkOPQDnCVQhJnDg==" -> "different hash"
!~      generation          = 1765492313215650 -> (known after apply)
        id                  = "calitp-staging-composer-dags/sync_ntd_data_api/METADATA.yml"
!~      md5hash             = "MKjOm+rmkOPQDnCVQhJnDg==" -> (known after apply)
        name                = "dags/sync_ntd_data_api/METADATA.yml"
#        (17 unchanged attributes hidden)
    }

  # google_storage_bucket_object.calitp-staging-composer["plugins/operators/bigquery_to_download_config_operator.py"] will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-staging-composer" {
!~      crc32c              = "6oRS3A==" -> (known after apply)
!~      detect_md5hash      = "xxkAT0jhkh3LKcuWUQ40AA==" -> "different hash"
!~      generation          = 1765492313306019 -> (known after apply)
        id                  = "calitp-staging-composer-plugins/operators/bigquery_to_download_config_operator.py"
!~      md5hash             = "xxkAT0jhkh3LKcuWUQ40AA==" -> (known after apply)
        name                = "plugins/operators/bigquery_to_download_config_operator.py"
#        (17 unchanged attributes hidden)
    }

  # google_storage_bucket_object.calitp-staging-composer["plugins/operators/gcs_to_gtfs_download_operator.py"] will be destroyed
  # (because key ["plugins/operators/gcs_to_gtfs_download_operator.py"] is not in for_each map)
-   resource "google_storage_bucket_object" "calitp-staging-composer" {
-       bucket              = "calitp-staging-composer" -> null
-       content_type        = "text/plain; charset=utf-8" -> null
-       crc32c              = "VYvt6w==" -> null
-       detect_md5hash      = "JUKhbExF+25ekmHo/QMIlg==" -> null
-       event_based_hold    = false -> null
-       generation          = 1765492313242286 -> null
-       id                  = "calitp-staging-composer-plugins/operators/gcs_to_gtfs_download_operator.py" -> null
-       md5hash             = "JUKhbExF+25ekmHo/QMIlg==" -> null
-       md5hexhash          = "2542a16c4c45fb6e5e9261e8fd030896" -> null
-       media_link          = "https://storage.googleapis.com/download/storage/v1/b/calitp-staging-composer/o/plugins%2Foperators%2Fgcs_to_gtfs_download_operator.py?generation=1765492313242286&alt=media" -> null
-       metadata            = {} -> null
-       name                = "plugins/operators/gcs_to_gtfs_download_operator.py" -> null
-       output_name         = "plugins/operators/gcs_to_gtfs_download_operator.py" -> null
-       self_link           = "https://www.googleapis.com/storage/v1/b/calitp-staging-composer/o/plugins%2Foperators%2Fgcs_to_gtfs_download_operator.py" -> null
-       source              = "../../../../airflow/plugins/operators/gcs_to_gtfs_download_operator.py" -> null
-       storage_class       = "STANDARD" -> null
-       temporary_hold      = false -> null
#        (6 unchanged attributes hidden)
    }

  # google_storage_bucket_object.calitp-staging-composer-catalog will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-staging-composer-catalog" {
!~      content             = (sensitive value)
!~      crc32c              = "C6pTUQ==" -> (known after apply)
!~      detect_md5hash      = "G1/zRN8BpLbORlo3Fq9mFQ==" -> "different hash"
!~      generation          = 1765493898541075 -> (known after apply)
        id                  = "calitp-staging-composer-data/warehouse/target/catalog.json"
!~      md5hash             = "G1/zRN8BpLbORlo3Fq9mFQ==" -> (known after apply)
        name                = "data/warehouse/target/catalog.json"
#        (16 unchanged attributes hidden)
    }

  # google_storage_bucket_object.calitp-staging-composer-dags["dbt_project.yml"] will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-staging-composer-dags" {
!~      crc32c              = "0nOaVQ==" -> (known after apply)
!~      detect_md5hash      = "J2zB6Wiqs9hDtqVOQnswlQ==" -> "different hash"
!~      generation          = 1765492314099631 -> (known after apply)
        id                  = "calitp-staging-composer-data/warehouse/dbt_project.yml"
!~      md5hash             = "J2zB6Wiqs9hDtqVOQnswlQ==" -> (known after apply)
        name                = "data/warehouse/dbt_project.yml"
#        (17 unchanged attributes hidden)
    }

  # google_storage_bucket_object.calitp-staging-composer-dags["models/intermediate/gtfs/_int_gtfs.yaml"] will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-staging-composer-dags" {
!~      crc32c              = "oUwGLw==" -> (known after apply)
!~      detect_md5hash      = "gC+bYu7qim9PuuJilF7+Iw==" -> "different hash"
!~      generation          = 1765492313208215 -> (known after apply)
        id                  = "calitp-staging-composer-data/warehouse/models/intermediate/gtfs/_int_gtfs.yaml"
!~      md5hash             = "gC+bYu7qim9PuuJilF7+Iw==" -> (known after apply)
        name                = "data/warehouse/models/intermediate/gtfs/_int_gtfs.yaml"
#        (17 unchanged attributes hidden)
    }

  # google_storage_bucket_object.calitp-staging-composer-dags["models/intermediate/gtfs/int_gtfs_datasets.sql"] will be destroyed
  # (because key ["models/intermediate/gtfs/int_gtfs_datasets.sql"] is not in for_each map)
-   resource "google_storage_bucket_object" "calitp-staging-composer-dags" {
-       bucket              = "calitp-staging-composer" -> null
-       content_type        = "text/plain; charset=utf-8" -> null
-       crc32c              = "KlXP/A==" -> null
-       detect_md5hash      = "vAQ3Hq6XQOd494WuB4Kpeg==" -> null
-       event_based_hold    = false -> null
-       generation          = 1765492314062235 -> null
-       id                  = "calitp-staging-composer-data/warehouse/models/intermediate/gtfs/int_gtfs_datasets.sql" -> null
-       md5hash             = "vAQ3Hq6XQOd494WuB4Kpeg==" -> null
-       md5hexhash          = "bc04371eae9740e778f785ae0782a97a" -> null
-       media_link          = "https://storage.googleapis.com/download/storage/v1/b/calitp-staging-composer/o/data%2Fwarehouse%2Fmodels%2Fintermediate%2Fgtfs%2Fint_gtfs_datasets.sql?generation=1765492314062235&alt=media" -> null
-       metadata            = {} -> null
-       name                = "data/warehouse/models/intermediate/gtfs/int_gtfs_datasets.sql" -> null
-       output_name         = "data/warehouse/models/intermediate/gtfs/int_gtfs_datasets.sql" -> null
-       self_link           = "https://www.googleapis.com/storage/v1/b/calitp-staging-composer/o/data%2Fwarehouse%2Fmodels%2Fintermediate%2Fgtfs%2Fint_gtfs_datasets.sql" -> null
-       source              = "../../../../warehouse/models/intermediate/gtfs/int_gtfs_datasets.sql" -> null
-       storage_class       = "STANDARD" -> null
-       temporary_hold      = false -> null
#        (6 unchanged attributes hidden)
    }

  # google_storage_bucket_object.calitp-staging-composer-dags["models/intermediate/payments/_int_payments.yml"] will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-staging-composer-dags" {
!~      crc32c              = "wODMcA==" -> (known after apply)
!~      detect_md5hash      = "tuYyCRja13CEhEblQn7u/A==" -> "different hash"
!~      generation          = 1749663116399571 -> (known after apply)
        id                  = "calitp-staging-composer-data/warehouse/models/intermediate/payments/_int_payments.yml"
!~      md5hash             = "tuYyCRja13CEhEblQn7u/A==" -> (known after apply)
        name                = "data/warehouse/models/intermediate/payments/_int_payments.yml"
#        (17 unchanged attributes hidden)
    }

  # google_storage_bucket_object.calitp-staging-composer-dags["models/intermediate/payments/int_payments__micropayments_adjustments_refunds_joined.sql"] will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-staging-composer-dags" {
!~      crc32c              = "XSqIkw==" -> (known after apply)
!~      detect_md5hash      = "elr0bcpIW6u8mBzWySRNdw==" -> "different hash"
!~      generation          = 1749663117432564 -> (known after apply)
        id                  = "calitp-staging-composer-data/warehouse/models/intermediate/payments/int_payments__micropayments_adjustments_refunds_joined.sql"
!~      md5hash             = "elr0bcpIW6u8mBzWySRNdw==" -> (known after apply)
        name                = "data/warehouse/models/intermediate/payments/int_payments__micropayments_adjustments_refunds_joined.sql"
#        (17 unchanged attributes hidden)
    }

  # google_storage_bucket_object.calitp-staging-composer-dags["models/intermediate/payments/int_payments__refunds_deduped.sql"] will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-staging-composer-dags" {
!~      crc32c              = "FP22ww==" -> (known after apply)
!~      detect_md5hash      = "M7ioJDOK4YrQB76AvfEJAw==" -> "different hash"
!~      generation          = 1749663116630393 -> (known after apply)
        id                  = "calitp-staging-composer-data/warehouse/models/intermediate/payments/int_payments__refunds_deduped.sql"
!~      md5hash             = "M7ioJDOK4YrQB76AvfEJAw==" -> (known after apply)
        name                = "data/warehouse/models/intermediate/payments/int_payments__refunds_deduped.sql"
#        (17 unchanged attributes hidden)
    }

  # google_storage_bucket_object.calitp-staging-composer-dags["models/intermediate/payments/int_payments__refunds_to_aggregations.sql"] will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-staging-composer-dags" {
!~      crc32c              = "HxjT6g==" -> (known after apply)
!~      detect_md5hash      = "zZIja39dr1TtsxxCA/4kIg==" -> "different hash"
!~      generation          = 1749663116775029 -> (known after apply)
        id                  = "calitp-staging-composer-data/warehouse/models/intermediate/payments/int_payments__refunds_to_aggregations.sql"
!~      md5hash             = "zZIja39dr1TtsxxCA/4kIg==" -> (known after apply)
        name                = "data/warehouse/models/intermediate/payments/int_payments__refunds_to_aggregations.sql"
#        (17 unchanged attributes hidden)
    }

  # google_storage_bucket_object.calitp-staging-composer-dags["models/intermediate/payments/littlepay_feed_unions/int_littlepay__unioned_refunds.sql"] will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-staging-composer-dags" {
!~      crc32c              = "L8ggEw==" -> (known after apply)
!~      detect_md5hash      = "glUiaPQIv7b7qMen0eNG5w==" -> "different hash"
!~      generation          = 1760473645067313 -> (known after apply)
        id                  = "calitp-staging-composer-data/warehouse/models/intermediate/payments/littlepay_feed_unions/int_littlepay__unioned_refunds.sql"
!~      md5hash             = "glUiaPQIv7b7qMen0eNG5w==" -> (known after apply)
        name                = "data/warehouse/models/intermediate/payments/littlepay_feed_unions/int_littlepay__unioned_refunds.sql"
#        (17 unchanged attributes hidden)
    }

  # google_storage_bucket_object.calitp-staging-composer-dags["models/intermediate/transit_database/dimensions/int_transit_database__gtfs_datasets_dim.sql"] will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-staging-composer-dags" {
!~      crc32c              = "AgIGYg==" -> (known after apply)
!~      detect_md5hash      = "+4puEf/Ho738P+fD579aiQ==" -> "different hash"
!~      generation          = 1765492314067263 -> (known after apply)
        id                  = "calitp-staging-composer-data/warehouse/models/intermediate/transit_database/dimensions/int_transit_database__gtfs_datasets_dim.sql"
!~      md5hash             = "+4puEf/Ho738P+fD579aiQ==" -> (known after apply)
        name                = "data/warehouse/models/intermediate/transit_database/dimensions/int_transit_database__gtfs_datasets_dim.sql"
#        (17 unchanged attributes hidden)
    }

  # google_storage_bucket_object.calitp-staging-composer-dags["models/mart/gtfs_audit/_mart_gtfs_audit.yml"] will be destroyed
  # (because key ["models/mart/gtfs_audit/_mart_gtfs_audit.yml"] is not in for_each map)
-   resource "google_storage_bucket_object" "calitp-staging-composer-dags" {
-       bucket              = "calitp-staging-composer" -> null
-       content_type        = "text/plain; charset=utf-8" -> null
-       crc32c              = "x1tlwA==" -> null
-       detect_md5hash      = "/GVdwrikyWDPvld4OYkPbg==" -> null
-       event_based_hold    = false -> null
-       generation          = 1765492314118753 -> null
-       id                  = "calitp-staging-composer-data/warehouse/models/mart/gtfs_audit/_mart_gtfs_audit.yml" -> null
-       md5hash             = "/GVdwrikyWDPvld4OYkPbg==" -> null
-       md5hexhash          = "fc655dc2b8a4c960cfbe577839890f6e" -> null
-       media_link          = "https://storage.googleapis.com/download/storage/v1/b/calitp-staging-composer/o/data%2Fwarehouse%2Fmodels%2Fmart%2Fgtfs_audit%2F_mart_gtfs_audit.yml?generation=1765492314118753&alt=media" -> null
-       metadata            = {} -> null
-       name                = "data/warehouse/models/mart/gtfs_audit/_mart_gtfs_audit.yml" -> null
-       output_name         = "data/warehouse/models/mart/gtfs_audit/_mart_gtfs_audit.yml" -> null
-       self_link           = "https://www.googleapis.com/storage/v1/b/calitp-staging-composer/o/data%2Fwarehouse%2Fmodels%2Fmart%2Fgtfs_audit%2F_mart_gtfs_audit.yml" -> null
-       source              = "../../../../warehouse/models/mart/gtfs_audit/_mart_gtfs_audit.yml" -> null
-       storage_class       = "STANDARD" -> null
-       temporary_hold      = false -> null
#        (6 unchanged attributes hidden)
    }

  # google_storage_bucket_object.calitp-staging-composer-dags["models/mart/gtfs_audit/dim_gtfs_download_configs.sql"] will be destroyed
  # (because key ["models/mart/gtfs_audit/dim_gtfs_download_configs.sql"] is not in for_each map)
-   resource "google_storage_bucket_object" "calitp-staging-composer-dags" {
-       bucket              = "calitp-staging-composer" -> null
-       content_type        = "text/plain; charset=utf-8" -> null
-       crc32c              = "Ba+u1Q==" -> null
-       detect_md5hash      = "/xVeNfCCbDEv584sC8VEjQ==" -> null
-       event_based_hold    = false -> null
-       generation          = 1765492313308128 -> null
-       id                  = "calitp-staging-composer-data/warehouse/models/mart/gtfs_audit/dim_gtfs_download_configs.sql" -> null
-       md5hash             = "/xVeNfCCbDEv584sC8VEjQ==" -> null
-       md5hexhash          = "ff155e35f0826c312fe7ce2c0bc5448d" -> null
-       media_link          = "https://storage.googleapis.com/download/storage/v1/b/calitp-staging-composer/o/data%2Fwarehouse%2Fmodels%2Fmart%2Fgtfs_audit%2Fdim_gtfs_download_configs.sql?generation=1765492313308128&alt=media" -> null
-       metadata            = {} -> null
-       name                = "data/warehouse/models/mart/gtfs_audit/dim_gtfs_download_configs.sql" -> null
-       output_name         = "data/warehouse/models/mart/gtfs_audit/dim_gtfs_download_configs.sql" -> null
-       self_link           = "https://www.googleapis.com/storage/v1/b/calitp-staging-composer/o/data%2Fwarehouse%2Fmodels%2Fmart%2Fgtfs_audit%2Fdim_gtfs_download_configs.sql" -> null
-       source              = "../../../../warehouse/models/mart/gtfs_audit/dim_gtfs_download_configs.sql" -> null
-       storage_class       = "STANDARD" -> null
-       temporary_hold      = false -> null
#        (6 unchanged attributes hidden)
    }

  # google_storage_bucket_object.calitp-staging-composer-dags["models/mart/gtfs_audit/dim_gtfs_schedule_download_outcomes.sql"] will be destroyed
  # (because key ["models/mart/gtfs_audit/dim_gtfs_schedule_download_outcomes.sql"] is not in for_each map)
-   resource "google_storage_bucket_object" "calitp-staging-composer-dags" {
-       bucket              = "calitp-staging-composer" -> null
-       content_type        = "text/plain; charset=utf-8" -> null
-       crc32c              = "rOnE3Q==" -> null
-       detect_md5hash      = "XNAaQIuES8NX0CJ6gD3tkg==" -> null
-       event_based_hold    = false -> null
-       generation          = 1765492313826250 -> null
-       id                  = "calitp-staging-composer-data/warehouse/models/mart/gtfs_audit/dim_gtfs_schedule_download_outcomes.sql" -> null
-       md5hash             = "XNAaQIuES8NX0CJ6gD3tkg==" -> null
-       md5hexhash          = "5cd01a408b844bc357d0227a803ded92" -> null
-       media_link          = "https://storage.googleapis.com/download/storage/v1/b/calitp-staging-composer/o/data%2Fwarehouse%2Fmodels%2Fmart%2Fgtfs_audit%2Fdim_gtfs_schedule_download_outcomes.sql?generation=1765492313826250&alt=media" -> null
-       metadata            = {} -> null
-       name                = "data/warehouse/models/mart/gtfs_audit/dim_gtfs_schedule_download_outcomes.sql" -> null
-       output_name         = "data/warehouse/models/mart/gtfs_audit/dim_gtfs_schedule_download_outcomes.sql" -> null
-       self_link           = "https://www.googleapis.com/storage/v1/b/calitp-staging-composer/o/data%2Fwarehouse%2Fmodels%2Fmart%2Fgtfs_audit%2Fdim_gtfs_schedule_download_outcomes.sql" -> null
-       source              = "../../../../warehouse/models/mart/gtfs_audit/dim_gtfs_schedule_download_outcomes.sql" -> null
-       storage_class       = "STANDARD" -> null
-       temporary_hold      = false -> null
#        (6 unchanged attributes hidden)
    }

  # google_storage_bucket_object.calitp-staging-composer-dags["models/mart/gtfs_audit/dim_gtfs_schedule_unzip_outcomes.sql"] will be destroyed
  # (because key ["models/mart/gtfs_audit/dim_gtfs_schedule_unzip_outcomes.sql"] is not in for_each map)
-   resource "google_storage_bucket_object" "calitp-staging-composer-dags" {
-       bucket              = "calitp-staging-composer" -> null
-       content_type        = "text/plain; charset=utf-8" -> null
-       crc32c              = "hMU1HQ==" -> null
-       detect_md5hash      = "G2++RVfcKDiSCGGLCc3sDw==" -> null
-       event_based_hold    = false -> null
-       generation          = 1765492313318672 -> null
-       id                  = "calitp-staging-composer-data/warehouse/models/mart/gtfs_audit/dim_gtfs_schedule_unzip_outcomes.sql" -> null
-       md5hash             = "G2++RVfcKDiSCGGLCc3sDw==" -> null
-       md5hexhash          = "1b6fbe4557dc28389208618b09cdec0f" -> null
-       media_link          = "https://storage.googleapis.com/download/storage/v1/b/calitp-staging-composer/o/data%2Fwarehouse%2Fmodels%2Fmart%2Fgtfs_audit%2Fdim_gtfs_schedule_unzip_outcomes.sql?generation=1765492313318672&alt=media" -> null
-       metadata            = {} -> null
-       name                = "data/warehouse/models/mart/gtfs_audit/dim_gtfs_schedule_unzip_outcomes.sql" -> null
-       output_name         = "data/warehouse/models/mart/gtfs_audit/dim_gtfs_schedule_unzip_outcomes.sql" -> null
-       self_link           = "https://www.googleapis.com/storage/v1/b/calitp-staging-composer/o/data%2Fwarehouse%2Fmodels%2Fmart%2Fgtfs_audit%2Fdim_gtfs_schedule_unzip_outcomes.sql" -> null
-       source              = "../../../../warehouse/models/mart/gtfs_audit/dim_gtfs_schedule_unzip_outcomes.sql" -> null
-       storage_class       = "STANDARD" -> null
-       temporary_hold      = false -> null
#        (6 unchanged attributes hidden)
    }

  # google_storage_bucket_object.calitp-staging-composer-dags["models/mart/gtfs_audit/dim_gtfs_schedule_validation_notices.sql"] will be destroyed
  # (because key ["models/mart/gtfs_audit/dim_gtfs_schedule_validation_notices.sql"] is not in for_each map)
-   resource "google_storage_bucket_object" "calitp-staging-composer-dags" {
-       bucket              = "calitp-staging-composer" -> null
-       content_type        = "text/plain; charset=utf-8" -> null
-       crc32c              = "yhy/jw==" -> null
-       detect_md5hash      = "G6+2DgppdGLlWMqoPubR2Q==" -> null
-       event_based_hold    = false -> null
-       generation          = 1765492314072070 -> null
-       id                  = "calitp-staging-composer-data/warehouse/models/mart/gtfs_audit/dim_gtfs_schedule_validation_notices.sql" -> null
-       md5hash             = "G6+2DgppdGLlWMqoPubR2Q==" -> null
-       md5hexhash          = "1bafb60e0a697462e558caa83ee6d1d9" -> null
-       media_link          = "https://storage.googleapis.com/download/storage/v1/b/calitp-staging-composer/o/data%2Fwarehouse%2Fmodels%2Fmart%2Fgtfs_audit%2Fdim_gtfs_schedule_validation_notices.sql?generation=1765492314072070&alt=media" -> null
-       metadata            = {} -> null
-       name                = "data/warehouse/models/mart/gtfs_audit/dim_gtfs_schedule_validation_notices.sql" -> null
-       output_name         = "data/warehouse/models/mart/gtfs_audit/dim_gtfs_schedule_validation_notices.sql" -> null
-       self_link           = "https://www.googleapis.com/storage/v1/b/calitp-staging-composer/o/data%2Fwarehouse%2Fmodels%2Fmart%2Fgtfs_audit%2Fdim_gtfs_schedule_validation_notices.sql" -> null
-       source              = "../../../../warehouse/models/mart/gtfs_audit/dim_gtfs_schedule_validation_notices.sql" -> null
-       storage_class       = "STANDARD" -> null
-       temporary_hold      = false -> null
#        (6 unchanged attributes hidden)
    }

  # google_storage_bucket_object.calitp-staging-composer-dags["models/mart/gtfs_audit/dim_gtfs_schedule_validation_outcomes.sql"] will be destroyed
  # (because key ["models/mart/gtfs_audit/dim_gtfs_schedule_validation_outcomes.sql"] is not in for_each map)
-   resource "google_storage_bucket_object" "calitp-staging-composer-dags" {
-       bucket              = "calitp-staging-composer" -> null
-       content_type        = "text/plain; charset=utf-8" -> null
-       crc32c              = "OGzhrg==" -> null
-       detect_md5hash      = "R7VbFRIIzMqIGbR2a50zgw==" -> null
-       event_based_hold    = false -> null
-       generation          = 1765492313294525 -> null
-       id                  = "calitp-staging-composer-data/warehouse/models/mart/gtfs_audit/dim_gtfs_schedule_validation_outcomes.sql" -> null
-       md5hash             = "R7VbFRIIzMqIGbR2a50zgw==" -> null
-       md5hexhash          = "47b55b151208ccca8819b4766b9d3383" -> null
-       media_link          = "https://storage.googleapis.com/download/storage/v1/b/calitp-staging-composer/o/data%2Fwarehouse%2Fmodels%2Fmart%2Fgtfs_audit%2Fdim_gtfs_schedule_validation_outcomes.sql?generation=1765492313294525&alt=media" -> null
-       metadata            = {} -> null
-       name                = "data/warehouse/models/mart/gtfs_audit/dim_gtfs_schedule_validation_outcomes.sql" -> null
-       output_name         = "data/warehouse/models/mart/gtfs_audit/dim_gtfs_schedule_validation_outcomes.sql" -> null
-       self_link           = "https://www.googleapis.com/storage/v1/b/calitp-staging-composer/o/data%2Fwarehouse%2Fmodels%2Fmart%2Fgtfs_audit%2Fdim_gtfs_schedule_validation_outcomes.sql" -> null
-       source              = "../../../../warehouse/models/mart/gtfs_audit/dim_gtfs_schedule_validation_outcomes.sql" -> null
-       storage_class       = "STANDARD" -> null
-       temporary_hold      = false -> null
#        (6 unchanged attributes hidden)
    }

  # google_storage_bucket_object.calitp-staging-composer-dags["models/staging/gtfs/_src_gtfs_schedule_external_tables.yml"] will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-staging-composer-dags" {
!~      crc32c              = "tFxfsQ==" -> (known after apply)
!~      detect_md5hash      = "cBcIe5336gm/PxfQReI9Cw==" -> "different hash"
!~      generation          = 1765492313722814 -> (known after apply)
        id                  = "calitp-staging-composer-data/warehouse/models/staging/gtfs/_src_gtfs_schedule_external_tables.yml"
!~      md5hash             = "cBcIe5336gm/PxfQReI9Cw==" -> (known after apply)
        name                = "data/warehouse/models/staging/gtfs/_src_gtfs_schedule_external_tables.yml"
#        (17 unchanged attributes hidden)
    }

  # google_storage_bucket_object.calitp-staging-composer-manifest will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-staging-composer-manifest" {
!~      content             = (sensitive value)
!~      crc32c              = "9rnV6w==" -> (known after apply)
!~      detect_md5hash      = "ByFXB40BeDe5NkvaWS4/Qg==" -> "different hash"
!~      generation          = 1765493899808953 -> (known after apply)
        id                  = "calitp-staging-composer-data/warehouse/target/manifest.json"
!~      md5hash             = "ByFXB40BeDe5NkvaWS4/Qg==" -> (known after apply)
        name                = "data/warehouse/target/manifest.json"
#        (16 unchanged attributes hidden)
    }

Plan: 0 to add, 15 to change, 9 to destroy.

📝 Plan generated in Plan Terraform for Warehouse and DAG changes #1191

@github-actions
Copy link

github-actions bot commented Dec 9, 2025 •
edited
Loading

Warehouse report 📦

DAG

Legend (in order of precedence)

Resource type Indicator Resolution
Large table-materialized model Orange Make the model incremental
Large model without partitioning or clustering Orange Add partitioning and/or clustering
View with more than one child Yellow Materialize as a table or incremental
Incremental Light green
Table Green
View White

@lauriemerrell lauriemerrell force-pushed the 3972-refund-dups branch 2 times, most recently from 07fd18e to 661c04d Compare December 9, 2025 23:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@evansiroky evansiroky Awaiting requested review from evansiroky evansiroky will be requested when the pull request is marked ready for review evansiroky is a code owner

@vevetron vevetron Awaiting requested review from vevetron vevetron will be requested when the pull request is marked ready for review vevetron is a code owner

@tiffanychu90 tiffanychu90 Awaiting requested review from tiffanychu90 tiffanychu90 will be requested when the pull request is marked ready for review tiffanychu90 is a code owner

@ohrite ohrite Awaiting requested review from ohrite ohrite will be requested when the pull request is marked ready for review ohrite is a code owner

@erikamov erikamov Awaiting requested review from erikamov erikamov will be requested when the pull request is marked ready for review erikamov is a code owner

@charlie-costanzo charlie-costanzo Awaiting requested review from charlie-costanzo charlie-costanzo will be requested when the pull request is marked ready for review charlie-costanzo is a code owner

@mjumbewu mjumbewu Awaiting requested review from mjumbewu mjumbewu will be requested when the pull request is marked ready for review mjumbewu is a code owner

@fsalemi fsalemi Awaiting requested review from fsalemi fsalemi will be requested when the pull request is marked ready for review fsalemi is a code owner

@raebot raebot Awaiting requested review from raebot raebot will be requested when the pull request is marked ready for review raebot is a code owner

@jparr jparr Awaiting requested review from jparr jparr will be requested when the pull request is marked ready for review jparr is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Payments: Littlepay quick refunds investigation

2 participants

@lauriemerrell