Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Replace NTD agency information source and deprecate manual process #3736

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Replace NTD agency information source and deprecate manual process #3736

wants to merge 1 commit into from
+70 −2,417

Conversation

erikamov
Copy link
Contributor

Description

This PR deprecates the manual NTD agency information and used new automatized data sources 2022__annual_database_agency_information and 2023__annual_database_agency_information.

Resolves #3735

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation

How has this been tested?

Tested locally.
poetry run dbt run -s +models/mart/ntd/dim_annual_agency_information.sql
Screenshot 2025-02-25 at 11 41 10 AM

poetry run dbt test -s +models/mart/ntd/dim_annual_agency_information.sql
Screenshot 2025-02-25 at 11 41 20 AM

Post-merge follow-ups

  • No action required
  • Actions required (specified below)
  • Drop deprecated staging table cal-itp-data-infra.staging.stg_ntd__annual_database_agency_information
  • Drop deprecated external table cal-itp-data-infra.external_ntd_data_products.annual_database_agency_information
  • Delete calitp-ntd-data-products and test-calitp-ntd-data-products buckets

@erikamov erikamov self-assigned this Feb 25, 2025
@github-actions GitHub Actions
Copy link

github-actions bot commented Feb 25, 2025 •
edited
Loading

Warehouse report 📦

DAG

Legend (in order of precedence)

Resource type Indicator Resolution
Large table-materialized model Orange Make the model incremental
Large model without partitioning or clustering Orange Add partitioning and/or clustering
View with more than one child Yellow Materialize as a table or incremental
Incremental Light green
Table Green
View White

@erikamov erikamov force-pushed the 3735-deprecate-ntd-agency-manual-process branch from 88b549e to e28010b Compare February 28, 2025 00:11
vevetron
vevetron approved these changes Feb 28, 2025
View reviewed changes
Copy link
Contributor

@vevetron vevetron left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a little emotionally triggering.

@vevetron
Copy link
Contributor

I assume the raw data this outputs to it gone.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@vevetron vevetron vevetron approved these changes

@evansiroky evansiroky Awaiting requested review from evansiroky evansiroky is a code owner

@tiffanychu90 tiffanychu90 Awaiting requested review from tiffanychu90 tiffanychu90 is a code owner

@hunterowens hunterowens Awaiting requested review from hunterowens hunterowens is a code owner

Assignees

@erikamov erikamov

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Deprecate Manual NTD Agency Information Script and Tables
2 participants
@erikamov @vevetron