From 5f28c1b305eb6c4e005f1e4aeed528fbc3b385e9 Mon Sep 17 00:00:00 2001 From: Charles Costanzo Date: Mon, 8 Dec 2025 13:49:47 -0500 Subject: [PATCH 1/4] Documentation (payments): modify workflow documentation to reflect updated processes --- .../1-create-and-maintain-littlepay-data-syncs.md} | 0 ...rce.md => 2-add-agency-metabase-data-source-and-dashboards.md} | 0 2 files changed, 0 insertions(+), 0 deletions(-) rename runbooks/workflow/{creating-maintaining-littlepay-data-syncs.md => payments/1-create-and-maintain-littlepay-data-syncs.md} (100%) rename runbooks/workflow/payments/{add_agency_dashboard_data_source.md => 2-add-agency-metabase-data-source-and-dashboards.md} (100%) diff --git a/runbooks/workflow/creating-maintaining-littlepay-data-syncs.md b/runbooks/workflow/payments/1-create-and-maintain-littlepay-data-syncs.md similarity index 100% rename from runbooks/workflow/creating-maintaining-littlepay-data-syncs.md rename to runbooks/workflow/payments/1-create-and-maintain-littlepay-data-syncs.md diff --git a/runbooks/workflow/payments/add_agency_dashboard_data_source.md b/runbooks/workflow/payments/2-add-agency-metabase-data-source-and-dashboards.md similarity index 100% rename from runbooks/workflow/payments/add_agency_dashboard_data_source.md rename to runbooks/workflow/payments/2-add-agency-metabase-data-source-and-dashboards.md From 90d734b6b88e0d9d1bcb46465a5fa0bc8a7711d3 Mon Sep 17 00:00:00 2001 From: Charles Costanzo Date: Mon, 8 Dec 2025 16:17:21 -0500 Subject: [PATCH 2/4] add new procedure and TF PR examples --- ...ncy-metabase-data-source-and-dashboards.md | 70 ++++++++++--------- 1 file changed, 38 insertions(+), 32 deletions(-) diff --git a/runbooks/workflow/payments/2-add-agency-metabase-data-source-and-dashboards.md b/runbooks/workflow/payments/2-add-agency-metabase-data-source-and-dashboards.md index 33509f0ece..39e9af2718 100644 --- a/runbooks/workflow/payments/2-add-agency-metabase-data-source-and-dashboards.md +++ b/runbooks/workflow/payments/2-add-agency-metabase-data-source-and-dashboards.md @@ -8,48 +8,54 @@ This documentation is broken out into three sections: ## Add a New Agency Data Source to Metabase and Create Permissions -As new agencies are introduced to the contactless payments program, we will need to access their data within Metabase for use in their payments dashboard and other analysis. Because we use a [row access policy](https://cloud.google.com/bigquery/docs/row-level-security-intro#how_row-level_security_works) ([configured here](https://github.com/cal-itp/data-infra/blob/main/warehouse/macros/create_row_access_policy.sql)) in the warehouse code to limit access to data to authorized parties, this is a multi-step process. +As new agencies are introduced to the contactless payments program, we will need to access their data within Metabase for use in their payments dashboard and other analyses. Because we use a [row access policy](https://cloud.google.com/bigquery/docs/row-level-security-intro#how_row-level_security_works) ([configured here](https://github.com/cal-itp/data-infra/blob/main/warehouse/macros/create_row_access_policy.sql)) in the warehouse code to limit access to data to authorized parties, this is a multi-step process, and it begins with creating a new service account through our Terraform project. -### Create a new service account and row access policy +### Create a new service account -**Permissions needed**: Minimum Google Cloud Platform role `roles/iam.serviceAccountCreator` ([more information can be found in the Google IAM documentation](https://cloud.google.com/iam/docs/understanding-roles#iam.serviceAccountCreator)) +**Permissions needed**: -1. To begin, create a new service account for the agency within the Google Cloud Platform console, which will be used to allow Metabase read-access to the agency's payments data. +- `Write` permissions to the (Cal-ITP `data-infra` repository)\[https://github.com/cal-itp/data-infra/tree/main\]. -- Navigate to: in the `cal-itp-data-infra` project -- Select `+ Create Service Account` in the top-center of the page -- Populate the `Service account ID` field using the convention: `[agency-name]-payments-user`, then select `Create and Continue` -- Within the `Grant this service account access to the project` section, assign the role `Agency Payments Service Reader` -- Select `Done` +A new agency-specific service account needs to be created through the Terraform project in the `data-infra` repository. You can use (this previously-merged PR)\[https://github.com/cal-itp/data-infra/pull/4374/files\] as a reference for the two files that need to be modified through this process: `iac/cal-itp-data-infra/iam/us/project_iam_member.tf`, and `iac/cal-itp-data-infra/iam/us/service_account.tf`. -2. Download the service account key for the service account you've just created +- To begin, create a new branch in the [cal-itp/data-infra Github repository](https://github.com/cal-itp/data-infra). +- Navigate to the first file to be modified: `iac/cal-itp-data-infra/iam/us/project_iam_member.tf` + - Create a new entry in the file that mirrors the contents of the El Dorado service account configuration, (found here)\[https://github.com/cal-itp/data-infra/pull/4374/files#diff-225faaedb56a2d0c4b52f2a95e354a35444decb4cc6cff1717ce53b9d6e094fbR67-R71\]. + - Keep the majority of the contents of the El Dorado entry, subsituting only the `eldorado` text for a more appropriate naming convention based on the agency you are creating the service account for (this will be done in two places, lines 67 and 68 in the PR example above) +- Navigate to the second file to be modified: `iac/cal-itp-data-infra/iam/us/service_account.tf` + - Create a new entry in the file that mirrors the contents of the El Dorado service account configuration, (found here)\[https://github.com/cal-itp/data-infra/pull/4374/files#diff-275fd84e89c5153d0616094b0a753a02143de7932dc96ca206a960a9bc6ef183R281-R285\]. + - Keep the majority of the contents of the El Dorado entry, subsituting only the `eldorao` portion of the `eldorado-payments-user` text for a more appropriate naming convention based on the agency you are creating the service account for (this will be done in two places, lines 218 and 282 in the PR example above) +- Once this has been completed, mark your PR as 'Ready for Review', and once approved by a reviewer, merge in your changes. +- Upon merge, and once the Github actions finish running successfully, your new service account will be available in the Google Cloud Platform. Navigate to `IAM &Admin` --> `Service Accounts` to verify. -- After selecting `Done` in the previous section you'll be returned to the list of existing service accounts. Click into the service account that you just created. -- Select `Keys` from the top-center of the page and then select the `Add Key` dropdown. Choose the `Create new key` selection within that. -- Keep the default key type `JSON` and select `Create` -- This will download a JSON copy of the service accout key to your local environment, which will be used in later steps within Metabase - -3. Open a new branch in the [cal-itp/data-infra Github repository](https://github.com/cal-itp/data-infra) and edit the [create_row_access_policy macro](https://github.com/cal-itp/data-infra/blob/main/warehouse/macros/create_row_access_policy.sql) - **Permissions needed**: Member of the [cal-itp Github organization](https://github.com/cal-itp) +### Create a new row access policy using the above service account -Duplicate an existing row access policy within the file and append to the bottom, before the `{% endmacro %}` text. +**Permissions needed**: -The contents of the policy you're duplicating should look like this: +- Minimum Google Cloud Platform role `roles/iam.serviceAccountKeyAdmin` (more information can be found in the Google IAM documentation). +- `Write` permissions to the (Cal-ITP `data-infra` repository)\[https://github.com/cal-itp/data-infra/tree/main\]. -``` -{{ create_row_access_policy( - filter_column = 'participant_id', - filter_value = '[agency-name]', - principals = ['serviceAccount:`agency-name`-payments-user@cal-itp-data-infra.iam.gserviceaccount.com'] -) }}; -``` +1. Download the service account key for the service account you've just created -Substitute the following fields with the appropriate information for the agency that you are adding: - -- `filter_value` which is the Littlepay `participant_id` for the agency -- `principals` which is the email address for the service account that was created in step #1. You can simply subsitute the agency name as used in that step as opposed to updating the whole string. - -Open a PR in Github to merge these changes. If you'd like access to the results of this policy before the next time the `transform_warehouse` DAG is run, you will need to run it manually. To do this, you should trigger the DAG with a selector [as described in the README for the DAG task](https://github.com/cal-itp/data-infra/tree/main/airflow/dags/transform_warehouse). Use selector: `{"dbt_select": "models/mart/payments"}`. +- After verifying that your new service account has been created in the list of existing service accounts, click into the service account that you just created. +- Select `Keys` from the top-center of the page and then select the `Add Key` dropdown. Choose the `Create new key` selection within that. +- Keep the default key type `JSON` and select `Create` +- This will download a JSON copy of the service accout key to your local environment, which will be used in later steps within Metabase. Store this in a secure, local location. + +2. Open a new branch in the [cal-itp/data-infra Github repository](https://github.com/cal-itp/data-infra). You can use (this previously-merged PR)\[https://github.com/cal-itp/data-infra/pull/4376/files\] as a reference for the two dbt macros that need to be modified through this process in this file: `warehouse/macros/create_row_access_policy.sql`. + +- Modify the first macro in the file: `payments_littlepay_row_access_policy` + - Duplicate an existing entry. In the El Dorado example linked in the PR above, (this would be rows 89-93)\[https://github.com/cal-itp/data-infra/pull/4376/files#diff-e32013136795892ab542f0571294fd65e723bc4085e41b5a52ac75d29e3503e4R89-R93\]. + - Modify two fields: + - Filter_value: substitute `'eldorado-transit'` with the **Littlepay participant_id** for the new agency. + - Principals: Subsitute the address for the newly created service account after `['serviceAccount:` +- Modify the second macro in the file: `payments_elavon_row_access_policy` + - Duplicate an existing entry. In the El Dorado example linked in the PR above, (this would be rows 184-188)\[https://github.com/cal-itp/data-infra/pull/4376/files#diff-e32013136795892ab542f0571294fd65e723bc4085e41b5a52ac75d29e3503e4R184-R188\]. + - Modify two fields: + - Filter_value: substitute `'El Dorado County Transit Authority'` with the **Elavon organization_name** for the new agency. + - Principals: Subsitute the address for the newly created service account after `['serviceAccount:` +- Once this has been completed, mark your PR as 'Ready for Review', and once approved by a reviewer, merge in your changes. +- Upon merge, and once the warehouse tables utilizing the row access polcies have been re-run, your new access policy will be in effect. ### Add a new `Database` in Metabase for the agency From 5dee999660c86c25fc306309689a9eac32ba0500 Mon Sep 17 00:00:00 2001 From: Charles Costanzo Date: Mon, 8 Dec 2025 16:20:37 -0500 Subject: [PATCH 3/4] fix silly url formatting mistake --- ...d-agency-metabase-data-source-and-dashboards.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/runbooks/workflow/payments/2-add-agency-metabase-data-source-and-dashboards.md b/runbooks/workflow/payments/2-add-agency-metabase-data-source-and-dashboards.md index 39e9af2718..d02647cd98 100644 --- a/runbooks/workflow/payments/2-add-agency-metabase-data-source-and-dashboards.md +++ b/runbooks/workflow/payments/2-add-agency-metabase-data-source-and-dashboards.md @@ -16,14 +16,14 @@ As new agencies are introduced to the contactless payments program, we will need - `Write` permissions to the (Cal-ITP `data-infra` repository)\[https://github.com/cal-itp/data-infra/tree/main\]. -A new agency-specific service account needs to be created through the Terraform project in the `data-infra` repository. You can use (this previously-merged PR)\[https://github.com/cal-itp/data-infra/pull/4374/files\] as a reference for the two files that need to be modified through this process: `iac/cal-itp-data-infra/iam/us/project_iam_member.tf`, and `iac/cal-itp-data-infra/iam/us/service_account.tf`. +A new agency-specific service account needs to be created through the Terraform project in the `data-infra` repository. You can use [this previously-merged PR](https://github.com/cal-itp/data-infra/pull/4374/files) as a reference for the two files that need to be modified through this process: `iac/cal-itp-data-infra/iam/us/project_iam_member.tf`, and `iac/cal-itp-data-infra/iam/us/service_account.tf`. - To begin, create a new branch in the [cal-itp/data-infra Github repository](https://github.com/cal-itp/data-infra). - Navigate to the first file to be modified: `iac/cal-itp-data-infra/iam/us/project_iam_member.tf` - - Create a new entry in the file that mirrors the contents of the El Dorado service account configuration, (found here)\[https://github.com/cal-itp/data-infra/pull/4374/files#diff-225faaedb56a2d0c4b52f2a95e354a35444decb4cc6cff1717ce53b9d6e094fbR67-R71\]. + - Create a new entry in the file that mirrors the contents of the El Dorado service account configuration, [found here](https://github.com/cal-itp/data-infra/pull/4374/files#diff-225faaedb56a2d0c4b52f2a95e354a35444decb4cc6cff1717ce53b9d6e094fbR67-R71). - Keep the majority of the contents of the El Dorado entry, subsituting only the `eldorado` text for a more appropriate naming convention based on the agency you are creating the service account for (this will be done in two places, lines 67 and 68 in the PR example above) - Navigate to the second file to be modified: `iac/cal-itp-data-infra/iam/us/service_account.tf` - - Create a new entry in the file that mirrors the contents of the El Dorado service account configuration, (found here)\[https://github.com/cal-itp/data-infra/pull/4374/files#diff-275fd84e89c5153d0616094b0a753a02143de7932dc96ca206a960a9bc6ef183R281-R285\]. + - Create a new entry in the file that mirrors the contents of the El Dorado service account configuration, [found here](https://github.com/cal-itp/data-infra/pull/4374/files#diff-275fd84e89c5153d0616094b0a753a02143de7932dc96ca206a960a9bc6ef183R281-R285). - Keep the majority of the contents of the El Dorado entry, subsituting only the `eldorao` portion of the `eldorado-payments-user` text for a more appropriate naming convention based on the agency you are creating the service account for (this will be done in two places, lines 218 and 282 in the PR example above) - Once this has been completed, mark your PR as 'Ready for Review', and once approved by a reviewer, merge in your changes. - Upon merge, and once the Github actions finish running successfully, your new service account will be available in the Google Cloud Platform. Navigate to `IAM &Admin` --> `Service Accounts` to verify. @@ -33,7 +33,7 @@ A new agency-specific service account needs to be created through the Terraform **Permissions needed**: - Minimum Google Cloud Platform role `roles/iam.serviceAccountKeyAdmin` (more information can be found in the Google IAM documentation). -- `Write` permissions to the (Cal-ITP `data-infra` repository)\[https://github.com/cal-itp/data-infra/tree/main\]. +- `Write` permissions to the [Cal-ITP `data-infra` repository](https://github.com/cal-itp/data-infra/tree/main). 1. Download the service account key for the service account you've just created @@ -42,15 +42,15 @@ A new agency-specific service account needs to be created through the Terraform - Keep the default key type `JSON` and select `Create` - This will download a JSON copy of the service accout key to your local environment, which will be used in later steps within Metabase. Store this in a secure, local location. -2. Open a new branch in the [cal-itp/data-infra Github repository](https://github.com/cal-itp/data-infra). You can use (this previously-merged PR)\[https://github.com/cal-itp/data-infra/pull/4376/files\] as a reference for the two dbt macros that need to be modified through this process in this file: `warehouse/macros/create_row_access_policy.sql`. +2. Open a new branch in the [cal-itp/data-infra Github repository](https://github.com/cal-itp/data-infra). You can use [this previously-merged PR](https://github.com/cal-itp/data-infra/pull/4376/files) as a reference for the two dbt macros that need to be modified through this process in this file: `warehouse/macros/create_row_access_policy.sql`. - Modify the first macro in the file: `payments_littlepay_row_access_policy` - - Duplicate an existing entry. In the El Dorado example linked in the PR above, (this would be rows 89-93)\[https://github.com/cal-itp/data-infra/pull/4376/files#diff-e32013136795892ab542f0571294fd65e723bc4085e41b5a52ac75d29e3503e4R89-R93\]. + - Duplicate an existing entry. In the El Dorado example linked in the PR above, [this would be rows 89-93](https://github.com/cal-itp/data-infra/pull/4376/files#diff-e32013136795892ab542f0571294fd65e723bc4085e41b5a52ac75d29e3503e4R89-R93). - Modify two fields: - Filter_value: substitute `'eldorado-transit'` with the **Littlepay participant_id** for the new agency. - Principals: Subsitute the address for the newly created service account after `['serviceAccount:` - Modify the second macro in the file: `payments_elavon_row_access_policy` - - Duplicate an existing entry. In the El Dorado example linked in the PR above, (this would be rows 184-188)\[https://github.com/cal-itp/data-infra/pull/4376/files#diff-e32013136795892ab542f0571294fd65e723bc4085e41b5a52ac75d29e3503e4R184-R188\]. + - Duplicate an existing entry. In the El Dorado example linked in the PR above, [this would be rows 184-188](https://github.com/cal-itp/data-infra/pull/4376/files#diff-e32013136795892ab542f0571294fd65e723bc4085e41b5a52ac75d29e3503e4R184-R188). - Modify two fields: - Filter_value: substitute `'El Dorado County Transit Authority'` with the **Elavon organization_name** for the new agency. - Principals: Subsitute the address for the newly created service account after `['serviceAccount:` From ef1d33cf2e024c006181719e288a0aa6f4b9c001 Mon Sep 17 00:00:00 2001 From: Charles Costanzo Date: Mon, 8 Dec 2025 16:22:09 -0500 Subject: [PATCH 4/4] fix another silly url formatting mistake --- .../2-add-agency-metabase-data-source-and-dashboards.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/runbooks/workflow/payments/2-add-agency-metabase-data-source-and-dashboards.md b/runbooks/workflow/payments/2-add-agency-metabase-data-source-and-dashboards.md index d02647cd98..25b883e575 100644 --- a/runbooks/workflow/payments/2-add-agency-metabase-data-source-and-dashboards.md +++ b/runbooks/workflow/payments/2-add-agency-metabase-data-source-and-dashboards.md @@ -14,7 +14,7 @@ As new agencies are introduced to the contactless payments program, we will need **Permissions needed**: -- `Write` permissions to the (Cal-ITP `data-infra` repository)\[https://github.com/cal-itp/data-infra/tree/main\]. +- `Write` permissions to the [Cal-ITP `data-infra` repository](https://github.com/cal-itp/data-infra/tree/main). A new agency-specific service account needs to be created through the Terraform project in the `data-infra` repository. You can use [this previously-merged PR](https://github.com/cal-itp/data-infra/pull/4374/files) as a reference for the two files that need to be modified through this process: `iac/cal-itp-data-infra/iam/us/project_iam_member.tf`, and `iac/cal-itp-data-infra/iam/us/service_account.tf`.