From 04ecb0e8955bb7130683a9f2c07db343af6603dc Mon Sep 17 00:00:00 2001 From: Katherine Cox Date: Mon, 24 Jan 2022 14:58:25 -0500 Subject: [PATCH 01/15] copying Google Doc draft --- 02-pis.Rmd | 20 +++++++++++++++++++- 1 file changed, 19 insertions(+), 1 deletion(-) diff --git a/02-pis.Rmd b/02-pis.Rmd index 279e5b71..f409107e 100644 --- a/02-pis.Rmd +++ b/02-pis.Rmd @@ -264,7 +264,25 @@ leanbuild::include_slide("https://docs.google.com/presentation/d/162GS7ArBPM4w_r ### Export Cost Data to BigQuery -Coming soon -- instructions on how to export your cost data so you can better analyze and control your expenses. +Several useful tools are available to help you further analyze and understand your GCP computing costs. Examples include + +The `AnVILBilling` [Bioconductor package](http://www.bioconductor.org/packages/release/bioc/html/AnVILBilling.html) +[Google Data Studio](https://cloud.google.com/billing/docs/how-to/visualize-data) + +[These tools are helpful because...] + +In order to make use of these tools, you will need to set up exporting of your cost data to BigQuery. This cannot be done retroactively (i.e. you can only export cost data as it is produced, there is no way to export your cost data history). Therefore, **we recommend setting up cost data export immediately, so the cost data will be available to you in the future**. + +**NOTE**: This will incur a minimal monthly fee, less than $1 / month (pricing as of Feb 2021) + +#### Set Up Data Export + + + + +To learn more about how to analyze your BigQuery cost data, see: +[] + ## Step 6: Add Users and Workspaces From 65726a2b43558ac8fcbde90cea1fc040554f8dda Mon Sep 17 00:00:00 2001 From: Katherine Cox Date: Tue, 25 Jan 2022 13:32:32 -0500 Subject: [PATCH 02/15] editing BigQuery section --- 02-pis.Rmd | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/02-pis.Rmd b/02-pis.Rmd index f409107e..b500684b 100644 --- a/02-pis.Rmd +++ b/02-pis.Rmd @@ -264,24 +264,24 @@ leanbuild::include_slide("https://docs.google.com/presentation/d/162GS7ArBPM4w_r ### Export Cost Data to BigQuery -Several useful tools are available to help you further analyze and understand your GCP computing costs. Examples include +Several useful tools are available to help you further analyze and understand your GCP computing costs. Examples include the `AnVILBilling` [Bioconductor package](http://www.bioconductor.org/packages/release/bioc/html/AnVILBilling.html) and [Google Data Studio](https://cloud.google.com/billing/docs/how-to/visualize-data). -The `AnVILBilling` [Bioconductor package](http://www.bioconductor.org/packages/release/bioc/html/AnVILBilling.html) -[Google Data Studio](https://cloud.google.com/billing/docs/how-to/visualize-data) +These tools can help give greater insight into the charges to your GCP Billing Account which can help you refine your workflows to minimize expenses. -[These tools are helpful because...] - -In order to make use of these tools, you will need to set up exporting of your cost data to BigQuery. This cannot be done retroactively (i.e. you can only export cost data as it is produced, there is no way to export your cost data history). Therefore, **we recommend setting up cost data export immediately, so the cost data will be available to you in the future**. +In order to make use of these tools, you will need to set up exporting of your cost data to BigQuery. This cannot be done retroactively (i.e. you can only export cost data as it is produced; there is no way to export your cost data history). Therefore, **we recommend setting up cost data export immediately, so the cost data will be available to you in the future**. **NOTE**: This will incur a minimal monthly fee, less than $1 / month (pricing as of Feb 2021) #### Set Up Data Export +```{r, child=c("_child_export_cost_data_to_BigQuery.Rmd")} +``` +Once you have set up data exporting, you will be able to analyze your cost data at your convenience. To learn more about how to analyze your cost data, see: -To learn more about how to analyze your BigQuery cost data, see: -[] +- The [vignette](http://www.bioconductor.org/packages/release/bioc/vignettes/AnVILBilling/inst/doc/billing.html) for the [`AnVILBilling`](http://www.bioconductor.org/packages/release/bioc/html/AnVILBilling.html) +- [Google's documentation](https://cloud.google.com/billing/docs/how-to/visualize-data) on visualizing spending with Google Data Studio ## Step 6: Add Users and Workspaces From 7a4e48947803e88c2adc35f77d26c5a0e8f85b0f Mon Sep 17 00:00:00 2001 From: Katherine Cox Date: Tue, 8 Feb 2022 16:03:50 -0500 Subject: [PATCH 03/15] starting on BigQuery instructions --- 02-pis.Rmd | 5 +++ _child_export_cost_data_to_BigQuery.Rmd | 46 +++++++++++++++++++++++++ 2 files changed, 51 insertions(+) create mode 100644 _child_export_cost_data_to_BigQuery.Rmd diff --git a/02-pis.Rmd b/02-pis.Rmd index b500684b..435f8746 100644 --- a/02-pis.Rmd +++ b/02-pis.Rmd @@ -283,6 +283,11 @@ Once you have set up data exporting, you will be able to analyze your cost data - The [vignette](http://www.bioconductor.org/packages/release/bioc/vignettes/AnVILBilling/inst/doc/billing.html) for the [`AnVILBilling`](http://www.bioconductor.org/packages/release/bioc/html/AnVILBilling.html) - [Google's documentation](https://cloud.google.com/billing/docs/how-to/visualize-data) on visualizing spending with Google Data Studio +**Having trouble?** + +- Visit our community support forum at [`help.anvilproject.org`](https://help.anvilproject.org) with any questions. +- View the [`Google documentation`](https://cloud.google.com/billing/docs/how-to/export-data-bigquery) + ## Step 6: Add Users and Workspaces diff --git a/_child_export_cost_data_to_BigQuery.Rmd b/_child_export_cost_data_to_BigQuery.Rmd new file mode 100644 index 00000000..969a746e --- /dev/null +++ b/_child_export_cost_data_to_BigQuery.Rmd @@ -0,0 +1,46 @@ + +Exporting billing data to BigQuery requires a few steps to set up: + +1. Create BigQuery Billing Project +1. Create BigQuery Dataset +1. Enable data export + +**Create BigQuery Billing Project** + +First you will need to create a Google Billing Project that will run the export process. + +::: {.fyi} +Reminder: + +- A **Google Billing Account** manages your *funding source* (e.g. credit card, cloud credits). You probably only need one Billing Account unless you have multiple funding sources. +- A **Google Billing Project** manages activity you do on GCP. Once Billing Account can fund multiple Billing Projects. +::: + +Usually you will create your Billing Projects through Terra and won't need to manage them through the Google Cloud interface. In this case, you will create a Billing Project directly within Google. **This must be under the same Google Billing Account as the Projects you want to track spending for.** + + +**Create BigQuery Dataset** + to store the billing export +**Enable data export** + Go to the billing tab in the gcloud console and enable export to this dataset +Make sure the user of this software has the BigQuery scope on the billing project + + +Note that + +- You only need one BigQuery project per Google Billing Account. One BigQuery Project can handle cost data for multiple lab members, cohorts, etc., as long as they are all under the same Google Billing Account (i.e. have the same funding source) +- If you are using multiple Google Billing Accounts (i.e. have multiple funding sources) you will need to set up a BigQuery project for each of them. + +1. [Launch Terra](https://anvil.terra.bio/#workspaces) + +1. Locate the Workspace you want to clone. If a Workspace has been shared with you ahead of time, it will appear in "MY WORKSPACES". You can clone a Workspace that was shared with you to perform your own analyses. In the screenshot below, no Workspaces have been shared. + + ```{r, echo=FALSE, fig.alt='Screenshot of Terra "MY WORKSPACES" menu. The "MY WORKSPACES" tab is highlighted. No Workspaces are visible because none have been shared with the user. There is an option to create a new Workspace.'} +leanbuild::include_slide("https://docs.google.com/presentation/d/1ioYY0n3CcrP934WRKEsQ1aKocPqfswdWWHxYCf76BCg/edit#slide=id.gf5172664d7_0_142") + ``` + +1. If a Workspace hasn't been shared with you, navigate to the "FEATURED" or "PUBLIC" Workspace tabs. + + ```{r, echo=FALSE, fig.alt='Screenshot of Terra "My Workspaces" menu. The "FEATURED" tab is highlighted.'} +leanbuild::include_slide("https://docs.google.com/presentation/d/1ioYY0n3CcrP934WRKEsQ1aKocPqfswdWWHxYCf76BCg/edit#slide=id.gf5172664d7_0_335") + ``` From b55ef704f6d4d45cc05d9d3626b78d03c5fc3046 Mon Sep 17 00:00:00 2001 From: Katherine Cox Date: Thu, 14 Apr 2022 11:17:58 -0400 Subject: [PATCH 04/15] rename BigQuery child --- 02-pis.Rmd | 2 +- ...BigQuery.Rmd => _child_google_billing_export_to_BigQuery.Rmd | 0 2 files changed, 1 insertion(+), 1 deletion(-) rename _child_export_cost_data_to_BigQuery.Rmd => _child_google_billing_export_to_BigQuery.Rmd (100%) diff --git a/02-pis.Rmd b/02-pis.Rmd index 6cf24ceb..14af9040 100644 --- a/02-pis.Rmd +++ b/02-pis.Rmd @@ -238,7 +238,7 @@ In order to make use of these tools, you will need to set up exporting of your c #### Set Up Data Export -```{r, child=c("_child_export_cost_data_to_BigQuery.Rmd")} +```{r, child=c("_child_google_biling_export_to_BigQuery.Rmd")} ``` diff --git a/_child_export_cost_data_to_BigQuery.Rmd b/_child_google_billing_export_to_BigQuery.Rmd similarity index 100% rename from _child_export_cost_data_to_BigQuery.Rmd rename to _child_google_billing_export_to_BigQuery.Rmd From a3e6ca02a293e07f83e1a15d99395483d919ea0c Mon Sep 17 00:00:00 2001 From: Katherine Cox Date: Mon, 25 Apr 2022 11:21:25 -0400 Subject: [PATCH 05/15] work on BigQuery section --- _child_google_billing_export_to_BigQuery.Rmd | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/_child_google_billing_export_to_BigQuery.Rmd b/_child_google_billing_export_to_BigQuery.Rmd index 969a746e..e7596eb8 100644 --- a/_child_google_billing_export_to_BigQuery.Rmd +++ b/_child_google_billing_export_to_BigQuery.Rmd @@ -1,4 +1,6 @@ +*Updated April 2022. See the [Google Cloud Documentation](https://cloud.google.com/billing/docs/how-to/export-data-bigquery-setup) for the latest documentation and additional details.* + Exporting billing data to BigQuery requires a few steps to set up: 1. Create BigQuery Billing Project @@ -13,12 +15,17 @@ First you will need to create a Google Billing Project that will run the export Reminder: - A **Google Billing Account** manages your *funding source* (e.g. credit card, cloud credits). You probably only need one Billing Account unless you have multiple funding sources. -- A **Google Billing Project** manages activity you do on GCP. Once Billing Account can fund multiple Billing Projects. +- A **Google Billing Project** manages activity you do on GCP. Multiple Billing Projects can be funded by a single Billing Account. ::: Usually you will create your Billing Projects through Terra and won't need to manage them through the Google Cloud interface. In this case, you will create a Billing Project directly within Google. **This must be under the same Google Billing Account as the Projects you want to track spending for.** +To create a new Google Billing Project: + +1. + + **Create BigQuery Dataset** to store the billing export **Enable data export** @@ -26,9 +33,9 @@ Usually you will create your Billing Projects through Terra and won't need to ma Make sure the user of this software has the BigQuery scope on the billing project -Note that +Note that: -- You only need one BigQuery project per Google Billing Account. One BigQuery Project can handle cost data for multiple lab members, cohorts, etc., as long as they are all under the same Google Billing Account (i.e. have the same funding source) +- You only need one BigQuery project per Google Billing Account. One BigQuery Project can handle cost data for multiple Billing Projects (i.e. lab members, cohorts, etc.), as long as they are all under the same Google Billing Account. - If you are using multiple Google Billing Accounts (i.e. have multiple funding sources) you will need to set up a BigQuery project for each of them. 1. [Launch Terra](https://anvil.terra.bio/#workspaces) From c14d18f41d7fccc94797959c29a3dbd9ad9a6c72 Mon Sep 17 00:00:00 2001 From: Katherine Cox Date: Mon, 18 Jul 2022 14:20:08 -0400 Subject: [PATCH 06/15] add instructions to create GCP project --- 02-pis.Rmd | 2 +- _child_google_billing_export_to_BigQuery.Rmd | 95 ++++++++++++++++++-- 2 files changed, 88 insertions(+), 9 deletions(-) diff --git a/02-pis.Rmd b/02-pis.Rmd index 14af9040..e3bd8d23 100644 --- a/02-pis.Rmd +++ b/02-pis.Rmd @@ -238,7 +238,7 @@ In order to make use of these tools, you will need to set up exporting of your c #### Set Up Data Export -```{r, child=c("_child_google_biling_export_to_BigQuery.Rmd")} +```{r, child=c("_child_google_billing_export_to_BigQuery.Rmd")} ``` diff --git a/_child_google_billing_export_to_BigQuery.Rmd b/_child_google_billing_export_to_BigQuery.Rmd index e7596eb8..27e8a558 100644 --- a/_child_google_billing_export_to_BigQuery.Rmd +++ b/_child_google_billing_export_to_BigQuery.Rmd @@ -1,30 +1,109 @@ -*Updated April 2022. See the [Google Cloud Documentation](https://cloud.google.com/billing/docs/how-to/export-data-bigquery-setup) for the latest documentation and additional details.* +*Updated July 2022. See the [Google Cloud Documentation](https://cloud.google.com/billing/docs/how-to/export-data-bigquery-setup) for the latest documentation and additional details.* Exporting billing data to BigQuery requires a few steps to set up: -1. Create BigQuery Billing Project +1. Create GCP Project 1. Create BigQuery Dataset 1. Enable data export -**Create BigQuery Billing Project** +**Create GCP Project** -First you will need to create a Google Billing Project that will run the export process. +First you will need to create a GCP Project that will run the export process. ::: {.fyi} Reminder: -- A **Google Billing Account** manages your *funding source* (e.g. credit card, cloud credits). You probably only need one Billing Account unless you have multiple funding sources. -- A **Google Billing Project** manages activity you do on GCP. Multiple Billing Projects can be funded by a single Billing Account. +- A Google **Billing Account** manages your *funding source* (e.g. credit card, cloud credits). You probably only need one Billing Account unless you have multiple funding sources. +- A Google **Project** manages your GCP activities and cloud resources. Multiple GCP Projects can be funded by a single Billing Account. ::: -Usually you will create your Billing Projects through Terra and won't need to manage them through the Google Cloud interface. In this case, you will create a Billing Project directly within Google. **This must be under the same Google Billing Account as the Projects you want to track spending for.** +Usually your GCP Projects are created and managed by Terra (corresponding to Terra Workspaces) and you don't need to manage them through the Google Cloud interface. In this case, you will need to create the GCP Project yourself. **This must be under the same Google Billing Account as the Projects you want to track spending for.** -To create a new Google Billing Project: +To create a new GCP Project: + +1. Log in to the [Google Cloud Platform](https://console.cloud.google.com/) console using the Google ID associated with your Google Cloud projects. + +1. Open the dropdown menu on the top left, mouseover IAM & Admin, and click “Create a Project.” + + ```{r, echo=FALSE, fig.alt='Screenshot of the Google Cloud Console drop-down menu, with the "IAM & Admin" submenu open. "IAM & Admin" and "Create a Project" are highlighted.'} +ottrpal::include_slide("https://docs.google.com/presentation/d/1JejmHJHLTF-eywlZs5GUQXb82FCGM5wIUNrbVKlJcrY/edit#slide=id.g13d748460ee_0_3") + ``` + +1. Enter a name for your new project. Project names can contain only letters, numbers, single quotes, hyphens, spaces, or exclamation points, and must be between 4 and 30 characters. + + ```{r, echo=FALSE, fig.alt='Screenshot of the GCP Project creation dialog. The text box labeled "Project name" is highlighted.'} '} +ottrpal::include_slide("https://docs.google.com/presentation/d/1JejmHJHLTF-eywlZs5GUQXb82FCGM5wIUNrbVKlJcrY/edit#slide=id.g13d748460ee_0_153") + ``` + +1. Choose an organization. Unless you are part of a larger GCP organization, you can choose “No organization” for the location. + + ```{r, echo=FALSE, fig.alt='Screenshot of the GCP Project creation dialog. The text box labeled "Location" is highlighted.'} '} +ottrpal::include_slide("https://docs.google.com/presentation/d/1JejmHJHLTF-eywlZs5GUQXb82FCGM5wIUNrbVKlJcrY/edit#slide=id.g13d748460ee_0_172") + ``` + +1. Click "Create". + + ```{r, echo=FALSE, fig.alt='Screenshot of the GCP Project creation dialog. The "Create" button is highlighted.'} '} +ottrpal::include_slide("https://docs.google.com/presentation/d/1JejmHJHLTF-eywlZs5GUQXb82FCGM5wIUNrbVKlJcrY/edit#slide=id.g13d748460ee_0_166") + ``` + + +**Create BigQuery Dataset** + +Once your GCP Project exists, it can be used to create a BigQuery Dataset to hold your billing data. + +To create a BigQuery Dataset: 1. +``` +# copied from GCP docs: + +Go to the Manage resources page in the Google Cloud console. +Go to the Manage Resources page +The remaining steps will appear automatically in the Google Cloud console. + +On the Select organization drop-down list at the top of the page, select the organization in which you want to create a project. If you are a free trial user, skip this step, as this list does not appear. +Click Create Project. +In the New Project window that appears, enter a project name and select a billing account as applicable. A project name can contain only letters, numbers, single quotes, hyphens, spaces, or exclamation points, and must be between 4 and 30 characters. +Enter the parent organization or folder in the Location box. That resource will be the hierarchical parent of the new project. +When you're finished entering new project details, click Create. + + +To create a BigQuery dataset: + +Sign in to the Google Cloud console and go to the BigQuery page. + +Go to BigQuery page + +In the project drop-down list ( My Project arrow_drop_down) at the top of the Google Cloud console page, select the project you set up to contain your dataset. Make note of the project ID, as you will use that in the next steps. + +In the Explorer panel, in the pinned projects section, click arrow_right your-project-ID to expand the project to view any existing datasets and saved queries. + +Next to the project ID, click the View actions menu (more_vert) and then click Create dataset. The Create dataset panel opens. + +Enter a Dataset ID. We recommend an ID that spans projects, such as all_billing_data, rather than a project-specific ID. +Select a Data location. The data location specifies the region where your data is stored. All tables within this dataset share this location. + +When creating a dataset, we recommend that you select a multi-region option for its location, such as United States (US) or European Union (EU). This is required for exporting the detailed usage cost data. + +After you create the dataset, the location cannot be changed. Learn more about locations. + +Ensure that the Enable table expiration option is cleared. + +If you enable table expiration and enter a number of days, any new table created in this dataset is automatically deleted following the specified number of days after creation. Important: If you delete any tables containing exported Cloud Billing data records, those records are gone and we cannot backfill the deleted records. + +In the Advanced options section, ensure that the Use a customer-managed encryption key (CMEK) option is cleared. + +Customer-managed key encryption is not supported for exporting Cloud Billing data records to BigQuery. + +To save, click Create dataset. + + +``` + **Create BigQuery Dataset** to store the billing export From 0778255c97180d6a448d8a54b8b3a69a5389ef4e Mon Sep 17 00:00:00 2001 From: Katherine Cox Date: Mon, 18 Jul 2022 14:23:13 -0400 Subject: [PATCH 07/15] edits to create GCP project --- _child_google_billing_export_to_BigQuery.Rmd | 13 ++----------- 1 file changed, 2 insertions(+), 11 deletions(-) diff --git a/_child_google_billing_export_to_BigQuery.Rmd b/_child_google_billing_export_to_BigQuery.Rmd index 27e8a558..00ba3cef 100644 --- a/_child_google_billing_export_to_BigQuery.Rmd +++ b/_child_google_billing_export_to_BigQuery.Rmd @@ -37,7 +37,7 @@ ottrpal::include_slide("https://docs.google.com/presentation/d/1JejmHJHLTF-eywlZ ottrpal::include_slide("https://docs.google.com/presentation/d/1JejmHJHLTF-eywlZs5GUQXb82FCGM5wIUNrbVKlJcrY/edit#slide=id.g13d748460ee_0_153") ``` -1. Choose an organization. Unless you are part of a larger GCP organization, you can choose “No organization” for the location. +1. You may see an option to choose a location. Unless you are part of a larger GCP organization, you can choose “No organization” for the location. ```{r, echo=FALSE, fig.alt='Screenshot of the GCP Project creation dialog. The text box labeled "Location" is highlighted.'} '} ottrpal::include_slide("https://docs.google.com/presentation/d/1JejmHJHLTF-eywlZs5GUQXb82FCGM5wIUNrbVKlJcrY/edit#slide=id.g13d748460ee_0_172") @@ -61,16 +61,7 @@ To create a BigQuery Dataset: ``` # copied from GCP docs: -Go to the Manage resources page in the Google Cloud console. -Go to the Manage Resources page -The remaining steps will appear automatically in the Google Cloud console. - -On the Select organization drop-down list at the top of the page, select the organization in which you want to create a project. If you are a free trial user, skip this step, as this list does not appear. -Click Create Project. -In the New Project window that appears, enter a project name and select a billing account as applicable. A project name can contain only letters, numbers, single quotes, hyphens, spaces, or exclamation points, and must be between 4 and 30 characters. -Enter the parent organization or folder in the Location box. That resource will be the hierarchical parent of the new project. -When you're finished entering new project details, click Create. - +In the New Project window that appears, enter a project name and select a billing account as applicable. To create a BigQuery dataset: From 1390dd9c6f527f45cb21717eaafe42e5ecbecf6e Mon Sep 17 00:00:00 2001 From: Katherine Cox Date: Tue, 19 Jul 2022 13:25:40 -0400 Subject: [PATCH 08/15] work on instructions to create BigQuery dataset --- _child_google_billing_export_to_BigQuery.Rmd | 53 ++++---------------- 1 file changed, 11 insertions(+), 42 deletions(-) diff --git a/_child_google_billing_export_to_BigQuery.Rmd b/_child_google_billing_export_to_BigQuery.Rmd index 00ba3cef..02cd4c6f 100644 --- a/_child_google_billing_export_to_BigQuery.Rmd +++ b/_child_google_billing_export_to_BigQuery.Rmd @@ -56,48 +56,30 @@ Once your GCP Project exists, it can be used to create a BigQuery Dataset to hol To create a BigQuery Dataset: -1. +1. Open the dropdown menu on the top left, and click “BigQuery.” You may need to scroll down quite a ways before you see it. -``` -# copied from GCP docs: - -In the New Project window that appears, enter a project name and select a billing account as applicable. - -To create a BigQuery dataset: - -Sign in to the Google Cloud console and go to the BigQuery page. + - If you’d like, you can pin BigQuery so it’s easier to find next time. -Go to BigQuery page +1. Make sure the correct GCP project is selected. You can see the current project at the top of the screen. If this is not correct, click on the project name to change it. -In the project drop-down list ( My Project arrow_drop_down) at the top of the Google Cloud console page, select the project you set up to contain your dataset. Make note of the project ID, as you will use that in the next steps. +1. Find your project in the Explorer pane. Click on the triple dots to see options, then click “Create dataset”. -In the Explorer panel, in the pinned projects section, click arrow_right your-project-ID to expand the project to view any existing datasets and saved queries. +1. Enter a Dataset ID. As a reminder, this dataset will hold data for all projects funded by your Billing Account. -Next to the project ID, click the View actions menu (more_vert) and then click Create dataset. The Create dataset panel opens. +1. Next, it asks for a location. You must select `US (multiple-regions)` in order to export detailed usage cost data. -Enter a Dataset ID. We recommend an ID that spans projects, such as all_billing_data, rather than a project-specific ID. -Select a Data location. The data location specifies the region where your data is stored. All tables within this dataset share this location. +1. Make sure that “Enable table expiration” is NOT checked. Otherwise your billing data will be deleted when it expires. -When creating a dataset, we recommend that you select a multi-region option for its location, such as United States (US) or European Union (EU). This is required for exporting the detailed usage cost data. +1. Click “CREATE DATASET”. -After you create the dataset, the location cannot be changed. Learn more about locations. -Ensure that the Enable table expiration option is cleared. - -If you enable table expiration and enter a number of days, any new table created in this dataset is automatically deleted following the specified number of days after creation. Important: If you delete any tables containing exported Cloud Billing data records, those records are gone and we cannot backfill the deleted records. - -In the Advanced options section, ensure that the Use a customer-managed encryption key (CMEK) option is cleared. - -Customer-managed key encryption is not supported for exporting Cloud Billing data records to BigQuery. - -To save, click Create dataset. +``` +# copied from GCP docs: +In the New Project window that appears, enter a project name and select a billing account as applicable. ``` - -**Create BigQuery Dataset** - to store the billing export **Enable data export** Go to the billing tab in the gcloud console and enable export to this dataset Make sure the user of this software has the BigQuery scope on the billing project @@ -108,16 +90,3 @@ Note that: - You only need one BigQuery project per Google Billing Account. One BigQuery Project can handle cost data for multiple Billing Projects (i.e. lab members, cohorts, etc.), as long as they are all under the same Google Billing Account. - If you are using multiple Google Billing Accounts (i.e. have multiple funding sources) you will need to set up a BigQuery project for each of them. -1. [Launch Terra](https://anvil.terra.bio/#workspaces) - -1. Locate the Workspace you want to clone. If a Workspace has been shared with you ahead of time, it will appear in "MY WORKSPACES". You can clone a Workspace that was shared with you to perform your own analyses. In the screenshot below, no Workspaces have been shared. - - ```{r, echo=FALSE, fig.alt='Screenshot of Terra "MY WORKSPACES" menu. The "MY WORKSPACES" tab is highlighted. No Workspaces are visible because none have been shared with the user. There is an option to create a new Workspace.'} -leanbuild::include_slide("https://docs.google.com/presentation/d/1ioYY0n3CcrP934WRKEsQ1aKocPqfswdWWHxYCf76BCg/edit#slide=id.gf5172664d7_0_142") - ``` - -1. If a Workspace hasn't been shared with you, navigate to the "FEATURED" or "PUBLIC" Workspace tabs. - - ```{r, echo=FALSE, fig.alt='Screenshot of Terra "My Workspaces" menu. The "FEATURED" tab is highlighted.'} -leanbuild::include_slide("https://docs.google.com/presentation/d/1ioYY0n3CcrP934WRKEsQ1aKocPqfswdWWHxYCf76BCg/edit#slide=id.gf5172664d7_0_335") - ``` From 8cbb4d1eac9fede7888f0892be3a0524020c4614 Mon Sep 17 00:00:00 2001 From: Katherine Cox Date: Tue, 19 Jul 2022 14:01:15 -0400 Subject: [PATCH 09/15] add screenshots for creating BigQuery dataset --- _child_google_billing_export_to_BigQuery.Rmd | 38 ++++++++++++++++++-- 1 file changed, 35 insertions(+), 3 deletions(-) diff --git a/_child_google_billing_export_to_BigQuery.Rmd b/_child_google_billing_export_to_BigQuery.Rmd index 02cd4c6f..88b680d8 100644 --- a/_child_google_billing_export_to_BigQuery.Rmd +++ b/_child_google_billing_export_to_BigQuery.Rmd @@ -33,19 +33,19 @@ ottrpal::include_slide("https://docs.google.com/presentation/d/1JejmHJHLTF-eywlZ 1. Enter a name for your new project. Project names can contain only letters, numbers, single quotes, hyphens, spaces, or exclamation points, and must be between 4 and 30 characters. - ```{r, echo=FALSE, fig.alt='Screenshot of the GCP Project creation dialog. The text box labeled "Project name" is highlighted.'} '} + ```{r, echo=FALSE, fig.alt='Screenshot of the GCP Project creation dialog. The text box labeled "Project name" is highlighted.'} ottrpal::include_slide("https://docs.google.com/presentation/d/1JejmHJHLTF-eywlZs5GUQXb82FCGM5wIUNrbVKlJcrY/edit#slide=id.g13d748460ee_0_153") ``` 1. You may see an option to choose a location. Unless you are part of a larger GCP organization, you can choose “No organization” for the location. - ```{r, echo=FALSE, fig.alt='Screenshot of the GCP Project creation dialog. The text box labeled "Location" is highlighted.'} '} + ```{r, echo=FALSE, fig.alt='Screenshot of the GCP Project creation dialog. The text box labeled "Location" is highlighted.'} ottrpal::include_slide("https://docs.google.com/presentation/d/1JejmHJHLTF-eywlZs5GUQXb82FCGM5wIUNrbVKlJcrY/edit#slide=id.g13d748460ee_0_172") ``` 1. Click "Create". - ```{r, echo=FALSE, fig.alt='Screenshot of the GCP Project creation dialog. The "Create" button is highlighted.'} '} + ```{r, echo=FALSE, fig.alt='Screenshot of the GCP Project creation dialog. The "Create" button is highlighted.'} ottrpal::include_slide("https://docs.google.com/presentation/d/1JejmHJHLTF-eywlZs5GUQXb82FCGM5wIUNrbVKlJcrY/edit#slide=id.g13d748460ee_0_166") ``` @@ -58,20 +58,52 @@ To create a BigQuery Dataset: 1. Open the dropdown menu on the top left, and click “BigQuery.” You may need to scroll down quite a ways before you see it. + ```{r, echo=FALSE, fig.alt='Screenshot of the Google Cloud Console drop-down menu. "BigQuery" is highlighted.'} +ottrpal::include_slide("https://docs.google.com/presentation/d/1JejmHJHLTF-eywlZs5GUQXb82FCGM5wIUNrbVKlJcrY/edit#slide=id.g13d748460ee_0_183") + ``` + - If you’d like, you can pin BigQuery so it’s easier to find next time. + + ```{r, echo=FALSE, fig.alt='Screenshot of the Google Cloud Console drop-down menu. The pin icon next to "BigQuery" is highlighted.'} +ottrpal::include_slide("https://docs.google.com/presentation/d/1JejmHJHLTF-eywlZs5GUQXb82FCGM5wIUNrbVKlJcrY/edit#slide=id.g13d748460ee_0_189") + ``` 1. Make sure the correct GCP project is selected. You can see the current project at the top of the screen. If this is not correct, click on the project name to change it. + ```{r, echo=FALSE, fig.alt='Screenshot of the GCP BigQuery interface. The project name at the top of the page is highlighted.'} +ottrpal::include_slide("https://docs.google.com/presentation/d/1JejmHJHLTF-eywlZs5GUQXb82FCGM5wIUNrbVKlJcrY/edit#slide=id.g13d748460ee_0_195") + ``` + 1. Find your project in the Explorer pane. Click on the triple dots to see options, then click “Create dataset”. + ```{r, echo=FALSE, fig.alt='Screenshot of the GCP BigQuery interface. In the Explorer pane, the triple dots next to the project named anvil-project-pi-billing are highlighted, the menu is expanded, and "Create dataset" is highlighted.'} +ottrpal::include_slide("https://docs.google.com/presentation/d/1JejmHJHLTF-eywlZs5GUQXb82FCGM5wIUNrbVKlJcrY/edit#slide=id.g13d748460ee_0_208") + ``` + 1. Enter a Dataset ID. As a reminder, this dataset will hold data for all projects funded by your Billing Account. + ```{r, echo=FALSE, fig.alt='Screenshot of the GCP BigQuery dataset creation dialog. The textbox labeled "Dataset ID" is highlighted.'} +ottrpal::include_slide("https://docs.google.com/presentation/d/1JejmHJHLTF-eywlZs5GUQXb82FCGM5wIUNrbVKlJcrY/edit#slide=id.g13d748460ee_0_215") + ``` + 1. Next, it asks for a location. You must select `US (multiple-regions)` in order to export detailed usage cost data. + ```{r, echo=FALSE, fig.alt='Screenshot of the GCP BigQuery dataset creation dialog. The dropdown menu labeled "Data location" is highlighted.'} +ottrpal::include_slide("https://docs.google.com/presentation/d/1JejmHJHLTF-eywlZs5GUQXb82FCGM5wIUNrbVKlJcrY/edit#slide=id.g13d748460ee_0_221") + ``` + 1. Make sure that “Enable table expiration” is NOT checked. Otherwise your billing data will be deleted when it expires. + ```{r, echo=FALSE, fig.alt='Screenshot of the GCP BigQuery dataset creation dialog. The checkbox labeled "Enable table expiration" is highlighted and is not checked.'} +ottrpal::include_slide("https://docs.google.com/presentation/d/1JejmHJHLTF-eywlZs5GUQXb82FCGM5wIUNrbVKlJcrY/edit#slide=id.g13d748460ee_0_227") + ``` + 1. Click “CREATE DATASET”. + ```{r, echo=FALSE, fig.alt='Screenshot of the GCP BigQuery dataset creation dialog. The "CREATE DATASET" button is highlighted.'} +ottrpal::include_slide("https://docs.google.com/presentation/d/1JejmHJHLTF-eywlZs5GUQXb82FCGM5wIUNrbVKlJcrY/edit#slide=id.g13d748460ee_0_233") + ``` + ``` # copied from GCP docs: From 8f2a6236478aaaac72fe9c87297f270c6cdb0142 Mon Sep 17 00:00:00 2001 From: Katherine Cox Date: Tue, 19 Jul 2022 18:41:35 -0400 Subject: [PATCH 10/15] start instructions to enable billing data export --- 02-pis.Rmd | 8 ++-- _child_google_billing_export_to_BigQuery.Rmd | 45 +++++++++++++++----- 2 files changed, 37 insertions(+), 16 deletions(-) diff --git a/02-pis.Rmd b/02-pis.Rmd index e3bd8d23..e4bb1a18 100644 --- a/02-pis.Rmd +++ b/02-pis.Rmd @@ -228,9 +228,7 @@ cow::borrow_chapter( ### Export Cost Data to BigQuery -Several useful tools are available to help you further analyze and understand your GCP computing costs. Examples include the `AnVILBilling` [Bioconductor package](http://www.bioconductor.org/packages/release/bioc/html/AnVILBilling.html) and [Google Data Studio](https://cloud.google.com/billing/docs/how-to/visualize-data). - -These tools can help give greater insight into the charges to your GCP Billing Account which can help you refine your workflows to minimize expenses. +Several useful tools are available to help you further analyze and understand your GCP computing costs. Examples include the `AnVILBilling` [Bioconductor package](http://www.bioconductor.org/packages/release/bioc/html/AnVILBilling.html), as well ass [Google Data Studio](https://cloud.google.com/billing/docs/how-to/visualize-data). These tools can help give greater insight into the charges to your GCP Billing Account which can help you refine your workflows to minimize expenses. In order to make use of these tools, you will need to set up exporting of your cost data to BigQuery. This cannot be done retroactively (i.e. you can only export cost data as it is produced; there is no way to export your cost data history). Therefore, **we recommend setting up cost data export immediately, so the cost data will be available to you in the future**. @@ -242,7 +240,7 @@ In order to make use of these tools, you will need to set up exporting of your c ``` -Once you have set up data exporting, you will be able to analyze your cost data at your convenience. To learn more about how to analyze your cost data, see: +Now that you have set up data export, your billing data will be available for you to analyze at your convenience. To learn more about how to analyze your cost data, see: - The [vignette](http://www.bioconductor.org/packages/release/bioc/vignettes/AnVILBilling/inst/doc/billing.html) for the [`AnVILBilling`](http://www.bioconductor.org/packages/release/bioc/html/AnVILBilling.html) - [Google's documentation](https://cloud.google.com/billing/docs/how-to/visualize-data) on visualizing spending with Google Data Studio @@ -250,7 +248,7 @@ Once you have set up data exporting, you will be able to analyze your cost data **Having trouble?** - Visit our community support forum at [`help.anvilproject.org`](https://help.anvilproject.org) with any questions. -- View the [`Google documentation`](https://cloud.google.com/billing/docs/how-to/export-data-bigquery) +- View the [`Google documentation`](https://cloud.google.com/billing/docs/how-to/export-data-bigquery) on exporting billing data. ## Step 6: Add Users and Workspaces diff --git a/_child_google_billing_export_to_BigQuery.Rmd b/_child_google_billing_export_to_BigQuery.Rmd index 88b680d8..6c8c4d53 100644 --- a/_child_google_billing_export_to_BigQuery.Rmd +++ b/_child_google_billing_export_to_BigQuery.Rmd @@ -5,7 +5,7 @@ Exporting billing data to BigQuery requires a few steps to set up: 1. Create GCP Project 1. Create BigQuery Dataset -1. Enable data export +1. Enable billing data export **Create GCP Project** @@ -25,7 +25,7 @@ To create a new GCP Project: 1. Log in to the [Google Cloud Platform](https://console.cloud.google.com/) console using the Google ID associated with your Google Cloud projects. -1. Open the dropdown menu on the top left, mouseover IAM & Admin, and click “Create a Project.” +1. Open the dropdown menu on the top left, mouseover IAM & Admin, and click "Create a Project". ```{r, echo=FALSE, fig.alt='Screenshot of the Google Cloud Console drop-down menu, with the "IAM & Admin" submenu open. "IAM & Admin" and "Create a Project" are highlighted.'} ottrpal::include_slide("https://docs.google.com/presentation/d/1JejmHJHLTF-eywlZs5GUQXb82FCGM5wIUNrbVKlJcrY/edit#slide=id.g13d748460ee_0_3") @@ -37,7 +37,7 @@ ottrpal::include_slide("https://docs.google.com/presentation/d/1JejmHJHLTF-eywlZ ottrpal::include_slide("https://docs.google.com/presentation/d/1JejmHJHLTF-eywlZs5GUQXb82FCGM5wIUNrbVKlJcrY/edit#slide=id.g13d748460ee_0_153") ``` -1. You may see an option to choose a location. Unless you are part of a larger GCP organization, you can choose “No organization” for the location. +1. You may see an option to choose a location. Unless you are part of a larger GCP organization, you can choose "No organization" for the location. ```{r, echo=FALSE, fig.alt='Screenshot of the GCP Project creation dialog. The text box labeled "Location" is highlighted.'} ottrpal::include_slide("https://docs.google.com/presentation/d/1JejmHJHLTF-eywlZs5GUQXb82FCGM5wIUNrbVKlJcrY/edit#slide=id.g13d748460ee_0_172") @@ -56,7 +56,7 @@ Once your GCP Project exists, it can be used to create a BigQuery Dataset to hol To create a BigQuery Dataset: -1. Open the dropdown menu on the top left, and click “BigQuery.” You may need to scroll down quite a ways before you see it. +1. Open the dropdown menu on the top left, and click "BigQuery". You may need to scroll down quite a ways before you see it. ```{r, echo=FALSE, fig.alt='Screenshot of the Google Cloud Console drop-down menu. "BigQuery" is highlighted.'} ottrpal::include_slide("https://docs.google.com/presentation/d/1JejmHJHLTF-eywlZs5GUQXb82FCGM5wIUNrbVKlJcrY/edit#slide=id.g13d748460ee_0_183") @@ -74,7 +74,7 @@ ottrpal::include_slide("https://docs.google.com/presentation/d/1JejmHJHLTF-eywlZ ottrpal::include_slide("https://docs.google.com/presentation/d/1JejmHJHLTF-eywlZs5GUQXb82FCGM5wIUNrbVKlJcrY/edit#slide=id.g13d748460ee_0_195") ``` -1. Find your project in the Explorer pane. Click on the triple dots to see options, then click “Create dataset”. +1. Find your project in the Explorer pane. Click on the triple dots to see options, then click "Create dataset". ```{r, echo=FALSE, fig.alt='Screenshot of the GCP BigQuery interface. In the Explorer pane, the triple dots next to the project named anvil-project-pi-billing are highlighted, the menu is expanded, and "Create dataset" is highlighted.'} ottrpal::include_slide("https://docs.google.com/presentation/d/1JejmHJHLTF-eywlZs5GUQXb82FCGM5wIUNrbVKlJcrY/edit#slide=id.g13d748460ee_0_208") @@ -92,30 +92,53 @@ ottrpal::include_slide("https://docs.google.com/presentation/d/1JejmHJHLTF-eywlZ ottrpal::include_slide("https://docs.google.com/presentation/d/1JejmHJHLTF-eywlZs5GUQXb82FCGM5wIUNrbVKlJcrY/edit#slide=id.g13d748460ee_0_221") ``` -1. Make sure that “Enable table expiration” is NOT checked. Otherwise your billing data will be deleted when it expires. +1. Make sure that "Enable table expiration" is NOT checked. Otherwise your billing data will be deleted when it expires. ```{r, echo=FALSE, fig.alt='Screenshot of the GCP BigQuery dataset creation dialog. The checkbox labeled "Enable table expiration" is highlighted and is not checked.'} ottrpal::include_slide("https://docs.google.com/presentation/d/1JejmHJHLTF-eywlZs5GUQXb82FCGM5wIUNrbVKlJcrY/edit#slide=id.g13d748460ee_0_227") ``` -1. Click “CREATE DATASET”. +1. Click "CREATE DATASET". ```{r, echo=FALSE, fig.alt='Screenshot of the GCP BigQuery dataset creation dialog. The "CREATE DATASET" button is highlighted.'} ottrpal::include_slide("https://docs.google.com/presentation/d/1JejmHJHLTF-eywlZs5GUQXb82FCGM5wIUNrbVKlJcrY/edit#slide=id.g13d748460ee_0_233") ``` +1. If you click the arrow next to the project name, you should see your new dataset. It's empty right now, so there's not much to see here yet. + + ```{r, echo=FALSE, fig.alt='Screenshot of the GCP BigQuery interface. In the Explorer pane, the arrow next to the project named anvil-project-pi-billing is highlighted, '} +ottrpal::include_slide("https://docs.google.com/presentation/d/1JejmHJHLTF-eywlZs5GUQXb82FCGM5wIUNrbVKlJcrY/edit#slide=id.g13d748460ee_0_239") + ``` + + +**Enable billing data export** + +Once the BigQuery dataset exists, you can export data to it. + +To enable billing data export: + +1. Open the dropdown menu on the top left and click "Billing". +1. Make sure the correct Billing Account is selected. You can see the current account in the upper left of the screen. If this is not correct, click on the Billing Account name to change it. +1. Click "Billing export". +1. On the "BIGQUERY EXPORT" tab, under "Detailed usage cost", click "EDIT SETTINGS". +1. Select the project that contains the BigQuery Dataset. +1. Select the BigQuery dataset. +1. Click "SAVE". +1. You should now see that export is enabled. It will take about a day before data starts showing up in your BigQuery Dataset. + + + + ``` # copied from GCP docs: In the New Project window that appears, enter a project name and select a billing account as applicable. -``` - -**Enable data export** - Go to the billing tab in the gcloud console and enable export to this dataset +# From AnVILBilling Make sure the user of this software has the BigQuery scope on the billing project +``` Note that: From 562e7caf412aac1954ad49a9439985d96e2b5672 Mon Sep 17 00:00:00 2001 From: Katherine Cox Date: Wed, 20 Jul 2022 14:27:24 -0400 Subject: [PATCH 11/15] screenshots for enabling billing data export --- 02-pis.Rmd | 4 +- _child_google_billing_export_to_BigQuery.Rmd | 47 ++++++++++++++++---- 2 files changed, 39 insertions(+), 12 deletions(-) diff --git a/02-pis.Rmd b/02-pis.Rmd index e4bb1a18..63ef0dfd 100644 --- a/02-pis.Rmd +++ b/02-pis.Rmd @@ -228,14 +228,12 @@ cow::borrow_chapter( ### Export Cost Data to BigQuery -Several useful tools are available to help you further analyze and understand your GCP computing costs. Examples include the `AnVILBilling` [Bioconductor package](http://www.bioconductor.org/packages/release/bioc/html/AnVILBilling.html), as well ass [Google Data Studio](https://cloud.google.com/billing/docs/how-to/visualize-data). These tools can help give greater insight into the charges to your GCP Billing Account which can help you refine your workflows to minimize expenses. +Several useful tools are available to help you further analyze and understand your GCP computing costs. Examples include the `AnVILBilling` [Bioconductor package](http://www.bioconductor.org/packages/release/bioc/html/AnVILBilling.html), as well as [Google Data Studio](https://cloud.google.com/billing/docs/how-to/visualize-data). These tools can help give greater insight into the charges to your GCP Billing Account which can help you refine your workflows to minimize expenses. In order to make use of these tools, you will need to set up exporting of your cost data to BigQuery. This cannot be done retroactively (i.e. you can only export cost data as it is produced; there is no way to export your cost data history). Therefore, **we recommend setting up cost data export immediately, so the cost data will be available to you in the future**. **NOTE**: This will incur a minimal monthly fee, less than $1 / month (pricing as of Feb 2021) -#### Set Up Data Export - ```{r, child=c("_child_google_billing_export_to_BigQuery.Rmd")} ``` diff --git a/_child_google_billing_export_to_BigQuery.Rmd b/_child_google_billing_export_to_BigQuery.Rmd index 6c8c4d53..a0b7c866 100644 --- a/_child_google_billing_export_to_BigQuery.Rmd +++ b/_child_google_billing_export_to_BigQuery.Rmd @@ -106,7 +106,7 @@ ottrpal::include_slide("https://docs.google.com/presentation/d/1JejmHJHLTF-eywlZ 1. If you click the arrow next to the project name, you should see your new dataset. It's empty right now, so there's not much to see here yet. - ```{r, echo=FALSE, fig.alt='Screenshot of the GCP BigQuery interface. In the Explorer pane, the arrow next to the project named anvil-project-pi-billing is highlighted, '} + ```{r, echo=FALSE, fig.alt='Screenshot of the GCP BigQuery interface. In the Explorer pane, the arrow next to the project named anvil-project-pi-billing is highlighted.'} ottrpal::include_slide("https://docs.google.com/presentation/d/1JejmHJHLTF-eywlZs5GUQXb82FCGM5wIUNrbVKlJcrY/edit#slide=id.g13d748460ee_0_239") ``` @@ -118,27 +118,56 @@ Once the BigQuery dataset exists, you can export data to it. To enable billing data export: 1. Open the dropdown menu on the top left and click "Billing". + + ```{r, echo=FALSE, fig.alt='Screenshot of the Google Cloud Console drop-down menu. "Billing" is highlighted.'} +ottrpal::include_slide("https://docs.google.com/presentation/d/1JejmHJHLTF-eywlZs5GUQXb82FCGM5wIUNrbVKlJcrY/edit#slide=id.g13d748460ee_0_250") + ``` + 1. Make sure the correct Billing Account is selected. You can see the current account in the upper left of the screen. If this is not correct, click on the Billing Account name to change it. + + ```{r, echo=FALSE, fig.alt='Screenshot of the GCP Billing interface. In the left panel, the name of the Billing Account is highlighted.'} +ottrpal::include_slide("https://docs.google.com/presentation/d/1JejmHJHLTF-eywlZs5GUQXb82FCGM5wIUNrbVKlJcrY/edit#slide=id.g13d748460ee_0_245") + ``` + 1. Click "Billing export". + + ```{r, echo=FALSE, fig.alt='Screenshot of the GCP Billing interface. In the left panel, "Billing export" is highlighted.'} +ottrpal::include_slide("https://docs.google.com/presentation/d/1JejmHJHLTF-eywlZs5GUQXb82FCGM5wIUNrbVKlJcrY/edit#slide=id.g13d748460ee_0_256") + ``` + 1. On the "BIGQUERY EXPORT" tab, under "Detailed usage cost", click "EDIT SETTINGS". + + ```{r, echo=FALSE, fig.alt='Screenshot of the GCP Billing export tab. The name of the tab is highlighted. Under "Detailed usage cost", the "EDIT SETTINGS" button is highlighted.'} +ottrpal::include_slide("https://docs.google.com/presentation/d/1JejmHJHLTF-eywlZs5GUQXb82FCGM5wIUNrbVKlJcrY/edit#slide=id.g13d748460ee_0_263") + ``` + 1. Select the project that contains the BigQuery Dataset. + + ```{r, echo=FALSE, fig.alt='Screenshot of the GCP Billing export dialog. The dropdown menu labeled "Projects" is highlighted.'} +ottrpal::include_slide("https://docs.google.com/presentation/d/1JejmHJHLTF-eywlZs5GUQXb82FCGM5wIUNrbVKlJcrY/edit#slide=id.g13d748460ee_0_270") + ``` + 1. Select the BigQuery dataset. -1. Click "SAVE". -1. You should now see that export is enabled. It will take about a day before data starts showing up in your BigQuery Dataset. + ```{r, echo=FALSE, fig.alt='Screenshot of the GCP Billing export dialog. The dropdown menu labeled "Dataset" is highlighted.'} +ottrpal::include_slide("https://docs.google.com/presentation/d/1JejmHJHLTF-eywlZs5GUQXb82FCGM5wIUNrbVKlJcrY/edit#slide=id.g13d748460ee_0_276") + ``` +1. Click "SAVE". + ```{r, echo=FALSE, fig.alt='Screenshot of the GCP Billing export dialog. The "SAVE" button is highlighted.'} +ottrpal::include_slide("https://docs.google.com/presentation/d/1JejmHJHLTF-eywlZs5GUQXb82FCGM5wIUNrbVKlJcrY/edit#slide=id.g13d748460ee_0_282") + ``` +1. You should now see that export is enabled. It will take about a day before data starts showing up in your BigQuery Dataset. -``` -# copied from GCP docs: + ```{r, echo=FALSE, fig.alt='Screenshot of the GCP Billing export tab. The checkmark indicating that export of detailed cost usage data is enabled is highlighted.'} +ottrpal::include_slide("https://docs.google.com/presentation/d/1JejmHJHLTF-eywlZs5GUQXb82FCGM5wIUNrbVKlJcrY/edit#slide=id.g13d748460ee_0_290") + ``` -In the New Project window that appears, enter a project name and select a billing account as applicable. -# From AnVILBilling -Make sure the user of this software has the BigQuery scope on the billing project -``` +#### Draft/notes Note that: From ee91dec8d92ef0d7935023060584904e258728cd Mon Sep 17 00:00:00 2001 From: Katherine Cox Date: Wed, 20 Jul 2022 18:14:48 -0400 Subject: [PATCH 12/15] add comments on BigQuery per Billing Account --- _child_google_billing_export_to_BigQuery.Rmd | 14 +++++--------- 1 file changed, 5 insertions(+), 9 deletions(-) diff --git a/_child_google_billing_export_to_BigQuery.Rmd b/_child_google_billing_export_to_BigQuery.Rmd index a0b7c866..5396b48d 100644 --- a/_child_google_billing_export_to_BigQuery.Rmd +++ b/_child_google_billing_export_to_BigQuery.Rmd @@ -20,6 +20,11 @@ Reminder: Usually your GCP Projects are created and managed by Terra (corresponding to Terra Workspaces) and you don't need to manage them through the Google Cloud interface. In this case, you will need to create the GCP Project yourself. **This must be under the same Google Billing Account as the Projects you want to track spending for.** +Note that: + +- You only need one BigQuery project per Google Billing Account. One BigQuery Project can handle cost data for multiple Projects (i.e. Terra Workspaces), as long as they are all under the same Google Billing Account. +- If you are using multiple Google Billing Accounts (i.e. have multiple funding sources) you will need to set up a BigQuery project for each of them. + To create a new GCP Project: @@ -165,12 +170,3 @@ ottrpal::include_slide("https://docs.google.com/presentation/d/1JejmHJHLTF-eywlZ ottrpal::include_slide("https://docs.google.com/presentation/d/1JejmHJHLTF-eywlZs5GUQXb82FCGM5wIUNrbVKlJcrY/edit#slide=id.g13d748460ee_0_290") ``` - - -#### Draft/notes - -Note that: - -- You only need one BigQuery project per Google Billing Account. One BigQuery Project can handle cost data for multiple Billing Projects (i.e. lab members, cohorts, etc.), as long as they are all under the same Google Billing Account. -- If you are using multiple Google Billing Accounts (i.e. have multiple funding sources) you will need to set up a BigQuery project for each of them. - From 8f52338f22a16ffb350db882f60b37a80a074cc6 Mon Sep 17 00:00:00 2001 From: Katherine Cox Date: Wed, 20 Jul 2022 18:30:44 -0400 Subject: [PATCH 13/15] updates to BigQuery section --- 02-pis.Rmd | 1 - _child_google_billing_export_to_BigQuery.Rmd | 4 ++-- 2 files changed, 2 insertions(+), 3 deletions(-) diff --git a/02-pis.Rmd b/02-pis.Rmd index 63ef0dfd..9a0a4990 100644 --- a/02-pis.Rmd +++ b/02-pis.Rmd @@ -232,7 +232,6 @@ Several useful tools are available to help you further analyze and understand yo In order to make use of these tools, you will need to set up exporting of your cost data to BigQuery. This cannot be done retroactively (i.e. you can only export cost data as it is produced; there is no way to export your cost data history). Therefore, **we recommend setting up cost data export immediately, so the cost data will be available to you in the future**. -**NOTE**: This will incur a minimal monthly fee, less than $1 / month (pricing as of Feb 2021) ```{r, child=c("_child_google_billing_export_to_BigQuery.Rmd")} diff --git a/_child_google_billing_export_to_BigQuery.Rmd b/_child_google_billing_export_to_BigQuery.Rmd index 5396b48d..f2aeec58 100644 --- a/_child_google_billing_export_to_BigQuery.Rmd +++ b/_child_google_billing_export_to_BigQuery.Rmd @@ -22,8 +22,8 @@ Usually your GCP Projects are created and managed by Terra (corresponding to Ter Note that: -- You only need one BigQuery project per Google Billing Account. One BigQuery Project can handle cost data for multiple Projects (i.e. Terra Workspaces), as long as they are all under the same Google Billing Account. -- If you are using multiple Google Billing Accounts (i.e. have multiple funding sources) you will need to set up a BigQuery project for each of them. +- You only need one GCP project and corresponding BigQuery Dataset per Google Billing Account. One BigQuery Dataset can handle cost data for multiple Projects (i.e. Terra Workspaces), as long as they are all under the same Google Billing Account. +- If you are using multiple Google Billing Accounts (i.e. have multiple funding sources) you will need to set up a GCP Project and a BigQuery Dataset for each of them. To create a new GCP Project: From b7fdea1176e2724d70e3b1469d12606f2f6ebcfd Mon Sep 17 00:00:00 2001 From: Katherine Cox Date: Wed, 20 Jul 2022 18:50:53 -0400 Subject: [PATCH 14/15] type --- 02-pis.Rmd | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/02-pis.Rmd b/02-pis.Rmd index 9a0a4990..53a1d8e9 100644 --- a/02-pis.Rmd +++ b/02-pis.Rmd @@ -239,7 +239,7 @@ In order to make use of these tools, you will need to set up exporting of your c Now that you have set up data export, your billing data will be available for you to analyze at your convenience. To learn more about how to analyze your cost data, see: -- The [vignette](http://www.bioconductor.org/packages/release/bioc/vignettes/AnVILBilling/inst/doc/billing.html) for the [`AnVILBilling`](http://www.bioconductor.org/packages/release/bioc/html/AnVILBilling.html) +- The [vignette](http://www.bioconductor.org/packages/release/bioc/vignettes/AnVILBilling/inst/doc/billing.html) for the [`AnVILBilling`](http://www.bioconductor.org/packages/release/bioc/html/AnVILBilling.html) package - [Google's documentation](https://cloud.google.com/billing/docs/how-to/visualize-data) on visualizing spending with Google Data Studio **Having trouble?** From 4cbf88c561815b4301d35c1f8d7639e3720eb8a6 Mon Sep 17 00:00:00 2001 From: Katherine Cox Date: Thu, 4 Aug 2022 15:53:35 -0400 Subject: [PATCH 15/15] revise some BigQuery screenshots --- _child_google_billing_export_to_BigQuery.Rmd | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/_child_google_billing_export_to_BigQuery.Rmd b/_child_google_billing_export_to_BigQuery.Rmd index f2aeec58..02da3e0f 100644 --- a/_child_google_billing_export_to_BigQuery.Rmd +++ b/_child_google_billing_export_to_BigQuery.Rmd @@ -42,9 +42,15 @@ ottrpal::include_slide("https://docs.google.com/presentation/d/1JejmHJHLTF-eywlZ ottrpal::include_slide("https://docs.google.com/presentation/d/1JejmHJHLTF-eywlZs5GUQXb82FCGM5wIUNrbVKlJcrY/edit#slide=id.g13d748460ee_0_153") ``` +1. If you have more than one Billing Account, select the Billing Account you wish to export data for. If you only have one Billing Account, you may not see this option. + + ```{r, echo=FALSE, fig.alt='Screenshot of the GCP Project creation dialog. The dropdown labeled "Billing account" is highlighted.'} +ottrpal::include_slide("https://docs.google.com/presentation/d/1JejmHJHLTF-eywlZs5GUQXb82FCGM5wIUNrbVKlJcrY/edit#slide=id.g1405b9d7496_0_21") + ``` + 1. You may see an option to choose a location. Unless you are part of a larger GCP organization, you can choose "No organization" for the location. - ```{r, echo=FALSE, fig.alt='Screenshot of the GCP Project creation dialog. The text box labeled "Location" is highlighted.'} + ```{r, echo=FALSE, fig.alt='Screenshot of the GCP Project creation dialog. The dropdown labeled "Location" is highlighted.'} ottrpal::include_slide("https://docs.google.com/presentation/d/1JejmHJHLTF-eywlZs5GUQXb82FCGM5wIUNrbVKlJcrY/edit#slide=id.g13d748460ee_0_172") ```