Merge pull request #415 from cloudfoundry/improve_db_setup

Improve db deletion protection

Author: jochenehret

docs/concourse/README.md

@@ -111,10 +111,26 @@ terragrunt run-all apply
 ## Recommendations
 ### Cloud SQL Instance deletion protection
 
-Terraform hashicorp provider includes a deletion protection flag however in some cases it's misleading as it's not setting it on Google Cloud.
-To avoid confusion we do not set it in the code and recommend altering your production SQL Instance to protect from the deletion on the cloud side.
+The [database.tf](../../terraform-modules/concourse/infra/database.tf) configuration enables deletion protection on multiple levels. The Terraform hashicorp provider includes a deletion protection flag:
+```
+resource "google_sql_database_instance" "concourse" {
+
+  # This option prevents Terraform from deleting an instance
+  deletion_protection = true
+```
+Note that if you really want to delete the database, Terraform will not allow this because `deletion_protection = true` is stored in the state. You first have to disable this flag, then run `apply` and then you can run a deletion operation.
+
+In addition, we are setting a flag that enables the "Prevent instance deletion" option from the GCP console:
+```
+  settings {
+    deletion_protection_enabled = "true"
+  }
+```
+
+:warning: The option "Retain backups after instance deletion" should also be enabled. There is no Terraform configuration parameter,
+so you have to set it manually in the GCP console:
 
-https://console.cloud.google.com/sql/instances/ -> select instance name -> edit ->  Data Protection -> tick: Enable delete protection
+Cloud SQL -> Instances -> Edit configuration -> Data Protection -> Retain backups after instance deletion
 
 ### End-to-end testing
 

docs/concourse/concourse_minor_version_upgrade.md

@@ -20,6 +20,15 @@ Please note the process should be also useful for upgrading major versions.
    ```
    terragrunt run-all plan --terragrunt-source-update
    ```
+   Note: The Cloud SQL database has "automatic storage increases" enabled. So the disk could have grown larger than the initial value from the `config.yaml` file. In that case, Terraform would have to shrink the disk which is not possible. Instead, it tries to destroy the database and recreate it, losing all data including backups:
+   ```
+   STDOUT [infra] tofu: -/+ resource "google_sql_database_instance" "concourse" {
+   STDOUT [infra] tofu:       ~ settings {
+   STDOUT [infra] tofu:           ~ disk_size                    = 44 -> 38 # forces replacement
+   (...)
+   STDOUT [infra] tofu: Plan: 1 to add, 1 to change, 1 to destroy.
+   ```
+   Deletion protection is enabled on Terraform level, so this change could not be applied. To proceed, configure a bigger `sql_instance_disk_size` in the `config.yaml`.
 
 4. Switch to `renovate's` pull request having bumped Concourse helm chart version
    ```
@@ -32,7 +41,16 @@ Please note the process should be also useful for upgrading major versions.
    ../terragrunt/scripts/concourse/create-sql-backup.sh
    ```
 
-6. Apply roll-out for new Concourse version
+6. Check Cloud SQL "Data Protection" settings
+
+In the GCP console, navigate to:
+https://console.cloud.google.com/sql/instances/ -> select instance name -> edit -> Data Protection
+
+Make sure the following flags are enabled:
+* "Prevent instance deletion"
+* "Retain backups after instance deletion"
+
+7. Apply roll-out for new Concourse version
    ```
    terragrunt run-all apply --terragrunt-source-update
    ```

terraform-modules/concourse/infra/database.tf

@@ -4,10 +4,8 @@ resource "google_sql_database_instance" "concourse" {
   project          = var.project
   region           = var.region
 
-  # recommended protection via GCP SQL Instance settings
-  # https://console.cloud.google.com/sql/instances/ -> select instance name -> edit
-  # ->  Data Protection -> tick: Enable delete protection
-  deletion_protection = false
+  # This option prevents Terraform from deleting an instance
+  deletion_protection = true
 
   settings {
     activation_policy = "ALWAYS"
@@ -28,6 +26,8 @@ resource "google_sql_database_instance" "concourse" {
       transaction_log_retention_days = "7"
     }
 
+    deletion_protection_enabled = "true"
+
     disk_autoresize       = "true"
     disk_autoresize_limit = "0"
     disk_size             = var.sql_instance_disk_size