Skip to content

Latest commit

 

History

History
325 lines (230 loc) · 9.45 KB

GCP.md

File metadata and controls

325 lines (230 loc) · 9.45 KB

Terraform Google Cloud Platform (GCP)

  1. Install gcloud CLI
  2. Log into GCP from CLI
  3. Push Docker image to GCP Artifact Registry
  4. Customize Terraform settings
  5. Deploy
  6. Misc

Terraform plan

You can run the following command to see all the resources that will be created by Terraform:

cd scripts/deploy/terraform/gcp
terraform plan

By default, here are the resources that will be created.


1. Install gcloud CLI

Follow these instructions to install the CLI on your workstation.

For macOS and Homebrew, you can do the following:

brew install --cask google-cloud-sdk

2. Log into GCP from CLI

If you don’t already have an account in GCP, create one here.

Once you created an account, from your terminal run the following command to log in:

gcloud init
gcloud auth application-default login

3. Push Docker image to GCP Artifact Registry

GCP doesn’t support using Docker images from Docker Hub. We’ll need to pull the Mage Docker image, tag it, and push it to GCP Artifact Registry.

Here are the steps we’ll take:

  1. Create a repository on GCP Artifact Registry by clicking the + CREATE REPOSITORY button.

    • Fill in the Name
    • Choose Docker as the Format
    • Choose any Location type
    • Choose any Encryption
    • Click CREATE

  2. Click on the newly created repository (e.g. mage-docker).

  3. Near the top of the page, click the link SETUP INSTRUCTIONS or read these instructions to set up authentication for Docker.

    TLDR - Run the following command in your terminal:

    gcloud auth configure-docker [region]-docker.pkg.dev

    An example command could look like this: bash gcloud auth configure-docker us-west2-docker.pkg.dev

  4. Pull the Mage Docker image:

    docker pull mageai/mageai:latest

    If you’re local workstation is using macOS and a silicon chip (e.g. M1, M2, etc), then run this command instead:

     docker pull --platform linux/amd64 mageai/mageai:latest

  5. Tag the pulled Mage docker image or use a previously tagged Docker image you built when following this CI/CD guide:

    1. Sample commands if using vanilla Mage Docker image:

      docker tag mageai/mageai:latest [region]-docker.pkg.dev/[project_id]/[repository]/mageai:latest

      An example command could look like this:

       docker tag mageai/mageai:latest \
   us-west2-docker.pkg.dev/materia-284023/mage-docker/mageai:latest

    2. Sample commands if using previously tagged custom Docker image:

      docker tag mageprod:latest [region]-docker.pkg.dev/[project_id]/[repository]/mageai:latest

      An example command could look like this:

       docker tag mageprod:latest \
   us-west2-docker.pkg.dev/materia-284023/mage-docker/mageai:latest

  6. Push the local Docker image to GCP Artifact Registry:

    docker push [region]-docker.pkg.dev/[project_id]/[repository]/mageai:latest

    An example command could look like this:

     docker push us-west2-docker.pkg.dev/materia-284023/mage-docker/mageai:latest

4. Customize Terraform settings

Project ID (REQUIRED)

Before running any Terraform commands, please change the default value of the variable named project_id in the ./scripts/deploy/terraform/gcp/variables.tf file.

variable "project_id" {
  type        = string
  description = "The name of the project"
  default     = "unique-gcp-project-id"
}

Docker image (REQUIRED)

In the repository you created in GCP Artifact Repository, you’ll see a list of Docker images. Click on an image name, then copy the full path to the image (e.g. us-west2-docker.pkg.dev/materia-284023/mage-docker/mageai).

In the file ./scripts/deploy/terraform/gcp/variables.tf, you can change the default value under docker_image:

variable "docker_image" {
  type        = string
  description = "The Docker image url in the Artifact Registry repository to be deployed to Cloud Run"
  default     = "us-west2-docker.pkg.dev/materia-284023/mage-docker/mageai"
}

Application Name

In the file ./scripts/deploy/terraform/gcp/variables.tf, you can change the default value under app_name:

variable "app_name" {
  type        = string
  description = "Application Name"
  default     = "mage-data-prep"
}

Region

In the file ./scripts/deploy/terraform/gcp/main.tf, you can change the default value under region:

variable "region" {
  type        = string
  description = "The default compute region"
  default     = "us-west2"
}

More

Other variables defined in ./scripts/deploy/terraform/gcp/variables.tf can also be customized to your needs.


5. Deploy

  1. Change directory into scripts folder:

    cd scripts/deploy/terraform/gcp

  2. Initialize Terraform:

    terraform init

    A sample output could look like this:

    Initializing the backend...

Initializing provider plugins...
- Finding hashicorp/google versions matching ">= 3.3.0"...
- Finding latest version of hashicorp/http...
- Installing hashicorp/google v4.38.0...
- Installed hashicorp/google v4.38.0 (signed by HashiCorp)
- Installing hashicorp/http v3.1.0...
- Installed hashicorp/http v3.1.0 (signed by HashiCorp)

Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

  3. Deploy:

    terraform apply

    A sample output could look like this:

    Apply complete! Resources: 7 added, 1 changed, 0 destroyed.

Outputs:

service_ip = "34.107.187.208"

    It’ll take a few minutes for GCP Cloud Run to start up and be ready to receive requests.

    After a few minutes, open a browser and go to http://[IP_address]/.

    Note

    Change the IP_address to the IP address that was output in your terminal after successfully running terraform apply.

Errors

403 Forbidden

If you get this error when trying to open the above IP address in your browser, open the security group named [application_name]-security-policy. Click on that security group and verify your IP address is whitelisted.

If it isn’t, add a new rule with the following values:

Mode Match Action Priority
Basic mode Enter your IP address Allow 100

If you still get 403, check to see if you’re using http:// and NOT https://.


Updating Mage versions

After Mage is running in your cloud, you can update the Mage Docker image version by running the following command in your CLI tool:

gcloud run deploy [app_name] --image [docker_image]

app_name

This is the value you changed when editing the ./scripts/deploy/terraform/gcp/variables.tf file.


Misc

Security

To enable other IP addresses access to Mage, open the security group named [application_name]-security-policy. Click on that security group and add a new rule with the following values:

Mode Match Action Priority
Basic mode Enter your IP address Allow 100

HTTPS enabling

TBD

Terminate all resources

If you want to delete all resources created from deploying, run the following command:

terraform destroy

A sample output could look like this:

Destroy complete! Resources: 10 destroyed.