Terraform MultiCloud K8S

Description

This terraform project provision simple nginx deployment on three cloud providers:

• AWS Elastic Kubernetes Service

• GCP Google Kubernetes Engine

• Azure Kubernetes Service

Table of Contents

• Prerequisites

• Usage

• Issues

Prerequisites

You need basic familiarity with Kubernetes and kubectl but does not assume any pre-existing deployment.

It also assumes that you are familiar with the usual Terraform plan/apply workflow. If you're new to Terraform itself, refer first to the Getting Started tutorial.

Before you start using this project, you will need:

• For EKS:

  • an AWS account

  • the AWS CLI, installed and configured

• For GKE:

  • a GCP account

  • a configured gcloud SDK

  • create project in GCP account with

• For AKS:

  • an Azure account

  • a configured Azure CLI

• Kubernetes CLI kubectl

• Git

• Terraform CLI

Usage

When you have all needed CLIs you can start using this project:

1. Clone this repo to local machine:

   git clone https://github.com/HubGab-Git/multicloud_k8s.git

2. Enter downloaded folder:

   cd multicloud_k8s

3. Open file terraform.tfvars ( for example in vs code):

   code terraform.tfvars

This file you have to fill your data from GCP and Azure:

• Create project in GCP:

   sudo gcloud projects create --name <your project name>

  • Confirm Project ID.

    console output should be like below:

    

  • Copy Project ID into terraform.tfvars file into line 7:

    

• Create Service Account for Terraform to be able provision resources:

  gcloud iam service-accounts create <Name of your Service Account>

  example:

  

• Type your service account email into terraform.tsvars file into line 22:

  

• Assign "owner" role to newly created service account:

   gcloud projects add-iam-policy-binding <ProjectID>  \  
   --member="serviceAccount:<SAName>@<ProjectID>.iam.gserviceaccount.com"  \
   role="roles/owner"

  example:

  

• Download json file with Service Account credentials:

   gcloud iam service-accounts keys create credentials.json \
       --iam-account=<SAName>@<ProjectID>.iam.gserviceaccount.com  

  example:

  

• Type path to credentials file in terraform.tfvar file into line 15:

  

• GCP data was full-filed now Azure, only one Azure command to get Service Principal credentials:

   az ad sp create-for-rbac --skip-assignment

  example:

  

• Type appId and password in terraform.tsvars file into lines 49 and 50:

  

5. Now we can start on terraform. Download all needed terraform providers and modules:

   terraform init

6. Check terraform plan if this is your desired project state:

   terraform plan

7. Finally provision all resources:

   terraform apply --auto-approve

8. Now you can check deployments in EKS, GKE and AKS:

• EKS:
  • Connect EKS with kubectl:

     aws eks --region $(terraform output -raw aws_region) update-kubeconfig \ 
     --name $(terraform output -raw aws_cluster_name)

  • Show e.q. pods:

     kubectl get po

• GKE:
  • Connect GKE with kubectl:

    gcloud container clusters get-credentials $(terraform output -raw gcp_cluster_name) --region $(terraform output -raw gcp_region)

  • Show e.q. pods:

     kubectl get po

• AKS:
  • Connect AKS with kubectl:

    az aks get-credentials --resource-group $(terraform output -raw azure_resource_group_name) --name $(terraform output -raw azure_cluster_name)

  • Show e.q. pods:

     az aks command invoke \
       --resource-group $(terraform output -raw azure_resource_group_name) \
       --name $(terraform output -raw azure_cluster_name) \
       --command "kubectl get po" 

Issues

Provisioning can take about 30-120 minutes.

If you face errors like below:

Error: Failed to identify fetch peer certificates
│
│ with module.eks_aws.module.eks.data.tls_certificate.this[0],
│ on .terraform/modules/eks_aws.eks/main.tf line 201, in data "tls_certificate" "this":
│ 201: data "tls_certificate" "this" {
│
│ failed to fetch certificates from URL 'https': Get "https://oidc.eks.eu-west-1.amazonaws.com:443/id/7BE4EB75A4A654B5D7B7246E8BD2BF4E": dial tcp: lookup
│ oidc.eks.eu-west-1.amazonaws.com on 62.179.1.62:53: read udp 192.168.1.145:0->62.179.1.62:53: i/o timeout


│ Error: Failed to create deployment: Post "https://7BE4EB75A4A654B5D7B7246E8BD2BF4E.yl4.eu-west-1.eks.amazonaws.com/apis/apps/v1/namespaces/default/deployments": dial tcp: lookup 7BE4EB75A4A654B5D7B7246E8BD2BF4E.yl4.eu-west-1.eks.amazonaws.com: i/o timeout
│
│ with module.k8s_aws.kubernetes_deployment.nginx,
│ on k8s_deployment/deployment.tf line 3, in resource "kubernetes_deployment" "nginx":
│ 3: resource "kubernetes_deployment" "nginx" {
│
╵

Or Any error which contains:

 ... dial tcp: lookup ...
 ... i/o timeout ...

This probably means that terraform cannot handle IPv6 addresses correctly. my fast workaround for this is assign IPv4 addresses for affected domains:

1. Copy domain and find IPv4 for it, for example by "ping" command:

2. Type domain name and IP into "hosts" file, bellow my example host file:

   ##
   # Host Database
   #
   # localhost is used to configure the loopback interface
   # when the system is booting. Do not change this entry.
   ##
   127.0.0.1  localhost
   255.255.255.255  broadcasthost
   ::1  localhost
   142.250.203.202  oauth2.googleapis.com
   216.58.215.74  compute.googleapis.com
   142.250.186.202  container.googleapis.com
   172.217.16.10  cloudresourcemanager.googleapis.com
   142.250.203.138  iam.googleapis.com
   216.58.208.202  serviceusage.googleapis.com
   # Added by Docker Desktop
   # To allow the same kube context to work on the host and the container:
   127.0.0.1  kubernetes.docker.internal
   # End of section
   51.116.62.189  management.azure.com
   51.11.173.167  k8s-3d28e68f1819d816-0a1504f2.hcp.uksouth.azmk8s.io
   34.248.242.151  78cda35666e1ca76dd11076693eaab38.gr7.eu-west-1.eks.amazonaws.com
   52.49.87.145  7be4eb75a4a654b5d7b7246e8bd2bf4e.yl4.eu-west-1.eks.amazonaws.com
   52.208.7.143  oidc.eks.eu-west-1.amazonaws.com
   51.143.243.78  k8s-1efe6846b93f2095-423df77c.hcp.uksouth.azmk8s.io
   54.229.44.223  E642F2F9F0953B1086B35B0BBA53E2B4.sk1.eu-west-1.eks.amazonaws.com

   What I understand is that issue is rather caused by "Go lang" rather than terraform which is developed by Go More information about this issue under this link: hashicorp/terraform-provider-google#6782

Google service errors:

There can appear errors like below:

```md
│ Error: Request `Enable Project Service "container.googleapis.com" for project "multicloud-demo-364707"` returned error: Batch request and retried single request "Enable Project Service \"container.googleapis.com\" for project \"multicloud-demo-364707\"" both failed. Final error: failed to send enable services request: googleapi: Error 400: Billing account for project '626917844434' is not found. Billing must be enabled for activation of service(s) 'container.googleapis.com,container.googleapis.com,compute.googleapis.com,compute.googleapis.com,compute.googleapis.com,containerregistry.googleapis.com' to proceed.
│ Help Token: AWzfkCMjI4BqYvhEaXaY93g33EdnpRaRr7LkR_2iAMQ2s8br-ygL9eANYXVkjum-SY315Q-TZA5D_rABinYSCZTOBeuG1FBXShXNEtfwAHTTxX11
│ Details:
│ [
│   {
│     "@type": "type.googleapis.com/google.rpc.PreconditionFailure",
│     "violations": [
│       {
│         "subject": "?error_code=390001\u0026project=626917844434\u0026services=container.googleapis.com\u0026services=container.googleapis.com\u0026services=compute.googleapis.com\u0026services=compute.googleapis.com\u0026services=compute.googleapis.com\u0026services=containerregistry.googleapis.com",
│         "type": "googleapis.com/billing-enabled"
│       }
│     ]
│   },
│   {
│     "@type": "type.googleapis.com/google.rpc.ErrorInfo",
│     "domain": "serviceusage.googleapis.com/billing-enabled",
│     "metadata": {
│       "project": "626917844434",
│       "services": "container.googleapis.com,container.googleapis.com,compute.googleapis.com,compute.googleapis.com,compute.googleapis.com,containerregistry.googleapis.com"
│     },
│     "reason": "UREQ_PROJECT_BILLING_NOT_FOUND"
│   }
│ ]
│ , failedPrecondition
│ 
│   with module.gke_gcp.google_project_service.container,
│   on gke_gcp/services.tf line 11, in resource "google_project_service" "container":
│   11: resource "google_project_service" "container" {
│ 
```

Workaroiund for them is to manualy enable Compute and Container services:

https://console.cloud.google.com/marketplace/product/google/container.googleapis.com
https://console.cloud.google.com/marketplace/product/google/compute.googleapis.com