Skip to content

Commit 41a2599

Browse files
authored
CI enhancements + staging cluster (GoogleCloudPlatform#425)
* init * cleanup * Remove kind CI * remove KIND CI * Remove license check * temporarily remove productcatalog unit test * Add redis wait condition * Update github actions readme * re-add loadgen smoke test * Move comment step to the end of ci-pr.yaml
1 parent 5372b01 commit 41a2599

File tree

12 files changed

+475
-153
lines changed

12 files changed

+475
-153
lines changed

.github/master-cluster/README.md

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
# onlineboutique.dev manifests
2+
3+
This directory contains extra deploy manifests for configuring a domain name/static IP to point to an Online Boutique deployment running in GKE.
Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
apiVersion: networking.k8s.io/v1beta1
2+
kind: Ingress
3+
metadata:
4+
name: frontend-ingress
5+
annotations:
6+
kubernetes.io/ingress.global-static-ip-name: ob-global-ip
7+
networking.gke.io/managed-certificates: ob-cert
8+
spec:
9+
backend:
10+
serviceName: frontend-nodeport
11+
servicePort: 80
Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
apiVersion: v1
2+
kind: Service
3+
metadata:
4+
name: frontend-nodeport
5+
spec:
6+
selector:
7+
app: frontend
8+
type: NodePort
9+
ports:
10+
- protocol: TCP
11+
port: 80
12+
targetPort: 8080
Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
apiVersion: networking.gke.io/v1beta2
2+
kind: ManagedCertificate
3+
metadata:
4+
name: ob-cert
5+
spec:
6+
domains:
7+
- onlineboutique.dev

.github/pull_request_template.md

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
Fixes # .
2+
3+
Change summary:
4+
-
5+
6+
7+
Remaining issues / concerns:

.github/workflows/README.md

Lines changed: 59 additions & 73 deletions
Original file line numberDiff line numberDiff line change
@@ -1,75 +1,61 @@
11
# GitHub Actions Workflows
22

3-
## Setup
4-
- workloads run using [GitHub self-hosted runners](https://help.github.com/en/actions/automating-your-workflow-with-github-actions/about-self-hosted-runners)
5-
- project admins maintain a private Google Compute Engine VM for running tests
6-
- VM should be at least n1-standard-4 with 50GB persistent disk
7-
- instructions for setting up the VM can be found in repo settings under "Actions"
8-
- ⚠️ WARNING: VM should be set up with no GCP service account
9-
- external contributors could contribute malicious PRs to run code on our test VM. Ensure no service accounts or other secrets exist on the VM
10-
- An empty GCP project should be used for extra security
11-
- to set up dependencies, run the following commands:
12-
```
13-
# install kubectl
14-
sudo apt-get install -yqq kubectl git
15-
16-
# install go
17-
curl -O https://storage.googleapis.com/golang/go1.12.9.linux-amd64.tar.gz
18-
tar -xvf go1.12.9.linux-amd64.tar.gz
19-
sudo chown -R root:root ./go
20-
sudo mv go /usr/local
21-
echo 'export GOPATH=$HOME/go' >> ~/.profile
22-
echo 'export PATH=$PATH:/usr/local/go/bin:$GOPATH/bin' >> ~/.profile
23-
source ~/.profile
24-
25-
# install addlicense
26-
go get -u github.com/google/addlicense
27-
sudo ln -s $HOME/go/bin/addlicense /bin
28-
29-
# install kind
30-
curl -Lo ./kind "https://github.com/kubernetes-sigs/kind/releases/download/v0.7.0/kind-$(uname)-amd64" && \
31-
chmod +x ./kind && \
32-
sudo mv ./kind /usr/local/bin
33-
34-
# install skaffold
35-
curl -Lo skaffold https://storage.googleapis.com/skaffold/releases/latest/skaffold-linux-amd64 && \
36-
chmod +x skaffold && \
37-
sudo mv skaffold /usr/local/bin
38-
39-
# install docker
40-
sudo apt install -yqq apt-transport-https ca-certificates curl gnupg2 software-properties-common && \
41-
curl -fsSL https://download.docker.com/linux/debian/gpg | sudo apt-key add - && \
42-
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/debian $(lsb_release -cs) stable" && \
43-
sudo apt-get update && \
44-
sudo apt-get install -yqq docker-ce && \
45-
sudo usermod -aG docker ${USER}
46-
47-
# logout and back on
48-
exit
49-
```
50-
- ensure GitHub Actions runs as background service:
51-
```
52-
sudo ∼/actions-runner/svc.sh install
53-
sudo ∼/actions-runner/svc.sh start
54-
```
55-
56-
57-
---
58-
## Workflows
59-
60-
### ci.yaml
61-
62-
#### Triggers
63-
- commits pushed to master
64-
- PRs to master
65-
- PRs to release/ branches
66-
67-
#### Actions
68-
- ensures kind cluster is running
69-
- builds all containers in src/
70-
- deploys local containers to kind
71-
- ensures all pods reach ready state
72-
- ensures HTTP request to frontend returns HTTP status 200
73-
- deploys manifests from /releases
74-
- ensures all pods reach ready state
75-
- ensures HTTP request to frontend returns HTTP status 200
3+
This page describes the CI/CD workflows for the Online Boutique app, which run in [Github Actions](https://github.com/GoogleCloudPlatform/microservices-demo/actions).
4+
5+
## Infrastructure
6+
7+
The CI/CD pipelines for Online Boutique run in Github Actions, using a pool of two [self-hosted runners]((https://help.github.com/en/actions/automating-your-workflow-with-github-actions/about-self-hosted-runners)). These runners are GCE instances (virtual machines) that, for every open Pull Request in the repo, run the code test pipeline, deploy test pipeline, and (on master) deploy the latest version of the app to [onlineboutique.dev](https://onlineboutique.dev)
8+
9+
We also host a test GKE cluster, which is where the deploy tests run. Every PR has its own namespace in the cluster.
10+
11+
## Workflows
12+
13+
**Note**: In order for the current CI/CD setup to work on your pull request, you must branch directly off the repo (no forks). This is because the Github secrets necessary for these tests aren't copied over when you fork.
14+
15+
### Code Tests - [ci-pr.yaml](ci-pr.yaml)
16+
17+
These tests run on every commit for every open PR, as well as any commit to master / any release branch. Currently, this workflow runs only Go unit tests.
18+
19+
20+
### Deploy Tests- [ci-pr.yaml](ci-pr.yaml)
21+
22+
These tests run on every commit for every open PR, as well as any commit to master / any release branch. This workflow:
23+
24+
1. Creates a dedicated GKE namespace for that PR, if it doesn't already exist, in the PR GKE cluster.
25+
2. Uses `skaffold run` to build and push the images specific to that PR commit. Then skaffold deploys those images, via `kubernetes-manifests`, to the PR namespace in the test cluster.
26+
3. Tests to make sure all the pods start up and become ready.
27+
4. Gets the LoadBalancer IP for the frontend service.
28+
5. Comments that IP in the pull request, for staging.
29+
30+
### Push and Deploy Latest - [push-deploy](push-deploy.yml)
31+
32+
This is the Continuous Deployment workflow, and it runs on every commit to the master branch. This workflow:
33+
34+
1. Builds the contaner images for every service, tagging as `latest`.
35+
2. Pushes those images to Google Container Registry.
36+
37+
Note that this workflow does not update the image tags used in `release/kubernetes-manifests.yaml` - these release manifests are tied to a stable `v0.x.x` release.
38+
39+
### Cleanup - [cleanup.yaml](cleanup.yaml)
40+
41+
This workflow runs when a PR closes, regardless of whether it was merged into master. This workflow deletes the PR-specific GKE namespace in the test cluster.
42+
43+
## Appendix - Creating a new Actions runner
44+
45+
Should one of the two self-hosted Github Actions runners (GCE instances) fail, or you want to add more runner capacity, this is how to provision a new runner. Note that you need IAM access to the admin Online Boutique GCP project in order to do this.
46+
47+
1. Create a GCE instance.
48+
- VM should be at least n1-standard-4 with 50GB persistent disk
49+
- VM should use custom service account with permissions to: access a GKE cluster, create GCS storage buckets, and push to GCR.
50+
2. SSH into new VM through the Google Cloud Console.
51+
3. Follow the instructions to add a new runner on the [Actions Settings page](https://github.com/GoogleCloudPlatform/bank-of-anthos/settings/actions) to authenticate the new runner
52+
4. Start GitHub Actions as a background service:
53+
```
54+
sudo ~/actions-runner/svc.sh install ; sudo ~/actions-runner/svc.sh start
55+
```
56+
5. Install project-specific dependencies, including go, docker, skaffold, and kubectl:
57+
58+
```
59+
wget -O - https://raw.githubusercontent.com/GoogleCloudPlatform/microservices-demo/master/.github/workflows/install-dependencies.sh | bash
60+
```
61+

.github/workflows/ci-master.yaml

Lines changed: 104 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,104 @@
1+
# Copyright 2020 Google LLC
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
name: "Continuous Integration - Master/Release"
16+
on:
17+
push:
18+
# run on pushes to master or release/*
19+
branches:
20+
- master
21+
- release/*
22+
jobs:
23+
code-tests:
24+
runs-on: self-hosted
25+
steps:
26+
- uses: actions/checkout@v2
27+
- name: Go Unit Tests
28+
timeout-minutes: 10
29+
run: |
30+
for SERVICE in "shippingservice"; do
31+
echo "testing $SERVICE..."
32+
pushd src/$SERVICE
33+
go test
34+
popd
35+
done
36+
# - name: TODO - run C# Unit Tests
37+
# timeout-minutes: 10
38+
# run: |
39+
# dotnet test src/cartservice/tests/cartservice.tests.csproj
40+
deployment-tests:
41+
runs-on: self-hosted
42+
needs: code-tests
43+
strategy:
44+
matrix:
45+
profile: ["local-code"]
46+
fail-fast: true
47+
steps:
48+
- uses: actions/checkout@v2
49+
- name: Build + Deploy PR images to GKE
50+
timeout-minutes: 20
51+
run: |
52+
PR_NUMBER=$(echo $GITHUB_REF | awk 'BEGIN { FS = "/" } ; { print $3 }')
53+
NAMESPACE="pr${PR_NUMBER}"
54+
echo "::set-env name=NAMESPACE::$NAMESPACE"
55+
echo "::set-env name=PR_NUMBER::$PR_NUMBER"
56+
57+
gcloud container clusters get-credentials $PR_CLUSTER --zone $ZONE --project $PROJECT_ID
58+
cat <<EOF | kubectl apply -f -
59+
apiVersion: v1
60+
kind: Namespace
61+
metadata:
62+
name: $NAMESPACE
63+
EOF
64+
echo Deploying application
65+
skaffold config set --global local-cluster false
66+
skaffold run --default-repo=gcr.io/$PROJECT_ID/$GITHUB_REF --tag=$GITHUB_SHA --namespace=$NAMESPACE
67+
env:
68+
PROJECT_ID: ${{ secrets.PROJECT_ID }}
69+
PR_CLUSTER: ${{ secrets.PR_CLUSTER }}
70+
ZONE: ${{ secrets.PR_CLUSTER_ZONE }}
71+
- name: Wait For Pods
72+
timeout-minutes: 20
73+
run: |
74+
set -x
75+
kubectl config set-context --current --namespace=$NAMESPACE
76+
kubectl wait --for=condition=available --timeout=1000s deployment/redis-cart
77+
kubectl wait --for=condition=available --timeout=1000s deployment/adservice
78+
kubectl wait --for=condition=available --timeout=1000s deployment/cartservice
79+
kubectl wait --for=condition=available --timeout=1000s deployment/checkoutservice
80+
kubectl wait --for=condition=available --timeout=1000s deployment/currencyservice
81+
kubectl wait --for=condition=available --timeout=1000s deployment/emailservice
82+
kubectl wait --for=condition=available --timeout=1000s deployment/frontend
83+
kubectl wait --for=condition=available --timeout=1000s deployment/loadgenerator
84+
kubectl wait --for=condition=available --timeout=1000s deployment/paymentservice
85+
kubectl wait --for=condition=available --timeout=1000s deployment/productcatalogservice
86+
kubectl wait --for=condition=available --timeout=1000s deployment/recommendationservice
87+
kubectl wait --for=condition=available --timeout=1000s deployment/shippingservice
88+
- name: Smoke Test
89+
timeout-minutes: 5
90+
run: |
91+
set -x
92+
# start fresh loadgenerator pod
93+
kubectl delete pod -l app=loadgenerator
94+
# wait for requests to come in
95+
REQUEST_COUNT="0"
96+
while [[ "$REQUEST_COUNT" -lt "50" ]]; do
97+
sleep 5
98+
REQUEST_COUNT=$(kubectl logs -l app=loadgenerator | grep Aggregated | awk '{print $2}')
99+
done
100+
# ensure there are no errors hitting endpoints
101+
ERROR_COUNT=$(kubectl logs -l app=loadgenerator | grep Aggregated | awk '{print $3}' | sed "s/[(][^)]*[)]//g")
102+
if [[ "$ERROR_COUNT" -gt "0" ]]; then
103+
exit 1
104+
fi

0 commit comments

Comments
 (0)