Gke lcs

.gitignore

@@ -2,3 +2,5 @@
 __pycache__
 *.sw[op]
 *.py[oc]
+.env
+/venv

GKE_README.txt

@@ -0,0 +1,63 @@
+make sure gcloud sdk is installed and pathed correctly
+
+gcloud auth login
+gcloud projects list
+gcloud config set project MY_PROJECT_NAME
+
+
+# push the images to google contianer registry
+gcloud auth configure-docker
+
+docker build --tag myproject:1 --file myproject/Dockerfile .
+docker tag myproject:1 gcr.io/first-304221/myproject:1
+docker push gcr.io/first-304221/myproject:1
+
+docker build --tag consumer-small:1 --file consumer-small/Dockerfile .
+docker tag consumer-small:1 gcr.io/first-304221/consumer-small:1
+docker push gcr.io/first-304221/consumer-small:1
+
+docker build --tag consumer-large:1 --file consumer-large/Dockerfile .
+docker tag consumer-large:1 gcr.io/first-304221/consumer-large:1
+docker push gcr.io/first-304221/consumer-large:1
+
+
+# GKE
+gcloud components install kubectl
+
+gcloud config set project MY_PROJECT_NAME
+gcloud config set compute/zone ZONE_NAME
+
+
+# -------- my-celery-cluster -------------
+gcloud container clusters create my-celery-cluster --num-nodes=1 \
+--preemptible \
+--enable-autoscaling --max-nodes=5 --min-nodes=1
+
+gcloud container clusters get-credentials my-celery-cluster
+
+
+kubectl create -f message_queue/rabbitmq-deployment.yaml
+kubectl create -f message_queue/rabbitmq-service.yaml
+
+
+# I changed the yml to point to the image in gcr
+kubectl create -f myproject/deployment.yaml
+kubectl create -f consumer-large/deployment.yaml
+
+kubectl get pods
+
+
+# to find external ip
+kubectl get service myproject-svc
+
+
+# DELETE CLUSTER to avoid addtional charges
+gcloud container clusters delete my-celery-cluster
+
+
+
+
+
+
+
+

README.md

@@ -36,23 +36,61 @@ Assuming `dockerd` is running and `minikube` is installed, let's deploy the syst
 
 ### Initialization
 
-Create a Minikube cluster that uses the local `dockerd` environment:
+Make sure you have google cloud sdk installed and pathed correctly.
+https://cloud.google.com/sdk/docs/install
+
+then auth you google credentials and either make a new project or use an exiting one.
+```
+gcloud auth login
+gcloud projects list
+gcloud projects create example-foo-bar-project-name-1 --name="Happy project"
+```
+or use an exiting project 
+```
+gcloud config set project MY_PROJECT_NAME
+```
+
+run this to allow you to push to google container registry and when prompted say yes.
 ```
-minikube start
-eval $(minikube -p minikube docker-env)
+gcloud auth configure-docker
 ```
-Build all Docker images for Minikube:
+
+
+Build and push you Docker images to GRC:
 ```
 docker build --tag myproject:1 --file myproject/Dockerfile .
+docker tag myproject:1 gcr.io/first-304221/myproject:1
+docker push gcr.io/first-304221/myproject:1
+
 docker build --tag consumer-small:1 --file consumer-small/Dockerfile .
+docker tag consumer-small:1 gcr.io/first-304221/consumer-small:1
+docker push gcr.io/first-304221/consumer-small:1
+
 docker build --tag consumer-large:1 --file consumer-large/Dockerfile .
+docker tag consumer-large:1 gcr.io/first-304221/consumer-large:1
+docker push gcr.io/first-304221/consumer-large:1
 ```
-Check that the images were created successfully:
+
+
+### Deploying applications
+
+Install gcloud kubectl and set project 
+```
+gcloud components install kubectl
+
+gcloud config set project MY_PROJECT_NAME
+gcloud config set compute/zone ZONE_NAME
 ```
-docker images
+
+Create a GKE cluster with preemptive (spot) instance processors. And pass credentials to kubectl.
+```
+gcloud container clusters create my-celery-cluster --num-nodes=1 \
+--preemptible \
+--enable-autoscaling --max-nodes=5 --min-nodes=1
+
+gcloud container clusters get-credentials my-celery-cluster
 ```
 
-### Deploying applications
 
 Deploy the RabbitMQ message broker as a service inside the cluster (I left some config files for Redis too but they are not used here):
 ```
@@ -106,11 +144,11 @@ I prefer to open new terminals or `tmux` for all applications and then use `kube
 
 ### Interacting with the web app
 
-Now everything is running and we can expose the Flask web app port to our local machine:
+Now everything is running and we can get the exposed port from the Load balancer service 
 ```
-kubectl port-forward deployment/myproject 5000:5000
+kubectl get service myproject-svc
 ```
-Then open http://localhost:5000/ in a browser and you should see a simple web UI.
+Then got to the exteranl ip in a browser, and you should see a simple web UI.
 
 Try copy-pasting some strings and compute the longest common substrings for them.
 E.g. first try short strings and check that the tasks show up in the Celery logs of pod `consumer-small`.
@@ -155,9 +193,8 @@ celery inspect report --broker=$CELERY_BROKER_URL
 ```
 
 ## Cleanup
-Terminate all pods by removing the deployments:
+Terminate the cluster to avoid any additional charges
 ```
-kubectl delete deploy myproject consumer-large rabbitmq
-kubectl delete service rabbitmq-service
+gcloud container clusters delete my-celery-cluster
 ```
 

consumer-large/celery_app.py

@@ -5,6 +5,7 @@
 from celery import Celery
 from lcs.tasks import longest_common_substr
 
+
 def make_celery():
     return Celery("consumer-large",
             backend=os.environ.get("CELERY_RESULT_BACKEND"),

consumer-large/deployment.yaml

@@ -16,8 +16,9 @@ spec:
     spec:
       containers:
       - name: consumer-large
-        image: consumer-large:1
-        imagePullPolicy: Never
+        # image: consumer-large:1
+        image: gcr.io/first-304221/consumer-large:1
+        imagePullPolicy: Always
         env:
         - name: CELERY_BROKER_URL
           value: "amqp://guest:guest@rabbitmq-service:5672"

lcs/build/lib/lcs/__init__.py

@@ -0,0 +1,12 @@
+def longest_common_substr(str_a, str_b):
+    longest_str = ''
+    for i in range(len(str_a)):
+        for j in range(len(str_b)):
+            current_str = ''
+            k = 0
+            while i + k < len(str_a) and j + k < len(str_b) and str_a[i + k] == str_b[j + k]:
+                current_str += str_a[i + k]
+                k += 1
+            if k > len(longest_str):
+                longest_str = current_str
+    return longest_str

lcs/build/lib/lcs/tasks.py

@@ -0,0 +1,9 @@
+"""
+Celery task wrapper for longest_common_substr.
+"""
+import lcs
+from celery import shared_task
+
+@shared_task
+def longest_common_substr(str_a, str_b):
+    return lcs.longest_common_substr(str_a, str_b)

myproject/Dockerfile

@@ -7,5 +7,6 @@ COPY myproject/templates /usr/src/myproject/templates
 RUN apk add --no-cache python3 py3-pip bash \
     && pip install /usr/src/lcs \
     && pip install -r /usr/src/myproject/requirements.txt
+EXPOSE 5000
 WORKDIR /usr/src/myproject
 ENTRYPOINT python3 main.py

myproject/deployment.yaml

@@ -1,3 +1,19 @@
+# LoadBalancer Service
+# Enables the pods in a deployment to be accessible from outside the cluster
+apiVersion: v1
+kind: Service
+metadata:
+  name: myproject-svc
+spec:
+  selector:
+    app: myproject
+  ports:
+    - protocol: "TCP"
+      port: 80
+      targetPort: 5000
+  type: LoadBalancer
+
+---
 apiVersion: apps/v1
 kind: Deployment
 metadata:
@@ -21,8 +37,12 @@ spec:
         emptyDir: {}
       containers:
       - name: myproject
-        image: myproject:1
-        imagePullPolicy: Never
+        # I added this
+        ports:
+          - containerPort: 5000
+        image: gcr.io/first-304221/myproject:1
+        # image: myproject:1
+        imagePullPolicy: Always
         volumeMounts:
         - mountPath: /data
           name: tmp-database
@@ -39,8 +59,9 @@ spec:
 #         - name: CELERY_RESULT_BACKEND
 #           value: 'redis://redis-service:6379/1'
       - name: consumer-small
-        image: consumer-small:1
-        imagePullPolicy: Never
+        # image: consumer-small:1
+        image: gcr.io/first-304221/consumer-small:1
+        imagePullPolicy: Always
         volumeMounts:
         - mountPath: /data
           name: tmp-database

myproject/main.py

@@ -7,11 +7,13 @@
 
 import flask
 
+
 import database
 import settings
 import tasks
 
 
+
 flask_app = flask.Flask(__name__)
 flask_app.logger.setLevel(logging.INFO)
 flask_app.config.update(
@@ -24,6 +26,8 @@
 def timestamp2iso(t):
     return '' if t is None else datetime.datetime.fromtimestamp(t).isoformat()
 
+
+
 @flask_app.route("/", methods=["GET"])
 def get_index():
     return flask.render_template("index.html")
@@ -58,4 +62,4 @@ def post_task():
         flask_app.logger.info("Created database %s", settings.database_path)
     else:
         flask_app.logger.info("Using existing database %s", settings.database_path)
-    flask_app.run()
+    flask_app.run(host='0.0.0.0', port='5000')