otel-demo

This repo contains instructions and scripts to run the open telemetry demo app in Minikube.

Installation

Installation on Debian/Ubuntu/PopOS Linux

You need to have direnv installed. If you do not already have it, you can install the Debian package:

sudo apt-get install direnv

Next, install minikube:

curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube_latest_amd64.deb
sudo dpkg -i minikube_latest_amd64.deb

Download the latest release of Helm for your platform from GitHub. You can untar the archive as follows:

tar xzf helm-v3.18.2-linux-amd64.tar.gz

Replace 3.18.2 with the latest version.

Now, copy the helm executable to ~/bin/helm.

Download and install the kubectl executable: https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/#install-kubectl-binary-with-curl-on-linux

Now, run minikube start

Remote proxy

If you are running minikube on a remote machine, keep the following running to have a remote proxy:

kubectl proxy --address='0.0.0.0' --accept-hosts='^*$'

This proxy is for the dashboard.

You will be able to access the dashboard at: http://REMOTE-HOST:8001/api/v1/namespaces/kubernetes-dashboard/services/http:kubernetes-dashboard:/proxy/#/workloads?namespace=default

Metrics server

You can enable basic kubernetes metrics by running minikube addons enable metrics-server

Starting the otel demo app

make otel-setup

Use the dashboard to follow the progress of the startup. When everything has started, run:

make otel-forward

You should now see the main app at http://HOSTNAME:8080. This is also the proxy for the other installed utilties, like Grafana (on http://HOSTNAME:8080/grafana).

Adding kube-state-metrics

First install it:

helm install kube-state-metrics prometheus-community/kube-state-metrics   --namespace kube-system   --create-namespace

Get the current otel config and update the prometheus scraper config:

helm get values my-otel-demo -a > otel-current-values.yaml
cp otel-current-values.yaml otel-new-values.yaml
vi otel-new-values.yaml # edit the file, removing all the unchanged configuration and adding the new scraper config
helm upgrade my-otel-demo open-telemetry/opentelemetry-demo --values otel-new-values.yaml

To see the metrics that are being published, forward its port:

kubectl --namespace kube-system port-forward --address='0.0.0.0' svc/kube-state-metrics 8081:8080

Search for the metric kube_pod_container_status_waiting_reason in the text.

Now go to http://HOSTNAME:8081/metrics

Grafana alert configuration

See grafana-config/README.md for details on configuring contact points, dashboards, and alert rules.

Remote access to minikube

I found it useful to run kubectl on my laptop to access minikube on a remote host. To do this, I had to do the following:

1. You need the kubnetes config file on your local machine. The file can be usually be found at ~/.kube/config on the remote host. You can scp it over to whereever you want to keep it on your laptop.
2. You need local copies of the certificates created by minikube and referenced in the config file. These will include the certificate authority (ca.crt), the client certificate (client.crt), and the client key (`client.key). Copy them to your laptop and then update the paths in the kubernetes confile file to point to the correct local locations.
3. The API server entry ("server" in the config file) is likely pointing to an IP on the host's private network. You can either run a proxy on the host or run an ssh tunnel from the client. Here's how to run a client-side tunnel:

   HOST_PRIVATE_IP="..." # you can get this via `minikube ip`
   HOST_PUBLIC_IP="..." # public ip or hostname for the remote host
   ssh -L 8443:$(HOST_PRIVATE_IP):8443 $(HOST_PUBLIC_IP) -N

4. You need to adjust the server entry in the config file. If you did client-side tunneling, this should point to https://localhost:8443.
5. On your labtop, set the environment variable KUBECONFIG to point to your new kubernetes config file.
6. As a sanity test, you can run something like kubectl get pods

Configuration Changes

Address OOM issues

• Updated the memory for the "ad" deployment from 300Mi to 400Mi
• Updated the memory for the "fraud-detection" deployment from 300Mi to 600Mi
• Updated the memory for the "prometheus-server" container in the "prometheus" deployment from 300Mi to 500Mi
• Updated the memory for the kafka deployment from 600Mi to 800Mi

See RCA/CrashLoop.md for details.