basic instrumentation for the operator

Signed-off-by: Elieser Pereira <[email protected]>

Author: elieserr

.github/workflows/linkinator.yaml

@@ -34,11 +34,7 @@ jobs:
             --retry-wait-time 2
             --timeout 30
             --github-token ${{ secrets.GITHUB_TOKEN }}
-            --exclude '^https://github\.com/kedacore/http-add-on/pkgs/container/http-add-on-interceptor$'
-            --exclude '^https://github\.com/kedacore/http-add-on/pkgs/container/http-add-on-operator$'
-            --exclude '^https://github\.com/kedacore/http-add-on/pkgs/container/http-add-on-scaler$'
-            --exclude '^http://opentelemetry-collector\.open-telemetry-system:4318$'
-            --exclude '^http://opentelemetry-collector\.open-telemetry-system:4318/$'
-            --exclude '^http://opentelemetry-collector\.open-telemetry-system:4318/v1/traces$'
+            --exclude '^https://github\.com/kedacore/http-add-on/pkgs/container/http-add-on-(interceptor|operator|scaler)(/.*)?$'
+            --exclude '^http://opentelemetry-collector\.open-telemetry-system:431[7-8](/.*)?$'
             --exclude '^https://www\.gnu\.org/software/make/$'
             "./**/*.md"

CHANGELOG.md

@@ -31,7 +31,7 @@ This changelog keeps track of work items that have been completed and are ready
 
 ### Improvements
 
-- **General**: TODO ([#TODO](https://github.com/kedacore/http-add-on/issues/TODO))
+- **General**: Add prometehus and otel instrumentation for the operator ([#965](https://github.com/kedacore/http-add-on/issues/965))
 
 ### Fixes
 

Makefile

@@ -230,6 +230,9 @@ deploy: manifests kustomize ## Deploy to the K8s cluster specified in ~/.kube/co
 	cd config/operator && \
 	$(KUSTOMIZE) edit set image ghcr.io/kedacore/http-add-on-operator=${IMAGE_OPERATOR_VERSIONED_TAG}
 
+	cd config/operator && \
+	$(KUSTOMIZE) edit add patch --path e2e-test/otel/deployment.yaml --group apps --kind Deployment --name operator --version v1
+
 	$(KUSTOMIZE) build config/default | kubectl apply -f -
 
 undeploy:

config/operator/e2e-test/otel/deployment.yaml

@@ -0,0 +1,23 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: operator
+spec:
+  replicas: 1
+  template:
+    spec:
+      containers:
+      - name: operator
+        env:
+        - name: OTEL_PROM_EXPORTER_ENABLED
+          value: "true"
+        - name: OTEL_PROM_EXPORTER_PORT
+          value: "8080"
+        - name: OTEL_EXPORTER_OTLP_METRICS_ENABLED
+          value: "true"
+        - name: OTEL_EXPORTER_OTLP_ENDPOINT
+          value: "http://opentelemetry-collector.open-telemetry-system:4318"
+        - name: OTEL_METRIC_EXPORT_INTERVAL
+          value: "1"
+        - name: OTEL_EXPORTER_OTLP_PROTOCOL
+          value: "http"

config/operator/e2e-test/otel/kustomization.yaml

@@ -0,0 +1,4 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+resources:
+- deployment.yaml

config/operator/kustomization.yaml

@@ -5,6 +5,7 @@ resources:
 - role.yaml
 - role_binding.yaml
 - service_account.yaml
+- metrics.service.yaml
 labels:
 - includeSelectors: true
   includeTemplates: true

config/operator/metrics.service.yaml

@@ -0,0 +1,11 @@
+apiVersion: v1
+kind: Service
+metadata:
+  name: operator-metrics
+spec:
+  type: ClusterIP
+  ports:
+  - name: metrics
+    protocol: TCP
+    port: 8080
+    targetPort: metrics

docs/faq.md

@@ -22,8 +22,8 @@ However, Osiris and KEDA-HTTP differ in several ways:
 Knative Serving and KEDA-HTTP both have core support for autoscaling, including scale-to-zero of compute workloads. KEDA-HTTP is focused solely on deploying production-grade autoscaling HTTP applications, while Knative builds in additional functionality:
 
 - Pure [event-based workloads](https://knative.dev/docs/eventing/). [KEDA core](https://github.com/kedacore/keda), without KEDA-HTTP, can support such workloads natively.
-- Complex deployment strategies like [blue-green](https://knative.dev/docs/serving/samples/blue-green-deployment/).
-- Supporting other autoscaling mechanisms beyond the built-in [HPA](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/), such as the [Knative Pod Autoscaler (KPA)](https://knative.dev/docs/serving/autoscaling/autoscaling-concepts/#knative-pod-autoscaler-kpa).
+- Complex deployment strategies like [blue-green](https://knative.dev/docs/serving/traffic-management/#routing-and-managing-traffic-with-bluegreen-deployment).
+- Supporting other autoscaling mechanisms beyond the built-in [HPA](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/), such as the [Knative Pod Autoscaler (KPA)](https://knative.dev/docs/serving/autoscaling/autoscaler-types/#knative-pod-autoscaler-kpa).
 
 Additionally, Knative supports a service mesh, while KEDA-HTTP does not out of the box (support for that is forthcoming).
 

docs/install.md

@@ -52,7 +52,7 @@ There are a few values that you can pass to the above `helm install` command by
 helm install http-add-on kedacore/keda-add-ons-http --create-namespace --namespace ${NAMESPACE} --set images.tag=canary
 ```
 
-For an exhaustive list of configuration options, see the official HTTP Add-on chart [values.yaml file](https://github.com/kedacore/charts/blob/master/http-add-on/values.yaml).
+For an exhaustive list of configuration options, see the official HTTP Add-on chart [values.yaml file](https://github.com/kedacore/charts/blob/main/http-add-on/values.yaml).
 
 ### A Note for Developers and Local Cluster Users
 

docs/operate.md

@@ -9,7 +9,7 @@ There are currently 2 supported methods for exposing metrics from the intercepto
 ### Configuring the Prometheus compatible metrics endpoint
 When configured, the interceptor proxy can expose metrics on a Prometheus compatible endpoint.
 
-This endpoint can be enabled by setting the `OTEL_PROM_EXPORTER_ENABLED` environment variable to `true` on the interceptor deployment (`true` by default) and by setting `OTEL_PROM_EXPORTER_PORT` to an unused port for the endpoint to be made avaialble on (`2223` by default).
+This endpoint can be enabled by setting the `OTEL_PROM_EXPORTER_ENABLED` environment variable to `true` on the interceptor deployment (`true` by default) and by setting `OTEL_PROM_EXPORTER_PORT` to an unused port for the endpoint to be made available on (`2223` by default).
 
 ### Configuring the OTEL HTTP exporter
 When configured, the interceptor proxy can export metrics to a OTEL HTTP collector.
@@ -71,3 +71,29 @@ Optional variables
 `OTEL_EXPORTER_OTLP_TRACES_TIMEOUT` - The batcher timeout in seconds to send batch of data points (`5` by default)
 
 ### Configuring Service Failover
+
+# Configuring metrics for the KEDA HTTP Add-on Operator
+
+### Exportable metrics:
+* **keda_http_scaled_object_total** - the number of http_scaled_objects  
+
+There are currently 2 supported methods for exposing metrics from the operator - via a Prometheus compatible metrics endpoint or by pushing metrics to a OTEL HTTP collector.
+
+### Configuring the Prometheus compatible metrics endpoint
+When configured, the operator can expose metrics on a Prometheus compatible endpoint.
+
+This endpoint can be enabled by setting the `OTEL_PROM_EXPORTER_ENABLED` environment variable to `true` on the operator deployment (`true` by default) and by setting `OTEL_PROM_EXPORTER_PORT` to an unused port for the endpoint to be made available on (`8080` by default).
+
+### Configuring the OTEL HTTP exporter
+
+When configured, the operator can export metrics to a OTEL HTTP collector.
+
+The OTEL exporter can be enabled by setting the `OTEL_EXPORTER_OTLP_METRICS_ENABLED` environment variable to `true` on the operator deployment (`false` by default). When enabled, the `OTEL_EXPORTER_OTLP_ENDPOINT` environment variable must also be configured so the exporter knows what collector to send the metrics to (e.g. http://opentelemetry-collector.open-telemetry-system:4318).
+
+If you need to provide any headers such as authentication details in order to utilise your OTEL collector you can add them into the `OTEL_EXPORTER_OTLP_HEADERS` environment variable. The frequency at which the metrics are exported can be configured by setting `OTEL_METRIC_EXPORT_INTERVAL` to the number of seconds you require between each export interval (`30` by default).
+
+The `OTEL_EXPORTER_OTLP_PROTOCOL` defaults to `http`
+
+### Configuring the OTEL GRPC exporter
+
+Please note that using `OTEL_EXPORTER_OTLP_PROTOCOL` will allows you to set it up to `grpc` to connect to otel collector. Also `OTEL_EXPORTER_OTLP_ENDPOINT` should be set to the right endpoint (eg: http://opentelemetry-collector.open-telemetry-system:4317)