Prometheus cannot connect to /metrics HTTPS endpoint

[d4l-adrian-castro]

Describe the bug
Prometheus cannot connect to /metrics HTTPS endpoint, throws "cannot validate certificate because it doesn't contain any IP SANs"

Expected behavior
Prometheus can connect to /metrics HTTPS endpoint.

Optional: To reproduce

• Have Prometheus running on the same cluster as Connaisseur
• Add the following configuration to Connaisseur:

connaisseur:
  deployment:
    annotations:
      prometheus.io/scrape: "true"
      prometheus.io/path: "/metrics"
      prometheus.io/port: "5000"
      prometheus.io/scheme: "https"
  ...

Optional: Versions (please complete the following information as relevant):

• OS:
• Kubernetes Cluster: v1.23.4
• Notary Server: 0.6.1
• Connaisseur: 2.6.1
• Prometheus: quay.io/prometheus/prometheus:v2.34.0

Optional: Additional context

[d4l-adrian-castro]

Probably related to #225

[xopham]

@d4l-adrian-castro interesting issue. I never tested the full integration with prometheus myself, but considered it would be working. We'll need to test it.
Could also be related or resolvable by #572

[xopham]

would you have any further details on cluster flavor (kind, k3s, aks, eks, gke, ...) or other information that may help reproducing?

[d4l-adrian-castro]

Just k8s, direct upstream

(sorry for the delay, forgot about the notification)

[xopham]

@phbelitz thoughts on this?

[phbelitz]

The more I look into it, the less trivial this issue seems to be ... let me outline you the problem as much as I have grasped it by now:

Currently we generate a certificate with helm, prior to even deploying Connaisseur or its pods. Thus we can't know the IP addresses the pods will have and can't add them to the certificate. Even if we would inject the certificate after creating the pods, injecting the certificate into the pods would restart them and give them yet another IP address 🤷 The auto discovery of prometheus doesn't work here unfortunatey ...

Solutions I see here are:

1. You configure prometheus to explicitly look for the Connaisseur service (not pods).
2. We Seperate the metrics endpoint from the rest and make it availabe via HTTP (+ Basic Auth probably).

For now you probably have to look for option 1, until we decide whether we want to go with option 2 or not (or maybe you want to create a PR for that 😄 ).

Anyways thanks again for raising this issue!

[hsudbrock]

I just ran into the same issue. I do not want to go via the service, because I want to get the metrics for each connaisseur pod separately.

The solution I came up with is to use a PodMonitor (a CRD provided by the prometheus operator), with which I can tell prometheus to ignore TLS verification. Not ideal, but OK for my needs.

To make this work, I had, in addition, to explicitly specify the exposed port 5000 in the connaisseur deployment, because PodMonitors require explicitly specified ports.

My PodMonitor looks as follows (defined as an addition to the connaisseur helm chart):

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  namespace: {{ .Release.Namespace }}
  name: {{ .Chart.Name }}-podmonitor
  labels:
    app.kubernetes.io/name: {{ .Chart.Name }}
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: {{ .Chart.Name }}
  podMetricsEndpoints:
    - path: /metrics
      targetPort: 5000
      scheme: https
      tlsConfig:
        insecureSkipVerify: true
  namespaceSelector:
    any: false
    matchNames:
      - {{ .Release.Namespace }}