Monitoring

[Gerrit91]

Monitoring

Control Plane

Basic Stack Deployment

• Create role in metal-roles (control-plane/roles/monitoring) and deploy to monitoring namespace
  • Prometheus, Grafana and Alertmanager
    • Alertmanager
      • Allow configuration of rules and backends
      • Bring some useful default rules
    • Grafana: Configurable Dashboards
      • metal-api Dashboard
      • Rethinkdb Dashboard
    • Deploy Service Monitors
      • metal-api
      • masterdata-api
      • masterdata-db (backup-restore-sidecar) / maybe add postgres-exporter as sidecar?
      • metal-db (+ backup-restore-sidecar)
      • ipam-db (backup-restore-sidecar) / maybe add postgres-exporter as sidecar?
      • gardener-metrics-exporter
  • Thanos
    • Configurable Alerts for the Ruler
• Create role in metal-roles (control-plane/roles/logging) and deploy to monitoring namespace
  • Loki
  • Promtail

Control Plane Metrics Exporters

• metal-metrics-exporter
  • Integrate into metal-control-plane helm-chart (https://github.com/metal-stack/helm-charts/tree/master/charts/metal-control-plane) Added to monitoring role
• rethinkdb-metrics-exporter
  • Integrate into rethinkdb-backup-restore role Added to monitoring role
• gardener-metrics-exporter
  • Add deployment to control-plane/roles/monitoring role in metal-roles

Discussions

• Thanos vs. Mimir
• Should we install a service mesh? (Linkerd)

Partition

• Create role in metal-roles (partition/roles/monitoring) and deploy to management server as systemd service (wrapping a Docker container)
  • Prometheus
    • Scrape exporters
    • Scrape metal-image-cache-sync
  • Exporters
    • node_exporter
    • ipmi_exporter
    • blackbox_exporter
• For the leaf switches
  • Deploy exporters in the leaf role
    • frr_exporter
    • cumulus_exporter
    • node_exporter
    • metal-core
  • Deploy promtail
• For routers
  • Deploy frr_exporter and node_exporter
  • Deploy promtail

Gardener Seeds

• Prometheus deployment
  • Service Monitors
    • lightbox_exporter
    • firewall node_exporter and nftables_exporter

Overview

[Gerrit91]

I guess the first version in place, so I close this issue. However, we want to move everything to a dedicated metal-monitoring repository and maybe also use some different deployment strategies than before.