The Monitoring component is in charge of the collection, process and storage of various signals (i.e. logs, metrics and distributed traces). It is extensively based upon OpenTelemetry, with its Collector.
Caution
This component is an internal work mostly used for development purposes. It is used for production purposes too, i.e. on Capture The Flag events.
Nonetheless, we do not include it in the repositories we are actively maintaining.
The Monitoring service's architecture currently provides:
The multiple parts passes information in a non-cyclic way to avoid deadlocks (DAG or dependencies), as summarized in the following diagram.
For research and/or development purposes, the architecture provide way to perform an extraction of the OpenTelemetry data.
- Activate cold extract:
pulumi config set cold-extract true-
Use the Monitoring through the OTEL Collector endpoint.
-
Run the extractor:
mkdir extract
go run cmd/extractor/main.go \
--namespace $(pulumi stack export namespace) \
--pvc-name $(pulumi stack export otel-cold-extract-pvc-name) \
--directory extract- Add Grafana (require Prometheus)
- Add AlertManager (require Prometheus)
- Think of Dashboard as Code, and if it is relevant to extract them (so technically how ? With Docker manifests inspection ? Service discovery and custom protocol ?) automatically ?