OTEL metrics - Figure out the "right stack".

[helio-frota]

No description provided.

[helio-frota]

The "right stack" is a 100% correct statement because of " "

ATM with the merge of #1214,

the "right stack" for both trace and metrics is moving to:

crates/libs:

• actix-web-opentelemetry
• actix-web-prom ( we do have plans to remove this )
• prometheus ( we do have plans to remove this )
• opentelemetry
• opentelemetry-otlp
• opentelemetry_sdk
• tracing
• tracing-opentelemetry
• tracing-subscriber
• tokio

sub-systems/back-ends:

• OpenTelemetry Controller
• Jaeger ( we can double check with specialist folks in relation to grafana tempo later )
• Prometheus

Respective helm-charts for:

• OpenTelemetry Controller
• Jaeger
• Prometheus

[helio-frota]

progress but still not working... probably something with ingress 🤷

https://github.com/helio-frota/m-o?tab=readme-ov-file#eyes-logs

[helio-frota]

progress:

Err: connection error: desc = "error reading server preface: http2: frame too large"
{"grpc_log": true}

[helio-frota]

progress:

communicating the collector with prometheus...although not seeing the application metrics (helm charts related)

prometheus_target_sync_length_seconds{scrape_job="infra-otelcol",quantile="0.01"} 0.000877825
prometheus_target_sync_length_seconds{scrape_job="infra-otelcol",quantile="0.05"} 0.000877825
prometheus_target_sync_length_seconds{scrape_job="infra-otelcol",quantile="0.5"} 0.000877825
prometheus_target_sync_length_seconds{scrape_job="infra-otelcol",quantile="0.9"} 0.00419501
prometheus_target_sync_length_seconds{scrape_job="infra-otelcol",quantile="0.99"} 0.00419501
prometheus_target_sync_length_seconds_sum{scrape_job="infra-otelcol"} 0.0050728349999999995
prometheus_target_sync_length_seconds_count{scrape_job="infra-otelcol"} 2

[helio-frota]

🤔 OTEL_EXPORTER_OTLP_METRICS_ENDPOINT

[helio-frota]

still on it...

"kind": "exporter", "data_type": "metrics", "name": "otlphttp", "error": 
"failed to make an HTTP request: 
Post \"infra-prometheus-server:9464/v1/metrics\": unsupported protocol scheme \"infra-prometheus-server\"", "interval": "26.602110355s"}
``

[helio-frota]

progress...

The app is sending the metrics to collector, but collector can't send to prometheus now
+
Good thing is the usage of the same env var used by traces , I think if we add an extra env var it will be a mess probably...

     -> Name: service_stuff_calls
     -> Description: Total calls of service_stuff function

Metric #0
Descriptor:
     -> Name: service_stuff_calls
     -> Description: Total calls of service_stuff function
     -> Unit:
     -> DataType: Sum
     -> IsMonotonic: true
     -> AggregationTemporality: Cumulative
NumberDataPoints #0
Data point attributes:
     -> service: Str(service_stuff)
     -> status: Str(success)
StartTimestamp: 2025-01-31 12:43:05.518984793 +0000 UTC
Timestamp: 2025-01-31 12:43:15.885930214 +0000 UTC
Value: 7
	{"kind": "exporter", "data_type": "metrics", "name": "debug"}
2025-01-31T12:43:16.044Z	info	internal/retry_sender.go:126	Exporting failed. Will retry the request after interval.	{"kind": "exporter", "data_type": "metrics", "name": "otlphttp", "error": "failed to make an HTTP request: Post \"infra-prometheus-server:9464/v1/metrics\": unsupported protocol scheme \"infra-prometheus-server\"", "interval": "2.721755684s"

[helio-frota]

😃