Skip to content

Conversation

@msgSend
Copy link

@msgSend msgSend commented Sep 12, 2025

Fixes #3281

Enabling streaming on Grafana does not work in certain cases (e.g. Istio) with the tempo-distributed Helm chart despite setting stream_over_http_enabled: true in the config. This is because the appProtocol is not defined in the Kubernetes services and so the gRPC payload is munged.

Relevant Istio reference: https://istio.io/latest/docs/ops/configuration/traffic-management/protocol-selection/#explicit-protocol-selection

Copy link
Collaborator

@Sheikh-Abubaker Sheikh-Abubaker left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@msgSend I believe appProtocol is already defined here for both the case could you please checkout if it works the same ?

query-frontend:

{{- if .Values.queryFrontend.appProtocol.grpc }}
appProtocol: {{ .Values.queryFrontend.appProtocol.grpc }}
{{- end }}

metrics-generator:

{{- if and (hasPrefix .name "grpc") ($.Values.metricsGenerator.appProtocol.grpc) }}
appProtocol: {{ $.Values.metricsGenerator.appProtocol.grpc }}
{{- end }}

@Sheikh-Abubaker
Copy link
Collaborator

@msgSend could you please also sign the DCO ?

@msgSend
Copy link
Author

msgSend commented Nov 7, 2025

@msgSend I believe appProtocol is already defined here for both the case could you please checkout if it works the same ?

query-frontend:

{{- if .Values.queryFrontend.appProtocol.grpc }}
appProtocol: {{ .Values.queryFrontend.appProtocol.grpc }}
{{- end }}

metrics-generator:

{{- if and (hasPrefix .name "grpc") ($.Values.metricsGenerator.appProtocol.grpc) }}
appProtocol: {{ $.Values.metricsGenerator.appProtocol.grpc }}
{{- end }}

I could not get this to work correctly. I have them set, but I never see them rendered in the service manifest.

@msgSend msgSend force-pushed the tempo_add_appprotocol branch from 1a3b9c6 to 6e6e5b6 Compare November 7, 2025 08:28
@msgSend msgSend force-pushed the tempo_add_appprotocol branch from 6e6e5b6 to e43409f Compare November 7, 2025 08:29
@msgSend msgSend closed this Nov 7, 2025
@msgSend msgSend force-pushed the tempo_add_appprotocol branch from e43409f to 665f0bf Compare November 7, 2025 08:30
@msgSend msgSend reopened this Nov 7, 2025
Signed-off-by: Erwin <[email protected]>
Signed-off-by: Sheikh-Abubaker <[email protected]>
@msgSend msgSend force-pushed the tempo_add_appprotocol branch from 1a3b9c6 to e100b6b Compare November 7, 2025 08:51
@Sheikh-Abubaker
Copy link
Collaborator

I could not get this to work correctly. I have them set, but I never see them rendered in the service manifest.

@msgSend No worries! here's how you could utilize the existing appProtocol feature:

In values.yaml set appProtocol.grpc to grpc here for query-frontend and do the similar for metrics-generator here, and it would be automatically rendered in the service manifests for both query-frontend and metrics-generator.

IMPORTANT NOTE: Set metricsGenerator.enabled to true here before setting the above for metrics generator.

ipFamilyPolicy: {{ .Values.tempo.service.ipFamilyPolicy }}
ports:
- name: http-metrics
- appProtocol: grpc
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could you please help me understand the need of setting grpc protocol for a http-metrics port ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants