Prometheus output example #121
Description
Attached is an example of the Prometheus metrics output from the S2S proxy. You can scrape this information with your preferred monitoring tool by observing [proxy-ip]:9090/metrics (you may set the metrics port in the proxy-config). Some key metrics to observe while your proxy is running:
- temporal_s2s_proxy_mux_connection_active: This tells you how many mux connections your proxy has made. By default, the s2s proxy will attempt to maintain 10 open connections.
- temporal_s2s_proxy_grpc_server_*: Standard GRPC server metrics are available in the "inbound" and "outbound" directions. The "inbound" server is the GRPC server receiving from your remote and transferring to the local Temporal cluster. The "outbound" server is the GRPC server receiving from your local Temporal cluster and transferring to the remote.
- temporal_s2s_proxy_outbound_grpc_client_handled_total / temporal_s2s_proxy_inbound_grpc_client_handled_total: These are the standard GRPC client metrics, and can be paired with the above server metrics to diagnose whether a problem happened inside or outside your proxy deployment.
- temporal_s2s_proxy_admin_service_streams_active: The number of replication streams active through the proxy. The sum of all active streams should equal your cluster's shard count.