[Epic][Feature] KubeRay v1.4.0 - Operator SLI Tracking

[win5923]

Description

https://docs.google.com/document/d/1zNiE7lVZYjhrxlTbh1UXOVpR6hh1GIeSfCfE9Lt5v6Y/edit?tab=t.0

Context

In production, SRE teams typically define Service Level Indicators (SLIs) to ensure that services meet expected performance and reliability standards. However, there are currently no dedicated SLIs for Ray Cluster, Ray Service, and Ray Job, which makes it challenging to monitor their health and performance.

Solution

We propose new metrics to enhance KubeRay's observability and providing better insights into the status and performance of Ray Cluster, Ray Service, and Ray Job.

sub-issues

• [Feature][metrics] ray_cluster_provisioned_duration_seconds #3172
• [Feature][metrics] kuberay_cluster_info #3557
• [SLI Metrics] Add metric kuberay_cluster_condition_provisioned #3635
• [SLI-Metrics] Ray service info #3604
• [Feature][metrics] kuberay_service_ready #3177
• [SLI-Metric] kuberay_service_condition_upgrade_in_progress #3663
• [SLI Metrics] Add metric kuberay_job_info #3621
• [Metric] kuberay_job_deployment_status #3656
• [SLI Metrics] kuberay_job_execution_duration_seconds #3488

[kevin85421]

/assign @troychiu

[troychiu]

I'll be working on this together.

[dushulin]

I use v1.2.2, Is this version adding kuberay metric reporting?And I found no helm chart for metric export, Will version 1.4.0 be added? Thanks

[win5923]

Hi @dushulin,

I use v1.2.2, Is this version adding kuberay metric reporting?

The feature is only supported in version 1.4.0 and above.

And I found no helm chart for metric export, Will version 1.4.0 be added? Thanks

This is mainly focused on the metrics for the Operator, so by default, these metrics will be exposed internally on the kuberay-operator pod at :8080/metrics.

If you're using the Prometheus Operator, you can use PodMonitor or ServiceMonitor to export the metrics to Prometheus.

We are also planning to add a flag that will allow users to enable or disable the metrics feature as needed.

[dushulin]

@win5923 Thanks for the reply, I'm recently supplementing the podMonitor and serviceMonitor of kuberay, and I want to contribute this part of the code if the community needs it.

[kevin85421]

Close this issue because all metrics for 1.4.0 have already been done.