feature: Kthena runtime adaptation for sglang by katara-Jayprakash · Pull Request #812 · volcano-sh/kthena

katara-Jayprakash · 2026-03-13T20:19:36Z

What type of PR is this?

This is a feature PR that adds the getPodModels functionality for SGLang inference engines, similar to the existing implementation for vLLM engines.It also adds a new metrics field, RequestRunningNum, to the SGLang engine metrics.

What are the learning i got from this issue:

Thanks to this issue, I had the opportunity to explore the Kthena runtime, sidecar, router, and autoscaler codebases more deeply. I learned how Kthena collects metrics from inference engines and how those metrics are used differently for autoscaling and intelligent routing.
Special thanks to @YaoZengzeng and @LiZhenCheng9527 for discussing the problem with me, answering my questions, and helping me understand the system better.

Which issue(s) this PR fixes:
Fixes #788

Special notes for your reviewer:

Does this PR introduce a user-facing change?: No

NONE

…ccroding to its type Signed-off-by: katara-Jayprakash <[email protected]>

Signed-off-by: katara-Jayprakash <[email protected]>

volcano-sh-bot · 2026-03-13T20:19:41Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign hzxuzhonghu for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

pkg/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

gemini-code-assist · 2026-03-13T20:19:53Z

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request enhances the Kthena runtime adaptation for SGLang by adding crucial model discovery capabilities and extending its observability with a new metric. These changes allow the system to dynamically identify models served by SGLang instances and provide more comprehensive insights into the engine's operational state, improving resource management and monitoring.

Highlights

SGLang Model Discovery: Implemented getPodModels functionality for SGLang inference engines, enabling the discovery of available models, similar to the existing vLLM engine support.
New SGLang Metric: Introduced a new metric, RequestRunningNum, to track the number of currently running requests in the SGLang engine.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog

pkg/kthena-router/backend/sglang/metrics.go
- Added RequestRunningNum metric constant and included it in metric mappings.
- Introduced Model and ModelList structs for JSON unmarshalling of model data.
- Updated metric collection logic to correctly handle both gauge and counter metric types.
- Implemented the GetPodModels function to retrieve model IDs from SGLang pods via HTTP.

Activity

No human activity has been recorded for this pull request yet.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request adds support for sglang inference engines by implementing getPodModels and adding the RequestRunningNum metric. The changes are generally in the right direction. However, the new implementation of GetPodModels and its associated data structures introduces significant code duplication with the existing vllm engine implementation. I've provided suggestions to refactor this duplicated code into shared components to improve maintainability. I also pointed out a potential issue with an HTTP request not having a timeout, which could lead to hanging processes. Finally, a minor comment on code documentation was added.

gemini-code-assist · 2026-03-13T20:21:32Z

pkg/kthena-router/backend/sglang/metrics.go

 func (engine *sglangEngine) GetPodModels(pod *corev1.Pod) ([]string, error) {
-	return nil, nil
+	url := fmt.Sprintf("http://%s:%d/v1/models", pod.Status.PodIP, engine.MetricPort)
+	resp, err := http.Get(url)


The call to http.Get does not have a timeout. This could cause the request to hang indefinitely if the server is unresponsive. It's recommended to use an http.Client with a configured timeout to prevent this. Note that this change will require importing the time package.

Suggested change

resp, err := http.Get(url)

resp, err := (&http.Client{Timeout: 5 * time.Second}).Get(url)

gemini-code-assist · 2026-03-13T20:21:32Z

pkg/kthena-router/backend/sglang/metrics.go

+type Model struct {
+	ID string `json:"id"`
+}
+
+type ModelList struct {
+	Data []Model `json:"data"`
+}


These Model and ModelList structs are duplicates of the ones defined in pkg/kthena-router/backend/vllm/models.go. To improve maintainability and avoid code duplication, these structs should be moved to a shared package, for example, a new pkg/kthena-router/backend/types.go file, and reused in both vllm and sglang packages.

gemini-code-assist · 2026-03-13T20:21:32Z

pkg/kthena-router/backend/sglang/metrics.go

 }

-// TODO： Methods to get Models from sglang
+// implemented the get models function for sglang engine, 


This comment is informal and grammatically incorrect. Please replace it with a standard Go doc comment explaining the function's purpose.

Suggested change

// implemented the get models function for sglang engine,

// GetPodModels retrieves the list of models from a pod running the sglang engine.

gemini-code-assist · 2026-03-13T20:21:32Z

pkg/kthena-router/backend/sglang/metrics.go

 func (engine *sglangEngine) GetPodModels(pod *corev1.Pod) ([]string, error) {
-	return nil, nil
+	url := fmt.Sprintf("http://%s:%d/v1/models", pod.Status.PodIP, engine.MetricPort)
+	resp, err := http.Get(url)
+	if err != nil {
+		return nil, err
+	}
+	defer resp.Body.Close()
+
+	if resp.StatusCode != http.StatusOK {
+		return nil, fmt.Errorf("failed to get models from pod %s/%s: HTTP %d", pod.GetNamespace(), pod.GetName(), resp.StatusCode)
+	}
+
+	body, err := io.ReadAll(resp.Body)
+	if err != nil {
+		return nil, err
+	}
+
+	var modelList ModelList
+	err = json.Unmarshal(body, &modelList)
+	if err != nil {
+		return nil, err
+	}
+
+	models := make([]string, 0, len(modelList.Data))
+	for _, model := range modelList.Data {
+		models = append(models, model.ID)
+	}
+	return models, nil
 }


This function GetPodModels is identical to the implementation for vllm in pkg/kthena-router/backend/vllm/models.go. To avoid code duplication, this logic should be extracted into a shared helper function. This function could be placed in a common backend package and called by both sglang and vllm implementations. Additionally, for consistency with the vllm backend, consider moving model-related logic into a new pkg/kthena-router/backend/sglang/models.go file.

Copilot

Pull request overview

Adds SGLang parity with existing vLLM runtime integration by exposing pod model discovery and extending SGLang engine metrics.

Changes:

Add RequestRunningNum to SGLang metrics and include it in metric name mapping.
Update SGLang count-metric extraction to support both Prometheus GAUGE and COUNTER metric families.
Implement GetPodModels for SGLang by calling the pod’s /v1/models endpoint and parsing the response.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

pkg/kthena-router/backend/sglang/metrics.go

+	url := fmt.Sprintf("http://%s:%d/v1/models", pod.Status.PodIP, engine.MetricPort)
+	resp, err := http.Get(url)
+	if err != nil {
+		return nil, err
+	}


katara-Jayprakash · 2026-03-13T20:33:06Z

pkg/kthena-router/backend/sglang/metrics.go

 }

-// TODO： Methods to get Models from sglang
+// implemented the get models function for sglang engine, 


fixed in next commit,

Signed-off-by: katara-Jayprakash <[email protected]>

katara-Jayprakash · 2026-03-13T20:32:02Z

pkg/kthena-router/backend/sglang/metrics.go

 		for _, metric := range metricInfo.Metric {
-			metricValue := metric.GetCounter().GetValue()
+			var metricValue float64
+			switch metricInfo.GetType() {


the reason for adding this switch statement is i'm not actually sure what kind of metricsType i am going to get like gauge or counter from sglang engine, so i ended up using switch case for both condtional's but would love to what exactly kind of data we are going to get? @YaoZengzeng @LiZhenCheng9527!

hzxuzhonghu

One thing I would like to clarify before this merges: what happens when /v1/models is temporarily unavailable on a pod?

The new GetPodModels implementation returns an error on any non-200, and in the router store updatePodModels still calls podInfo.UpdateModels(models) even when GetPodModels failed. Since models is then nil/empty, that effectively clears the pod's model set.

If every SGLang pod that the router watches always exposes /v1/models, then this is fine. But if there are SGLang topologies where some pods only expose metrics/health and not model listing, a transient failure here would silently drop model associations for that pod. It may be worth either preserving the previous model set on error or adding a test that proves this endpoint is guaranteed for the pods we register.

katara-Jayprakash · 2026-03-16T15:36:59Z

One thing I would like to clarify before this merges: what happens when /v1/models is temporarily unavailable on a pod?

The new GetPodModels implementation returns an error on any non-200, and in the router store updatePodModels still calls podInfo.UpdateModels(models) even when GetPodModels failed. Since models is then nil/empty, that effectively clears the pod's model set.

If every SGLang pod that the router watches always exposes /v1/models, then this is fine. But if there are SGLang topologies where some pods only expose metrics/health and not model listing, a transient failure here would silently drop model associations for that pod. It may be worth either preserving the previous model set on error or adding a test that proves this endpoint is guaranteed for the pods we register.

Makes sense. I think we can’t test these PRs due to resource constraints. I’ll still try to add helpful reviews and suggestions.

LiZhenCheng9527 · 2026-04-07T06:57:45Z

@katara-Jayprakash Is there anything else that needs to be revised in this PR?
If this PR is no longer needed, can I close this PR?

hzxuzhonghu

Sorry, i forgot to submit my previous review

hzxuzhonghu · 2026-03-17T09:37:53Z

pkg/kthena-router/backend/sglang/metrics.go

+// GetPodModels retrieves the list of models from a pod running the sglang engine.
 func (engine *sglangEngine) GetPodModels(pod *corev1.Pod) ([]string, error) {
-	return nil, nil
+	url := fmt.Sprintf("http://%s:%d/v1/models", pod.Status.PodIP, engine.MetricPort)


can we abstract a common helper function from vllm backend,

hzxuzhonghu · 2026-03-17T09:38:25Z

pkg/kthena-router/backend/sglang/metrics.go

 func (engine *sglangEngine) GetPodModels(pod *corev1.Pod) ([]string, error) {
-	return nil, nil
+	url := fmt.Sprintf("http://%s:%d/v1/models", pod.Status.PodIP, engine.MetricPort)
+	resp, err := http.Get(url)


hzxuzhonghu · 2026-03-17T09:47:23Z

pkg/kthena-router/backend/sglang/metrics.go

 	GPUCacheUsage     = "sglang:token_usage"
 	RequestWaitingNum = "sglang:num_queue_reqs"
+	RequestRunningNum = "sglang:num_running_reqs"
 	TPOT              = "sglang:time_per_output_token_seconds"


this metric has been replaced by sglang:inter_token_latency_seconds https://github.com/sgl-project/sglang/blob/385a35bd118d2e420be1fd6978c703d22a3bcc77/python/sglang/srt/observability/metrics_collector.py#L188

katara-Jayprakash · 2026-04-08T05:16:15Z

Sorry, i forgot to submit my previous review

@hzxuzhonghu @LiZhenCheng9527 i think we can work on this thing, i am currently working on e2e test, but have this also in my bucket list sorry for delay responses! i'm gonna make sure to finish it asap

katara-Jayprakash added 2 commits March 14, 2026 01:22

implemented RequestRunningNum matrices field and getting its values a…

6aa9979

…ccroding to its type Signed-off-by: katara-Jayprakash <[email protected]>

implemented the GetPodModels sglang feature in kthena

b4121d8

Signed-off-by: katara-Jayprakash <[email protected]>

Copilot AI review requested due to automatic review settings March 13, 2026 20:19

volcano-sh-bot requested review from LiZhenCheng9527 and YaoZengzeng March 13, 2026 20:19

volcano-sh-bot added the size/M label Mar 13, 2026

Copilot started reviewing on behalf of katara-Jayprakash March 13, 2026 20:20 View session

katara-Jayprakash changed the title ~~feat: Kthena runtime adaptation for sglang~~ feature: Kthena runtime adaptation for sglang Mar 13, 2026

gemini-code-assist bot reviewed Mar 13, 2026

View reviewed changes

Copilot AI reviewed Mar 13, 2026

View reviewed changes

fix the grammer in sglang getpodmodels

a3c8b3d

Signed-off-by: katara-Jayprakash <[email protected]>

katara-Jayprakash commented Mar 13, 2026

View reviewed changes

hzxuzhonghu reviewed Mar 16, 2026

View reviewed changes

hzxuzhonghu reviewed Apr 8, 2026

View reviewed changes

	resp, err := http.Get(url)
	resp, err := (&http.Client{Timeout: 5 * time.Second}).Get(url)

	// implemented the get models function for sglang engine,
	// GetPodModels retrieves the list of models from a pod running the sglang engine.

Conversation

katara-Jayprakash commented Mar 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

volcano-sh-bot commented Mar 13, 2026

Uh oh!

gemini-code-assist bot commented Mar 13, 2026

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist bot Mar 13, 2026

Choose a reason for hiding this comment

Uh oh!

hzxuzhonghu Mar 17, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Mar 13, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Mar 13, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Mar 13, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

katara-Jayprakash Mar 13, 2026

Choose a reason for hiding this comment

Uh oh!

katara-Jayprakash Mar 13, 2026

Choose a reason for hiding this comment

Uh oh!

hzxuzhonghu left a comment

Choose a reason for hiding this comment

Uh oh!

katara-Jayprakash commented Mar 16, 2026

Uh oh!

LiZhenCheng9527 commented Apr 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hzxuzhonghu left a comment

Choose a reason for hiding this comment

Uh oh!

hzxuzhonghu Mar 17, 2026

Choose a reason for hiding this comment

Uh oh!

hzxuzhonghu Mar 17, 2026

Choose a reason for hiding this comment

Uh oh!

hzxuzhonghu Mar 17, 2026

Choose a reason for hiding this comment

Uh oh!

katara-Jayprakash commented Apr 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

katara-Jayprakash commented Mar 13, 2026 •

edited

Loading

LiZhenCheng9527 commented Apr 7, 2026 •

edited

Loading

katara-Jayprakash commented Apr 8, 2026 •

edited

Loading