Proposal: Prometheus + KEDA autoscaling for ModelServing

[WHOIM1205]

Proposal: Prometheus + KEDA autoscaling for ModelServing

This PR adds a design proposal for integrating Prometheus + KEDA-based autoscaling into Kthena ModelServing.

The proposal outlines:

• Problem statement and gaps in current autoscaling
• Proposed architecture (Prometheus → KEDA → HPA → ModelServing)
• Key design decisions and trade-offs
• Failure modes and rollout plan
• Learnings from the working prototype

This builds on earlier work where the end-to-end flow has already been validated locally.

---

Related

• Understanding current Kthena autoscaler for HPA/KEDA integration #799 (umbrella issue)
• feat: add KEDA autoscaling example manifests #831 (example manifests)
• Add Prometheus monitoring and KEDA autoscaling support #836 (Helm integration)
• Add KEDA autoscaling support for ModelServing via Prometheus #839 (controller fix for status.labelSelector)

---

Notes

• This proposal introduces KEDA as an optional autoscaling path
• It does not modify or replace the existing AutoscalingPolicy system
• The goal is to align on design before continuing further implementation

[gemini-code-assist]

Code Review

This pull request introduces a proposal and supporting Kubernetes manifests for implementing KEDA and Prometheus-based autoscaling for ModelServing. The changes include new CRD definitions, deployment configurations, and a detailed design document. Feedback on the manifests identifies several critical issues: crd.json contains a leading typo and cluster-specific metadata that should be removed; fixed-crd.yaml is truncated and incomplete; and prom.yaml has invalid YAML syntax in its selector definitions. Additionally, the documentation includes redundant proposal files that should be consolidated.

[gemini-code-assist]

The JSON file starts with an 'i' character, which is likely a typo (e.g., from a vim insert command). This makes the JSON invalid and unparseable.

Suggested change

	i{
	{

[gemini-code-assist]

This file appears to be truncated. It ends abruptly at line 249 with properties:, leaving the ModelServing CRD definition incomplete.

[gemini-code-assist]

This manifest appears to be a raw export from a running cluster. It contains cluster-specific metadata such as uid, resourceVersion, creationTimestamp, and a large last-applied-configuration annotation. These should be removed to ensure the manifest is clean and portable.

[gemini-code-assist]

The status block should not be included in a static CRD definition file stored in the repository. Please remove this section.

[gemini-code-assist]

The podMonitorSelector is incorrectly defined. The empty braces {} on line 52 make the subsequent matchLabels block orphaned or invalid depending on the YAML parser.

  podMonitorSelector:
    matchLabels:
      release: prometheus

[gemini-code-assist]

The serviceMonitorSelector is incorrectly defined. The empty braces {} on line 81 make the subsequent matchLabels block orphaned.

  serviceMonitorSelector:
    matchLabels:
      release: prometheus

[gemini-code-assist]

This file is redundant as it is nearly identical to docs/proposals/keda-autoscaling.md. Please consolidate the proposal into a single file to avoid confusion and maintenance overhead.

[Copilot]

Pull request overview

This PR proposes adding Prometheus + KEDA-based autoscaling support for Kthena ModelServing. The proposal is comprehensive, addressing the limitation of the existing AutoscalingPolicy that cannot directly query Prometheus and lacks per-model demand signals from the router.

Changes:

• Design proposal document for KEDA + Prometheus autoscaling architecture, including problem statement, proposed approach, design decisions, failure modes, and rollout plan
• Example YAML artifacts and CRD definitions (though these appear to be temporary test files that should not be committed)

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
docs/proposals/keda-prometheus-autoscaling.md	Comprehensive design proposal covering architecture, metrics, design decisions, alternatives, failure modes, and rollout plan
docs/proposals/keda-autoscaling.md	Near-duplicate of the above proposal with only date difference
prom.yaml	Prometheus CRD example with YAML syntax errors
crd.json	KEDA ScaledObject CRD definition with JSON syntax error at the start
svc-backup.yaml, deploy-backup.yaml, service.yaml, fixed-crd.yaml	Test/backup artifacts that appear unrelated to this proposal

---

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

This file contains malformed YAML. Lines 52-54 show podMonitorSelector: {} followed by indented matchLabels: which is invalid YAML structure. An empty {} object cannot have properties. Similarly, lines 81-83 show the same issue with serviceMonitorSelector:. The selectors should either be {} (empty) or contain the matchLabels: at the same indentation level without the preceding {}.

[Copilot]

This is a duplicate of docs/proposals/keda-prometheus-autoscaling.md with only a different date (2026-04-02 vs 2026-03-31). Having two identical proposal files in the repository is confusing and should be avoided. Consider removing this file and keeping only the more recent version, or clarifying why both files are needed.

Suggested change

	**Status:** Draft
	**Authors:** @david_laid
	**Date:** 2026-04-02
	### Related PRs
	| PR | Description |
	|----|-------------|
	| [#831](https://github.com/volcano-sh/kthena/pull/831) | Initial prototype -- example manifests (ServiceMonitor, PodMonitor, ScaledObject) |
	| [#836](https://github.com/volcano-sh/kthena/pull/836) | Helm integration -- templates + `values.yaml` for monitoring and autoscaling |
	| [#839](https://github.com/volcano-sh/kthena/pull/839) | Controller fix -- populates `status.labelSelector` so HPA can actually find pods |
	All three reference [#799](https://github.com/volcano-sh/kthena/issues/799). We've validated the full flow end-to-end before writing this up.
	---
	## 1. Problem
	LLM inference traffic is bursty. Without autoscaling, you either overprovision GPUs or eat latency spikes when load surges.
	Kthena already has an autoscaler (`AutoscalingPolicy` + `AutoscalingPolicyBinding`). It scrapes metrics from pod endpoints, has panic mode, supports heterogeneous cost-optimized scaling. Works fine for pod-level signals like `kthena:num_requests_waiting`.
	Where it falls short:
	1. **Can't talk to Prometheus.** It scrapes pods directly. Most teams already have Prometheus running -- the autoscaler can't use it.
	2. **No per-model demand signal.** The router exposes `kthena_router_active_downstream_requests{model="..."}` which tells you how much traffic a model is getting *before* it hits backends. The built-in autoscaler doesn't use this.
	3. **Extra moving parts.** Teams already running KEDA end up maintaining two autoscaling systems side by side.
	The goal here is to add KEDA as an optional autoscaling path. We're not touching AutoscalingPolicy.
	### Non-goals
	- Modifying or replacing `AutoscalingPolicy` / `AutoscalingPolicyBinding`
	- Building a custom metrics adapter
	- Multi-model-per-ModelServing (we assume 1:1)
	- Role-level scaling via KEDA (built-in autoscaler handles that with `subTargets.kind: Role`)
	- Auto-generating ScaledObjects (Phase 3 at earliest)
	---
	## 2. Proposed Approach
	**Status:** Superseded
	**Authors:** @david_laid
	**Date:** 2026-04-02
	This file previously duplicated the full proposal now maintained at
	[`docs/proposals/keda-prometheus-autoscaling.md`](docs/proposals/keda-prometheus-autoscaling.md).
	To avoid keeping two identical proposal documents in the repository, that file is the
	canonical version and should be used for all future updates and references.
	This file is retained only as a compatibility pointer for any existing links to
	`docs/proposals/keda-autoscaling.md`.

[Copilot]

The JSON file starts with invalid syntax: i{ on line 1 instead of just {. This makes the entire JSON file invalid and unparseable. This appears to be a typo or corruption.

Suggested change

	i{
	{

[hzxuzhonghu]

Please reemove all the yamls in root dir.

[hzxuzhonghu]

Thanks for the point, i am wondering we can support it later.

[hzxuzhonghu]

It does not support using kthena router metrics

[hzxuzhonghu]

Prefer leaving it as second step work

[hzxuzhonghu]

can you replace with mermaid, which is more readable

[hzxuzhonghu]

kthena_router_active_downstream_requests is just an example metric type, not a must

[hzxuzhonghu]

Keep in mind the model name from the router matric can be different from the real model name running in vlllm or sglang.

Because we can do model name match in router and route to the real backend model instance

[hzxuzhonghu]

Can use a placeholder here

[hzxuzhonghu]

This seems too short as a llm can takes 5minutes to start from ground

[hzxuzhonghu]

isnot it same as keda-autoscaling.md?

[hzxuzhonghu]

what's this, why put it in proposal pr

[volcano-sh-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign lizhencheng9527 for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• docs/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[WHOIM1205]

hey @hzxuzhonghu
thanks a lot for the detailed review this was really helpful
i’ve addressed all the points:

1. cleaned up root-level files
2. reduced scope and separated future work
3. replaced diagrams with mermaid
4. made metrics generic
5. clarified model-to-ModelServing mapping
6. improved examples and placeholders
   Would really appreciate another look whenever you get time