Skip to content

Conversation

@samzong
Copy link
Contributor

@samzong samzong commented Nov 20, 2025

FIX #658 (link existing issues this PR will resolve)

LLM-D profile e2e

Summary

Add an LLM‑D e2e profile to validate LLM-based decision routing in a standard K8s/Kind environment, with focus on correctness, resilience, and basic observability.

What's changed

  • Added LLM-D profile under e2e/profiles/llm-d/ (install, config, health checks, cleanup), selectable via E2E_PROFILE=llm-d.
  • Registered the LLM-D profile in e2e/cmd/e2e/main.go so it is managed through the unified e2e entrypoint.
  • Hooked the existing core e2e suites (routing, failover, multi‑replica serving, health/readiness, basic performance) to also run against the llm-d deployment.
  • Updated .github/workflows/integration-test-k8s.yml to include E2E_PROFILE=llm-d in the k8s integration test matrix.
  • Updated tools/make/e2e.mk and e2e/README.md with LLM-D usage, examples, and basic troubleshooting.
  • Added a short stabilization window and explicit health checks to reduce flakiness on Kind/CI clusters.

How to run

  • Setup only: make e2e-setup E2E_PROFILE=llm-d
  • Run tests: make e2e-test E2E_PROFILE=llm-d
  • Optional verbose logs: make e2e-test E2E_PROFILE=llm-d E2E_VERBOSE=1

@netlify
Copy link

netlify bot commented Nov 20, 2025

Deploy Preview for vllm-semantic-router ready!

Name Link
🔨 Latest commit 189754b
🔍 Latest deploy log https://app.netlify.com/projects/vllm-semantic-router/deploys/6926d7c3af82ff00087167d7
😎 Deploy Preview https://deploy-preview-705--vllm-semantic-router.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

@github-actions
Copy link

github-actions bot commented Nov 20, 2025

👥 vLLM Semantic Team Notification

The following members have been identified for the changed files in this PR and have been automatically assigned:

📁 e2e

Owners: @Xunzhuo
Files changed:

  • e2e/profiles/llm-d/manifests/httproute-services.yaml
  • e2e/profiles/llm-d/manifests/inference-sim.yaml
  • e2e/profiles/llm-d/manifests/rbac.yaml
  • e2e/profiles/llm-d/profile.go
  • e2e/profiles/llm-d/values.yaml
  • e2e/README.md
  • e2e/cmd/e2e/main.go

📁 Root Directory

Owners: @rootfs, @Xunzhuo
Files changed:

  • .github/workflows/integration-test-k8s.yml

📁 tools

Owners: @yuluo-yx, @rootfs, @Xunzhuo
Files changed:

  • tools/make/e2e.mk

vLLM

🎉 Thanks for your contributions!

This comment was automatically generated based on the OWNER files in the repository.

@samzong samzong changed the title ✨ feat(e2e/llm-d): add LLM-D profile and test cases [DO-NOT-MERGED] ✨ feat(e2e/llm-d): add LLM-D profile and test cases Nov 20, 2025
@samzong samzong force-pushed the e2e-llmd branch 4 times, most recently from 5f82dc9 to f3998d5 Compare November 21, 2025 18:11
@samzong samzong changed the title [DO-NOT-MERGED] ✨ feat(e2e/llm-d): add LLM-D profile and test cases ✨ feat(e2e/llm-d): add LLM-D profile and test cases Nov 21, 2025
@samzong samzong marked this pull request as ready for review November 21, 2025 18:16
@samzong
Copy link
Contributor Author

samzong commented Nov 21, 2025

@Xunzhuo It's ready for review.

@Xunzhuo
Copy link
Member

Xunzhuo commented Nov 22, 2025

Cool, a quick question, any test cases for features combine VSR features and llm-d ?

@samzong
Copy link
Contributor Author

samzong commented Nov 22, 2025

@Xunzhuo Yes

  • VSR => Auto routing: e2e/testcases/llmd_auto_routing.go
  • LLM-d => Distributed inference: e2e/testcases/llmd_distributed_inference.go

@samzong
Copy link
Contributor Author

samzong commented Nov 22, 2025

/gemini summary

@samzong
Copy link
Contributor Author

samzong commented Nov 25, 2025

@Xunzhuo @rootfs

Hi, I've add details with "What's Changed" and "TestCases".

Could you please review it and let me know if there are any other issues?

@Xunzhuo
Copy link
Member

Xunzhuo commented Nov 25, 2025

My question is looking at the testcases they are around the functionality of llm-d, that is good but the integration test should add more cases around vsr + llm-d like verifying model auto selection + llm-d, ideally should reuse the shared test cases

@samzong
Copy link
Contributor Author

samzong commented Nov 26, 2025

@Xunzhuo Thanks for the review.

I will add VSR + LLM-D integration tests, reuse existing VSR auto-routing cases, verify selected-model consistency with the LLM-D backend, and include the failover path.

@samzong samzong marked this pull request as draft November 26, 2025 01:34
@samzong
Copy link
Contributor Author

samzong commented Nov 26, 2025

update: all use shared testcase.

================================================================================
TEST RESULTS
================================================================================
✅ PASSED - chat-completions-request (2.143574634s)
✅ PASSED - chat-completions-stress-request (20.746133879s)
✅ PASSED - chat-completions-progressive-stress (22.525980752s)
✅ PASSED - domain-classify (1m15.987869552s)
================================================================================
Total: 4 | Passed: 4 | Failed: 0
================================================================================
[Runner] 📝 Collecting semantic-router logs...
[Runner] ✅ Semantic router logs saved to: semantic-router-logs.txt
[Runner] ✅ All tests passed!

@samzong samzong marked this pull request as ready for review November 26, 2025 08:05
@samzong
Copy link
Contributor Author

samzong commented Nov 26, 2025

@Xunzhuo Now it has been processed, llmd_* testcases have been removed. all cases have been reused, and has test passed.

@samzong samzong changed the title ✨ feat(e2e/llm-d): add LLM-D profile and test cases ✨ feat(e2e/llm-d): add LLM-D profile for E2E testing framework Nov 26, 2025
@samzong samzong changed the title ✨ feat(e2e/llm-d): add LLM-D profile for E2E testing framework feat: add LLM-D profile for E2E testing framework Nov 26, 2025
@Xunzhuo
Copy link
Member

Xunzhuo commented Nov 26, 2025

very cool. Thanks

@Xunzhuo Xunzhuo merged commit 37e5941 into vllm-project:main Nov 26, 2025
22 checks passed
samzong added a commit to samzong/semantic-router that referenced this pull request Nov 27, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[E2E] Add llm-d profile for E2E testing framework

3 participants