feat: reasoning model controller #56

tao12345666333 · 2025-09-04T22:24:26Z

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Fixes #33

Release Notes: Yes/No

github-actions · 2025-09-04T22:24:36Z

👥 vLLM Semantic Team Notification

The following members have been identified for the changed files in this PR and have been automatically assigned:

📁 `src`

Owners: @rootfs, @Xunzhuo, @wangchen615
Files changed:

src/semantic-router/pkg/extproc/reason_mode_selector.go
src/semantic-router/pkg/extproc/request_handler.go
src/semantic-router/pkg/metrics/metrics.go

This comment was automatically generated based on the OWNER files in the repository.

netlify · 2025-09-04T22:27:47Z

✅ Deploy Preview for vllm-semantic-router ready!

Name	Link
🔨 Latest commit	`467f36a`
🔍 Latest deploy log	https://app.netlify.com/projects/vllm-semantic-router/deploys/68bafc77c782d60008b9810d
😎 Deploy Preview	https://deploy-preview-56--vllm-semantic-router.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

src/semantic-router/pkg/metrics/metrics.go

rootfs · 2025-09-04T22:33:08Z

@tao12345666333 thanks for starting up this! I am also working on the benchmark script to make it easier to use. So when you are testing the reasoning controller, please verify it with a reasoning model.

We are also going to add an e2e CI to test this out.

tao12345666333 · 2025-09-04T22:35:28Z

@rootfs Glad to hear that, I'll verify it this way. Thank you!

Xunzhuo · 2025-09-05T01:03:51Z

Cool, welcome @tao12345666333 :)

rootfs · 2025-09-05T14:35:30Z

@tao12345666333 thanks for the prompt action! Would you please rebase?

rootfs · 2025-09-05T14:36:16Z

src/semantic-router/pkg/extproc/reason_mode_selector.go

 }

+// getModelFamilyAndTemplateParam returns a normalized model family name and the template param to be used (if any)
+func getModelFamilyAndTemplateParam(model string) (string, string) {


can you add a unit test for this? you can do it in a follow up PR.

I think we should add an UT in this PR, let us make sure it does not break the strategy for modifying the body.

rootfs · 2025-09-05T14:36:46Z

src/semantic-router/pkg/metrics/metrics.go

 	)
+
+	// ReasoningDecisions tracks the reasoning mode decision outcome by category, model, and effort
+	ReasoningDecisions = promauto.NewCounterVec(


please come up with a follow up PR to add them to the doc

Added an issue for tracking #62

rootfs · 2025-09-05T14:39:12Z

src/semantic-router/pkg/extproc/reason_mode_selector.go

+	if strings.Contains(lower, "qwen3") {
+		return "qwen3", "enable_thinking"
+	}
+	if strings.Contains(lower, "deepseek") || strings.Contains(lower, "ds") {


ds is too broad to me. Maybe we should also add a config validation to detect if there are conflict or duplicate model names with ds in them but not actually deepseek..

would you add an issue and follow up with this in the next PR for more robust model name filter?

I write "ds" here because many people (including on social media) refer to DeepSeek as "ds" and Claude Code as "cc." 🤣

I created an issue for tracking #61

my setup modelname for deepseek-v3 is also ds-v3 😄

Signed-off-by: Jintao Zhang <[email protected]>

rootfs · 2025-09-05T15:13:02Z

@tao12345666333 for the reasoning e2e test on CI github action, do you like to give it a try? You can start by creating the script to setup the env and verification process and we'll use your work in github action. Thanks

Xunzhuo

plz add UT in a follow-up

Xunzhuo · 2025-09-05T15:15:35Z

src/semantic-router/pkg/extproc/reason_mode_selector.go

 }

+// getModelFamilyAndTemplateParam returns a normalized model family name and the template param to be used (if any)
+func getModelFamilyAndTemplateParam(model string) (string, string) {


I think we should add an UT in this PR, let us make sure it does not break the strategy for modifying the body.

Xunzhuo · 2025-09-05T15:16:24Z

src/semantic-router/pkg/extproc/reason_mode_selector.go

+	if strings.Contains(lower, "qwen3") {
+		return "qwen3", "enable_thinking"
+	}
+	if strings.Contains(lower, "deepseek") || strings.Contains(lower, "ds") {


my setup modelname for deepseek-v3 is also ds-v3 😄

tao12345666333 · 2025-09-05T15:17:13Z

@tao12345666333 for the reasoning e2e test on CI github action, do you like to give it a try? You can start by creating the script to setup the env and verification process and we'll use your work in github action. Thanks

@rootfs Sure. Please assign an issue to me. I'm happy to give it a try

tao12345666333 · 2025-09-05T15:18:48Z

plz add UT in a follow-up

Sure. #56 (comment)

I will submit the follow-up PR soon.

rootfs · 2025-09-05T15:36:23Z

plz add UT in a follow-up

Sure. #56 (comment)

I will submit the follow-up PR soon.

Great! please create an issue and we'll assign it to you

tao12345666333 · 2025-09-05T16:03:53Z

@rootfs I have submitted a PR #63

github-actions bot assigned rootfs, wangchen615 and Xunzhuo Sep 4, 2025

rootfs reviewed Sep 4, 2025

View reviewed changes

src/semantic-router/pkg/metrics/metrics.go Outdated Show resolved Hide resolved

tao12345666333 marked this pull request as ready for review September 5, 2025 11:31

tao12345666333 requested review from Xunzhuo and wangchen615 as code owners September 5, 2025 11:31

rootfs reviewed Sep 5, 2025

View reviewed changes

feat: reasoning model controller

dca5149

Signed-off-by: Jintao Zhang <[email protected]>

tao12345666333 force-pushed the feat-reasoning-mode-ctl branch from 7496067 to 5f6fd67 Compare September 5, 2025 14:56

tao12345666333 mentioned this pull request Sep 5, 2025

add more robust model name filter #61

Closed

add model name for metrics

467f36a

Signed-off-by: Jintao Zhang <[email protected]>

tao12345666333 force-pushed the feat-reasoning-mode-ctl branch from 5f6fd67 to 467f36a Compare September 5, 2025 15:06

tao12345666333 mentioned this pull request Sep 5, 2025

docs: add description about Reasoning mode controller #62

Closed

rootfs approved these changes Sep 5, 2025

View reviewed changes

rootfs merged commit ddc86e3 into vllm-project:main Sep 5, 2025
9 checks passed

Xunzhuo reviewed Sep 5, 2025

View reviewed changes

tao12345666333 deleted the feat-reasoning-mode-ctl branch September 5, 2025 15:18

tao12345666333 mentioned this pull request Sep 5, 2025

test: add unit tests for getModelFamilyAndTemplateParam #63

Merged

JaredforReal mentioned this pull request Sep 15, 2025

docs: Model Performance Evaluation Guide #136

Merged

feat: reasoning model controller #56

feat: reasoning model controller #56

Uh oh!

Conversation

tao12345666333 commented Sep 4, 2025

Uh oh!

github-actions bot commented Sep 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

👥 vLLM Semantic Team Notification

📁 src

Uh oh!

netlify bot commented Sep 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for vllm-semantic-router ready!

Uh oh!

Uh oh!

rootfs commented Sep 4, 2025

Uh oh!

tao12345666333 commented Sep 4, 2025

Uh oh!

Xunzhuo commented Sep 5, 2025

Uh oh!

rootfs commented Sep 5, 2025

Uh oh!

rootfs Sep 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rootfs Sep 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rootfs commented Sep 5, 2025

Uh oh!

Uh oh!

Xunzhuo left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tao12345666333 commented Sep 5, 2025

Uh oh!

tao12345666333 commented Sep 5, 2025

Uh oh!

rootfs commented Sep 5, 2025

Uh oh!

tao12345666333 commented Sep 5, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

github-actions bot commented Sep 4, 2025 •

edited

Loading

📁 `src`

netlify bot commented Sep 4, 2025 •

edited

Loading

rootfs Sep 5, 2025 •

edited

Loading

rootfs Sep 5, 2025 •

edited

Loading