Skip to content

Conversation

@rootfs
Copy link
Collaborator

@rootfs rootfs commented Oct 1, 2025

What type of PR is this?

Add an endpoint to set system prompt injection to allow runtime management. This endpoint is only turned on with explicit command line option

What this PR does / why we need it:

Which issue(s) this PR fixes:

Fixes #

Release Notes: Yes/No

@netlify
Copy link

netlify bot commented Oct 1, 2025

Deploy Preview for vllm-semantic-router ready!

Name Link
🔨 Latest commit 063a280
🔍 Latest deploy log https://app.netlify.com/projects/vllm-semantic-router/deploys/68de8c86e9de9a0008830f1a
😎 Deploy Preview https://deploy-preview-301--vllm-semantic-router.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

@rootfs rootfs requested a review from Copilot October 1, 2025 14:37
@rootfs rootfs marked this pull request as draft October 1, 2025 14:37
@github-actions
Copy link

github-actions bot commented Oct 1, 2025

👥 vLLM Semantic Team Notification

The following members have been identified for the changed files in this PR and have been automatically assigned:

📁 src

Owners: @rootfs, @Xunzhuo, @wangchen615
Files changed:

  • src/semantic-router/cmd/main.go
  • src/semantic-router/pkg/api/server.go
  • src/semantic-router/pkg/api/server_test.go
  • src/semantic-router/pkg/config/config.go
  • src/semantic-router/pkg/extproc/request_handler.go
  • src/semantic-router/pkg/services/classification.go

📁 tools

Owners: @yuluo-yx, @rootfs, @Xunzhuo
Files changed:

  • tools/make/build-run-test.mk

📁 website

Owners: @Xunzhuo
Files changed:

  • website/docs/overview/categories/configuration.md

vLLM

🎉 Thanks for your contributions!

This comment was automatically generated based on the OWNER files in the repository.

Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

Adds runtime management of category-specific system prompts, including enable/disable and injection mode control, exposed via optional API endpoints.

  • Introduces system prompt configuration endpoints guarded by a CLI flag.
  • Adds per-category enable flag and injection mode ("replace" vs "insert") and modifies injection logic accordingly.
  • Adds tests for endpoint security and functionality plus CLI integration.

Reviewed Changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
src/semantic-router/pkg/services/classification.go Adds getters/setters to expose and update router config at runtime.
src/semantic-router/pkg/extproc/request_handler.go Adds injection mode support and switches to using live global config; adds extensive logging.
src/semantic-router/pkg/config/config.go Adds category-level system prompt enable flag, mode handling, and lookup helpers.
src/semantic-router/pkg/api/server.go Adds flag-controlled system prompt management endpoints and handlers.
src/semantic-router/pkg/api/server_test.go Introduces tests for security gating and functional updates of system prompt settings.
src/semantic-router/cmd/main.go Adds CLI flag and passes it into server startup.

Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

rootfs added 6 commits October 1, 2025 15:18
Signed-off-by: Huamin Chen <[email protected]>
Signed-off-by: Huamin Chen <[email protected]>
Signed-off-by: Huamin Chen <[email protected]>
Signed-off-by: Huamin Chen <[email protected]>
Signed-off-by: Huamin Chen <[email protected]>
@rootfs rootfs marked this pull request as ready for review October 2, 2025 14:30
@rootfs
Copy link
Collaborator Author

rootfs commented Oct 2, 2025

merging it for now, need it for next PR.

@rootfs rootfs merged commit 077b8d0 into vllm-project:main Oct 2, 2025
9 checks passed
@Xunzhuo
Copy link
Member

Xunzhuo commented Oct 2, 2025

overall LGTM

Aias00 pushed a commit to Aias00/semantic-router that referenced this pull request Oct 4, 2025
* feat: add system prompt toggle endpoint

Signed-off-by: Huamin Chen <[email protected]>

* add cli option to explicitly enable the prompt toggle

Signed-off-by: Huamin Chen <[email protected]>

* fix test failure

Signed-off-by: Huamin Chen <[email protected]>

* fix test failure

Signed-off-by: Huamin Chen <[email protected]>

* fix test failure

Signed-off-by: Huamin Chen <[email protected]>

* adding system prompt endpoint option to makefile target

Signed-off-by: Huamin Chen <[email protected]>

* update doc

Signed-off-by: Huamin Chen <[email protected]>

* address review comment

Signed-off-by: Huamin Chen <[email protected]>

---------

Signed-off-by: Huamin Chen <[email protected]>
Signed-off-by: liuhy <[email protected]>
Aias00 pushed a commit to Aias00/semantic-router that referenced this pull request Oct 4, 2025
* feat: add system prompt toggle endpoint

Signed-off-by: Huamin Chen <[email protected]>

* add cli option to explicitly enable the prompt toggle

Signed-off-by: Huamin Chen <[email protected]>

* fix test failure

Signed-off-by: Huamin Chen <[email protected]>

* fix test failure

Signed-off-by: Huamin Chen <[email protected]>

* fix test failure

Signed-off-by: Huamin Chen <[email protected]>

* adding system prompt endpoint option to makefile target

Signed-off-by: Huamin Chen <[email protected]>

* update doc

Signed-off-by: Huamin Chen <[email protected]>

* address review comment

Signed-off-by: Huamin Chen <[email protected]>

---------

Signed-off-by: Huamin Chen <[email protected]>
Signed-off-by: liuhy <[email protected]>
Aias00 pushed a commit to Aias00/semantic-router that referenced this pull request Oct 4, 2025
* feat: add system prompt toggle endpoint

Signed-off-by: Huamin Chen <[email protected]>

* add cli option to explicitly enable the prompt toggle

Signed-off-by: Huamin Chen <[email protected]>

* fix test failure

Signed-off-by: Huamin Chen <[email protected]>

* fix test failure

Signed-off-by: Huamin Chen <[email protected]>

* fix test failure

Signed-off-by: Huamin Chen <[email protected]>

* adding system prompt endpoint option to makefile target

Signed-off-by: Huamin Chen <[email protected]>

* update doc

Signed-off-by: Huamin Chen <[email protected]>

* address review comment

Signed-off-by: Huamin Chen <[email protected]>

---------

Signed-off-by: Huamin Chen <[email protected]>
Signed-off-by: liuhy <[email protected]>
Aias00 pushed a commit to Aias00/semantic-router that referenced this pull request Oct 4, 2025
* feat: add system prompt toggle endpoint

Signed-off-by: Huamin Chen <[email protected]>

* add cli option to explicitly enable the prompt toggle

Signed-off-by: Huamin Chen <[email protected]>

* fix test failure

Signed-off-by: Huamin Chen <[email protected]>

* fix test failure

Signed-off-by: Huamin Chen <[email protected]>

* fix test failure

Signed-off-by: Huamin Chen <[email protected]>

* adding system prompt endpoint option to makefile target

Signed-off-by: Huamin Chen <[email protected]>

* update doc

Signed-off-by: Huamin Chen <[email protected]>

* address review comment

Signed-off-by: Huamin Chen <[email protected]>

---------

Signed-off-by: Huamin Chen <[email protected]>
Signed-off-by: liuhy <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants