Skip to content

Conversation

@OneZero-Y
Copy link
Contributor

What type of PR is this?

Feature - implement batch classification API

What this PR does / why we need it:

This PR implements batch classification API functionality, resolving the API completeness issue where POST /api/v1/classify/batch returned HTTP 501 "Not Implemented" error.

Changes:

  1. Add batch classification API: Support processing multiple texts in a single request (configurable batch size)
  2. Implement configurable processing: Sequential for small batches, concurrent for large batches
  3. Add resource control: Configurable concurrency limits to prevent system overload

Configuration Examples:

Small batch request (sequential processing):

{
  "texts": ["solve math equation", "write business plan", "chemistry experiment"]
}

Large batch request (concurrent processing):

{
  "texts": [
    "solve differential equation", "business strategy analysis", 
    "chemistry reaction", "physics calculation", "market research",
    "mathematical modeling", "financial planning", "scientific experiment"
  ],
  "options": {"return_probabilities": true}
}

Configuration Examples:

# API Configuration
api:
  batch_classification:
    max_batch_size: 100          # Maximum number of texts in a single batch
    concurrency_threshold: 5     # Switch to concurrent processing when batch size > this value
    max_concurrency: 8           # Maximum number of concurrent goroutines

Which issue(s) this PR fixes:

Fixes: implement batch classification API

@github-actions
Copy link

github-actions bot commented Sep 3, 2025

👥 vLLM Semantic Team Notification

The following members have been identified for the changed files in this PR and have been automatically assigned:

📁 src

Owners: @rootfs, @Xunzhuo, @wangchen615
Files changed:

  • src/semantic-router/pkg/api/server_test.go
  • src/semantic-router/pkg/api/server.go
  • src/semantic-router/pkg/config/config.go

📁 config

Owners: @rootfs
Files changed:

  • config/config.yaml

📁 website

Owners: @Xunzhuo
Files changed:

  • website/docs/api/classification.md

This comment was automatically generated based on the OWNER files in the repository.

@rootfs
Copy link
Collaborator

rootfs commented Sep 3, 2025

@OneZero-Y thanks for contributing again! Would you please run pre-commit run --all-files? Maybe you can open another PR to add this to CONTRIBUTING.md

@netlify
Copy link

netlify bot commented Sep 3, 2025

Deploy Preview for vllm-semantic-router ready!

Name Link
🔨 Latest commit 76e3916
🔍 Latest deploy log https://app.netlify.com/projects/vllm-semantic-router/deploys/68b916324f8aac0008dc912e
😎 Deploy Preview https://deploy-preview-24--vllm-semantic-router.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

@rootfs
Copy link
Collaborator

rootfs commented Sep 3, 2025

@OneZero-Y can you share your local test with batch classify and potentially add this to the doc too?

Copy link
Member

@Xunzhuo Xunzhuo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for working on it, please update the classify API docs.

@OneZero-Y OneZero-Y force-pushed the feat/batch-classification-api branch from e933caa to 289d231 Compare September 4, 2025 04:23
@OneZero-Y OneZero-Y force-pushed the feat/batch-classification-api branch from 289d231 to 76e3916 Compare September 4, 2025 04:31
@rootfs rootfs merged commit 35c8ec8 into vllm-project:main Sep 4, 2025
9 checks passed
@OneZero-Y OneZero-Y deleted the feat/batch-classification-api branch September 5, 2025 12:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants