Create Specialized LLM Models for the Lightweight LLM Server for Integration Test

[rootfs]

Is your feature request related to a problem? Please describe.
Since the router is to support Mixture-of-Models, we need to have integration test suites to verify such functionalities.

Now with #228, we are able to use a real yet lightweight models as LLM server, we can extend such server to support multiple and specialized models.

Describe the solution you'd like

• Fine tune the lightweight models, Qwen3 0.6B or Gemma 270m, for different specializations.
• Run the entire model_eval and config generation workflow
• Run benchmark to evaluate the router's performance and accuracy

@Xunzhuo @yossiovadia @yuluo-yx @JaredforReal WDYT?

[JaredforReal]

@rootfs I like this idea! Following the Docker Compose Path with mock-vllm (the most self-completed path currently) won't set up a real and complete workflow of this project. We still need a set of real LLM endpoints for Prometheus+Grafana and other features. Following the k8s path, we lack Envoy and LLM endpoints at the same time.

Share my thoughts and plans for the next days:

• Replace mock-vllm with lightweight models like Qwen3 0.6B in Docker Compose to enable a real and complete experience.
• Set up a k8s Github Action CI test.
• Finally, write an end-to-end quickstart tutorial (from training, eval to deploy) in docs.

Love to hear suggestions from the community.

[rootfs]

@JaredforReal that's a great plan! Thank you for making them happen!

[rootfs]

@JaredforReal for the end-to-end quick start, do you mind also adding the milvus for semantic cache validation?

[JaredforReal]

@JaredforReal for the end-to-end quick start, do you mind also adding the milvus for semantic cache validation?

@rootfs Sure! ​Scheduled