LLM to generate router configuration

[rootfs]

We have a rapidly expanding configurations, it is likely a usability issue in the long term. We can keep up the documentation update but a more AI native solution is to use LLM to generate the configuration manifests.

A few actions for reviews:

• The manifest dataset. The router and semantic cache configurations, integration profiles, used in best practices as well as e2e.
• Training process. The model is to be trained for each releases.
• Validation. The model generated manifests must be validated before use.
• Deployment. Upon router deployment, the model is used to generate the configurations during initialization.

Anything else?

[szedan-rh]

hi @rootfs ,
Could you please provide for me examples to clarify the request here? If i'm understand the issue correctly is that we have a lot of configuration files and better to use AI for generate them instead of following the documentation that we have right now?

[senanz]

@noalimoy - Could you please add you idea that we discusses earlier today? I think it's related to the last point that mentioned by @rootfs (Deployment point)

[rootfs]

@senanz @noalimoy that's great! Intuitively, we want to have a chatbot-like UX that for a user input, the model will generate the config file. The workload can be improved to have an api service too so that the init container can use the api service to grab the config.
The model (and potentially with a RAG) handles the config generation in the backend.

The tuning and config verification will be in the heart of this project. If you can take these into action, that'll be great!

[liavweiss]

Hey, before starting the POC, I need some clarification (for anyone who would like to be involved). @senanz @rootfs @noalimoy

1. Use Case & User Workflow

Primary Use Case - When is Config Generated?

Current Understanding:

• Dashboard starts AFTER semantic-router is deployed and healthy
• Router already has a config file at that point
• Original requirement mentions "init container can use the api service to grab the config"

Clarification Needed:

• Option A (Primary): Generate config BEFORE deployment

  • Developer writes intent → API generates config → Review → Deploy with generated config
  • Init container calls API during deployment to generate config
  • Use case: Creating new deployments from scratch

• Option B (Secondary): Modify existing configs via Dashboard UI

  • Router already deployed with config
  • User uses Dashboard chat UI to modify existing config
  • Use case: Updating/running existing deployments

Question:
Is the primary goal to generate configs from scratch BEFORE deployment? Or is it to modify existing configs via Dashboard UI after deployment? Or both use cases?

---

2. API Service Deployment & Architecture

Current Plan:

• API service is a separate service that can be called independently
• Can be called before deployment (for init containers)
• Can be integrated with dashboard

Questions:
Where does the API service run?

• Separate Kubernetes service?
• Part of dashboard deployment?
• Standalone service that needs to be deployed separately?

---

3. Model Deployment

Questions:
For POC:

• Can we use external LLM API (OpenAI, Anthropic) instead of fine-tuning?
• Or must we fine-tune our own model?

---

3. RAG System & Milvus

Current Plan:

• Use Milvus (already used in semantic-router)
• Store config example embeddings

Questions:

• Can we use the same instance the semantic-router is using? Or do we need a separate Milvus instance for config-gen?

---

4. Dashboard Integration

Current Understanding:

• Dashboard starts after router is deployed
• Router already has config file
• Dashboard can read/update config via existing endpoints

Questions:

• Is this for generating new configs (not related to existing deployment configs), or for modifying existing configs after deployment—or both?

---

5. Scope & Priorities

Questions:
What is the minimum viable POC?

• API service + RAG + basic model + UI dashboard?

---

[rootfs]

@liavweiss cool! That's very comprehensive design. Let't have a trained model and see what it can do. The UX can come up next.