Provide a way not to double tokenize the prompt in case of the token-aware KV routing

[atchernych]

What would you like to be added:
A need a way to have a plugin which can modify the request body when the EPP picks the worker.

Not sure what the best way to go about it. I envision:

1. The director creates the LLMRequest once (with an empty map) and hands the same object through scheduling so the scorer’s annotation sticks around. It needs to have an extra annotations field.
2. The EPP routing plugin fills in req.Annotations[tokenDataAnnotationKey] =
3. After scheduling succeeds, runPreRequestPlugins invokes each plugin and (for mutators) passes the live body map. It needs to include the body mutations.
4. Another pre-request plugin pkg/epp/requestcontrol/plugins/new/plugin.go reads this body and copies the data into the body to send to the workers.
5. Once the director returns, StreamingServer.Process marshals reqCtx.Request.Body (now containing token_data) and stores it in the ext-proc responses that go back to Envoy.
6. From there Envoy forwards those mutated bytes downstream. In short: router plugin → request annotations → PreRequest mutator → marshalled body → Envoy → worker.

Why is this needed:

I have an EPP routing plugin which does token-aware kv-routing. It uses the model's tokenizer. My plugin identifies the best worker but also tokenizes the prompt in the process. I want to be able to pass these tokens to the serving workers in the request body. Otherwise my workers will de-tokenize again. This would introduce additional latency.

[nirrozenbaum]

@atchernych thanks for opening this issue.
PreRequest plugin sounds like what you need.
I still trying to understand the annotations part, which I'm not sure I fully understood.
is it the same plugin (same struct) that implements both Score function (of scorer interface) and PreRequest function?

am I correct to say that your plugin implements both interfaces, it tokenizes the prompt in the scoring function and then you want to make these tokens available in the request body using PreRequest function?

[atchernych]

For context please take a look at this: https://github.com/atchernych/gateway-api-inference-extension-dynamo/pull/7/files

[atchernych]

My plugin is implemented as a scorer at this time. By annotations I mean additional data in the request body (possible custom flags I want to add and tokens). yes, It is correct to say that my plugin provides a score and tokenizes the prompt (since it does token aware kv-routing it has to tokenize). The issue is I do not want to tokenize again during request serving.

[nirrozenbaum]

@atchernych thanks. got it.
I think what you’re looking for is “tokens in tokens out”. we just initiated discussions on the topic in llm-d, where we’re going to design a solution for this. it should land eventually in this repo but would be great if you join the discussions to make sure your requirements are met.

the dedicated slack channel #disaggregated-tokenization is here:
👋 Let’s work together in Slack.
https://join.slack.com/share/enQtOTk0MTc1ODM4NTk1OS0xYWVjZmY0NzY5YzNjYzc4ZDcwNmQ5ZWNiM2EwZmJkMjg0ZDdiMDQ5OTg0ODUzMDJkMWRiMTIzZGNiZWZiMzI5