Dynamic chunk selection based on tocken limits

[oceans2892]

Hi Y'all,

Is there a way to select only the top k chunks that fit into the token window for the input? Example with made up numbers:

1. Pinecone will provide top 5 docs with a token size of 1000 each = 5000 tokens
2. The LLM is able to handle 5000 tokens but the user question itself is also part of the input and therefore only the top 4 docs should be taken to give space to the user message into the prompt.

I understand that you can fix it manually to be a low number of tokens to give enough space, there's also the ability to rerank retrievals. But is there a way to actually make it dynamic so depending on the input, a specific number of chunks will be selected to be included?

[HenryHengZJ]

Because Flowise has different LLM integrations so it's impossible for us to dynamically calculate the topK according to LLM context window and user prompt.

You'll have to implement the logic on your side, then you can call the API and override the topK config