Dynamic example selection

[aryamccarthy]

When doing ICL, sometimes there's reason to select amongst the five-star options based on similarity to the input; it can lead to better performance. Is there any interest in creating hooks for this dynamic example selection within the core library?

It'd take some finagling with the existing structure: right now, the input and the prompt are pretty well separated from each other until they're combined in the chat formatter. I'm curious whether there's effort in this direction—I do see some vector store PRs out there.

For my use case, this dynamic selection for ICL would be useful, and I'm going to wind up building it either way. It'd be great to work within the library instead of trying to circumvent it with some odd inheritance/monkeypatching. Grateful for y'all's thoughts!

[scosman]

Oh interesting. We're in the process of adding RAG now, which is similar in tooling (embeddings/search/tools). It would be possible to do something similar with tools+RAG, although you'd need to load the examples into documents in our initial version; it can't automatically pull from the dataset and filter to highly rated like we do for prompt gen. We could add that in a future version, and it does make sense.

@leonardmq possible extension to your work for future iteration.

@aryamccarthy I wouldn't suggest jumping into the RAG/index branches quite yet, they are changing often. I'd probably suggest implementing as a tool vs prompt generator (like you say, monkey patching will be needed, not setup for that). Basically a "fetch_examples(input)" tool that can add N relevant examples to the context. Tool calling isn't in main yet, but it's a lot closer (the tools branch). cc @aiyakitori

[aryamccarthy]

Really interesting stuff now that the tools branch is merged; congrats y'all!

Looking into it, I'm curious about a few things.

1. I'm having trouble seeing (as of 0.19.1) how to integrate tools that are not MCP, like the built-in math tools y'all define. E.g. if we wanted to define a local, in-memory vector store. Right now, looks like it's required that a tool either be a built-in math tool or an MCP tool.
2. How would you suggest to manage the token budget? E.g. populate the prompt with the tool-called examples, but only insofar as it doesn't exceed the context window size?

[scosman]

Re 1 -- For external integrations, it's all MCP for now. Are you working in the app or with the library?

If app, MCP is the way to go long term. You can create a MCP wrapper over any custom tool pretty easily with a MCP server library.

If library, we could add the ability to register custom non-MCP tools in a future release. That way you could keep it in process. If that's the case let us know the use case. For now, you'll have to register a MCP server. We have lots of tools enhancements coming, but there's a good queue 😀

2. No specific guidance on that front. The tools definitions themselves take some of the context window. If you need examples will be case-by-case.

Note: try expanding the "Show Raw Output" in app to see the trace including tool definitions calls. It's helpful for debugging. Video with examples: https://www.youtube.com/watch?v=qh0FIrLMrII

[aryamccarthy]

Thanks - that's a lot of useful context! I'm using the library, not the app, and the use case is: "I'd love to dynamically select the most relevant examples (not just highest rated) without asking Gemini/Claude/whichever to decide that a tool invocation is needed."

At this point, the easiest way I see (as a library consumer, not maintainer) is to subclass LiteLlmAdapter and override _run, manually injecting a tool-invocation turn (the fetch_examples(input) you pitched above) into the chat history. (Do you see a better approach?)

It seems a bit heavy to set up an MCP server for this. The tool should be deterministically invoked, and it depends on local data: the existing examples in the task's dataset. That's why I'm pitching the in-process tool—but maybe that's me solving an XY problem.