Algorithms to aid towards agent development

[Bytes-Explorer]

Based on work done on Z agents, we propose to add two algorithms that will help towards tool selection problem for agent development.

1. Selecting top k tools : with few strategies on how to select top k. Each strategy can be a parameter that a user can select.
2. Tournament style (pairwise) comparison to find best tool.

Param: please add the suggested design here, so that we can iterate with @nrfulton and @HendrikStrobelt on the same and also agree on what is the best place for these components to be added.

[Param-S]

Design Proposal: Contribute to core algorithms for top k selection.

Overview:

To enhance Mellea's ability to build agents, we propose the development of a generic construct that allows for top k selection, a common usage pattern for tool selection (APIs, callable components etc). This algorithm will support various options for top k selection. This has been tried and tested for Z agents.

Objective:

Implement a core construct within Mellea that:
• Accepts a natural language task/instruction and a collection of available tools/APIs (we will broadly call them as functionalities below).
• Returns the top-k matching functionalities that best align with the user's intent.
• Supports control over how functionalities are presented and evaluated.

API Details:

• Input:

• user_instruction: A natural language description of the task or query.
• functionality_set: A structured list of available functionalities (e.g., tools, APIs), each with metadata describing its purpose, usage, examples & etc.
• k: The number of top matches to return.
• strategy_config: A configuration object that defines the matching strategy, including:
  1. Sliding Window: Controls how subsets of functionalities are chunked and evaluated. Break the functionality set into manageable chunks for efficient LLM processing.
  2. Tool Sequence Manipulation: Allows reordering or prioritization of tools before matching. Adjust the order of functionalities to influence relevance scoring
  3. Batching : Find top k tools per batch and then do a union of all the tools.
  4. Iterative Batching: Find top n tools per batch and group the results from different batches until we get top k tools.

• Output:

• A ranked list of the top-k functionalities most relevant to the user instruction

Use Cases:

• Selecting tools to execute a user task.
• Routing API calls based on user queries.
• Mapping user intents to internal services or workflows.

[Param-S]

Design Proposal: Tournament-Style Selection Function for Mellea Core Package

Objective:

To enhance Mellea's agent-building capabilities, we propose adding a built-in function to its core package that implements a tournament-style selection algorithm. This function will support dynamic selection of functionalities (e.g., APIs, callable components) based on user task/prompt relevance.

Implement a function within Mellea that:
• Accepts a natural language task/instruction and a collection of available tools/APIs (referred as functionalities).
• Returns the winning or most relevant functionality that best matches with the user's intent.
• Supports control over how functionalities are presented and evaluated (e.g., strategy type, scoring criteria).

Inputs:

• task: A natural language instruction or query.
• functionalities: A structured list of available functionalities (e.g., tools, APIs), each with metadata describing its purpose, usage, examples & etc. For eg. [{"name": ..., "description":...}, {...}]
• strategy (optional): Selection strategy to apply (e.g., "single_elimination", "round_robin").
• return_top (default=True): Return top-ranked functionality or a ranked list of functionalities.

Logic:

• The function uses a tournament-style strategy to evaluate and compare the provided functionalities.
• It leverages a large language model (LLM) to interpret the user's task and assess how well each functionality aligns with the intent.
• The LLM is used to:
- Analyze the semantic relevance of each functionality to the task.
- Generate pairwise comparisons or scores based on the configured strategy.
• The selection strategy guides how comparisons are structured:
- Single-Elimination: Functionalities are paired and compared; losers are eliminated in each round.
- Round-Robin: Each functionality is compared against all others; scores are aggregated.
• To improve accuracy and reduce variance in LLM-based evaluations, the function can be configured to repeat the tournament process multiple times. Results from repeated runs can be aggregated (e.g., via majority vote or averaged scores) to produce a more robust final selection.

Output:

• Returns a list of functionalities ranked by relevance. If the input configuration specifies a single winner (e.g., return_top=True), the list will contain only the top-ranked functionality. Otherwise, the function returns a ranked list of functionalities, each with an associated numerical score. The output format will be similar to the input structure, but instead of descriptions, each item will include a score like [ { "name": "Tool-A", "score": 5 }, { "name": "Tool-B", "score": 2 }, ...]

Design Considerations:

• Extensibility: Easy to add new strategies or scoring mechanisms.
• Configurability: Allows control over scoring criteria, number of tournament runs, and output format.

[nrfulton]

We created a mellea-contribs respository, which we plan to use as an incubation point.

I propose we incubate both of these selection strategies in mellea_contribs/tools/selectors.py. can be probably be impelemented as functions of type mellea.stdlib.base.Context -> list[X]. Here, X is whatever type want to use for tool options.

[nrfulton]

As discussed yesterday, this will be added to mellea-contribs instead of mellea itself to allow faster iteration and experimentation. I'm therefore transferring this issue to the mellea-contribs repo.