-
Notifications
You must be signed in to change notification settings - Fork 771
feat: Implement memory sharing for EvaluatorOptimizerLLM #227
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
jingx8885
wants to merge
9
commits into
lastmile-ai:main
Choose a base branch
from
jingx8885:main
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from 8 commits
Commits
Show all changes
9 commits
Select commit
Hold shift + click to select a range
bbe1903
fix: Update _get_next_step method in orchestrator.py
jingx8885 80312c5
Merge branch 'lastmile-ai:main' into main
jingx8885 ec1e42f
Refactor: Improve AsyncEventBus initialization
jingx8885 51f6bdb
Merge branch 'main' of github.com:jingx8885/mcp-agent
jingx8885 295fe3c
feat: Implement memory sharing for EvaluatorOptimizerLLM
jingx8885 c729535
Merge branch 'lastmile-ai:main' into main
jingx8885 98bb921
Add input validation and documentation for the memory sharing method.
jingx8885 e741171
Merge branch 'main' of github.com:jingx8885/mcp-agent
jingx8885 5cabdf0
resolve PR feedback
jingx8885 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -274,6 +274,26 @@ def __init__( | |
self.model_selector = self.context.model_selector | ||
self.type_converter = type_converter | ||
|
||
def share_memory_from(self, other: "AugmentedLLM"): | ||
|
||
""" | ||
Share memory from another AugmentedLLM instance. | ||
|
||
This creates a reference to the other instance's history, meaning both | ||
instances will share the same memory object and any changes to history | ||
in either instance will be reflected in both. | ||
|
||
Args: | ||
other: The AugmentedLLM instance to share memory from | ||
|
||
Raises: | ||
ValueError: If other is None or not an AugmentedLLM instance | ||
""" | ||
if other is None: | ||
raise ValueError("Cannot share memory from None") | ||
if not isinstance(other, AugmentedLLM): | ||
raise ValueError("Can only share memory from another AugmentedLLM instance") | ||
self.history = other.history | ||
|
||
@abstractmethod | ||
async def generate( | ||
self, | ||
|
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is very interesting @jingx8885, but I'm curious if this works well in practice.
I understand your point that the evaluator should consider the full interaction to be able to determine the quality. But does it catch the issues you mentioned in the PR description when it has access to the full history? Also, does the generator get confused in follow-up interactions, or does it work well?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In my project, since the optimizer_llm calls the MCP to retrieve data and complete tasks, it is prone to hallucinations during task execution—generating results inconsistent with MCP data. Therefore, I need the evaluator_llm to verify whether the optimizer_llm adheres to the MCP data, which requires the evaluator_llm to access the optimizer_llm's historical information. I consider this scenario highly common, so I have implemented a shared memory functionality.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I strongly recommend adding this feature, as it is essential for effectively enhancing the capabilities of the evaluator_optimizer. @saqadri
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@jingx8885 can you please resolve my PR feedback to get this ready for merging. Specifically, remove the updates to AugmentedLLM class (see other comment)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
sorry, got it!