Skip to content
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,7 @@ def __init__(
max_refinements: int = 3,
llm_factory: Callable[[Agent], AugmentedLLM] | None = None,
context: Optional["Context"] = None,
share_memory: bool = False,
):
"""
Initialize the evaluator-optimizer workflow.
Expand All @@ -89,6 +90,7 @@ def __init__(
min_rating: Minimum acceptable quality rating
max_refinements: Maximum refinement iterations
llm_factory: Optional factory to create LLMs from agents
share_memory: Whether to share the memory between the optimizer and evaluator
"""
super().__init__(
name=name,
Expand Down Expand Up @@ -147,7 +149,8 @@ def __init__(
)
else:
raise ValueError(f"Unsupported evaluator type: {type(evaluator)}")

if share_memory:
self.evaluator_llm.share_memory_from(self.optimizer_llm)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is very interesting @jingx8885, but I'm curious if this works well in practice.

I understand your point that the evaluator should consider the full interaction to be able to determine the quality. But does it catch the issues you mentioned in the PR description when it has access to the full history? Also, does the generator get confused in follow-up interactions, or does it work well?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In my project, since the optimizer_llm calls the MCP to retrieve data and complete tasks, it is prone to hallucinations during task execution—generating results inconsistent with MCP data. Therefore, I need the evaluator_llm to verify whether the optimizer_llm adheres to the MCP data, which requires the evaluator_llm to access the optimizer_llm's historical information. I consider this scenario highly common, so I have implemented a shared memory functionality.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I strongly recommend adding this feature, as it is essential for effectively enhancing the capabilities of the evaluator_optimizer. @saqadri

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jingx8885 can you please resolve my PR feedback to get this ready for merging. Specifically, remove the updates to AugmentedLLM class (see other comment)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sorry, got it!

self.min_rating = min_rating
self.max_refinements = max_refinements

Expand Down
3 changes: 3 additions & 0 deletions src/mcp_agent/workflows/llm/augmented_llm.py
Original file line number Diff line number Diff line change
Expand Up @@ -274,6 +274,9 @@ def __init__(
self.model_selector = self.context.model_selector
self.type_converter = type_converter

def share_memory_from(self, other: "AugmentedLLM"):
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would prefer not to add this to the main API, since someone can just do

llm.history = other_llm.history

themselves

self.history = other.history

@abstractmethod
async def generate(
self,
Expand Down
Loading