[pull] main from openai:main #10
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This pull request resolves #1564 --------- Co-authored-by: Rohan Mehta <[email protected]>
) When using LiteLLM with agents framework, the agent run pollutes metadata with non-string values (like hidden_params dict), causing subsequent dataclasses.replace() calls to fail with Pydantic validation errors. This happens because: 1. Agent.reset_tool_choice defaults to True 2. After tool usage, maybe_reset_tool_choice() calls dataclasses.replace() 3. LiteLLM adds hidden_params (dict) to ModelSettings.metadata 4. ModelSettings expects metadata: dict[str, str] but gets dict[str, Any] ```python #!/usr/bin/env python3 """ Reproduce script for agents framework bug with LiteLLM metadata pollution. """ import asyncio from agents import Agent, ModelSettings, Runner, function_tool, set_tracing_disabled from agents.extensions.models.litellm_model import LitellmModel set_tracing_disabled(disabled=True) API_KEY = "xxx" BASE_URL = "xxx" # Create a simple tool that the agent can call @function_tool def get_time() -> str: """Get the current time.""" from datetime import datetime return f"Current time: {datetime.now().strftime('%H:%M:%S')}" async def main(): print("=== Reproducing LiteLLM metadata pollution bug ===") agent = Agent[str]( name="test_agent", model=LitellmModel(model="gpt-4o-mini", base_url=BASE_URL, api_key=API_KEY), instructions="You are a helpful assistant. Use the get_time tool when asked about time.", tools=[get_time], # Add the tool to trigger reset_tool_choice behavior reset_tool_choice=True, # This is the default, triggers maybe_reset_tool_choice() ) print(f"Agent reset_tool_choice: {agent.reset_tool_choice}") print(f"Agent tools: {[tool.name for tool in agent.tools]}") print("\n--- First run: Agent calls tool without metadata ---") try: result1 = await Runner.run( starting_agent=agent, input="What time is it?", # This should trigger the get_time tool max_turns=3, ) print(f"✓ First run successful") print(f"Result: {result1.final_output}") except Exception as e: print(f"✗ Run failed: {e}") return metadata = { "version": "1.0.0", "user_id": "test_user", "session_id": "test_session", } model_settings = ModelSettings(metadata=metadata) agent = Agent[str]( name="test_agent", model=LitellmModel(model="gpt-4o-mini", base_url=BASE_URL, api_key=API_KEY), instructions="You are a helpful assistant. Use the get_time tool when asked about time.", tools=[get_time], # Add the tool to trigger reset_tool_choice behavior model_settings=model_settings, reset_tool_choice=True, # This is the default, triggers maybe_reset_tool_choice() ) print(f"Initial metadata: {list(model_settings.metadata.keys()) if model_settings.metadata else None}") print(f"Agent reset_tool_choice: {agent.reset_tool_choice}") print(f"Agent tools: {[tool.name for tool in agent.tools]}") print("\n--- Second run: Agent calls tool with metadata ---") try: result1 = await Runner.run( starting_agent=agent, input="What time is it?", # This should trigger the get_time tool max_turns=3, ) print(f"✓ Second run successful") print(f"Result: {result1.final_output}") # Check metadata pollution if agent.model_settings.metadata: print(f"Metadata keys after second run: {list(agent.model_settings.metadata.keys())}") # Show the problematic hidden_params if "hidden_params" in agent.model_settings.metadata: hidden_params = agent.model_settings.metadata["hidden_params"] print( f"✓ hidden_params added: {type(hidden_params)} with {len(hidden_params) if isinstance(hidden_params, dict) else 'N/A'} items" ) print(f" This violates ModelSettings.metadata: dict[str, str] constraint") except Exception as e: import traceback print(f"✗ Run failed:\n{e}") print(f"Traceback:\n{traceback.format_exc()}") return if __name__ == "__main__": asyncio.run(main()) ``` output ```text === Reproducing LiteLLM metadata pollution bug === Agent reset_tool_choice: True Agent tools: ['get_time'] --- First run: Agent calls tool without metadata --- ✓ First run successful Result: The current time is 14:18:47. Initial metadata: ['version', 'user_id', 'session_id'] Agent reset_tool_choice: True Agent tools: ['get_time'] --- Second run: Agent calls tool with metadata --- ✗ Run failed: 1 validation error for ModelSettings metadata.hidden_params Input should be a valid string [type=string_type, input_value={'custom_llm_provider': '...'_response_ms': 797.449}, input_type=dict] For further information visit https://errors.pydantic.dev/2.11/v/string_type """ ```
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
See Commits and Changes for more details.
Created by
pull[bot] (v2.0.0-alpha.3)
Can you help keep this open source service alive? 💖 Please sponsor : )