LLM Fails to Extract Specific String Argument Values from State

[utgarda]

Problem:
The agent's LLM sometimes fails to correctly extract specific string values (especially large numeric strings representing IDs) from the provided AgentState when constructing arguments for GameFunction calls. Instead of the actual value from the state, it passes placeholders, hallucinated values (e.g., "12345"), or incorrectly formatted strings (e.g., empty string).

Observation:
This issue seems model-dependent. It occurs frequently with DeepSeek V3 but is resolved or less frequent when switching to R1. This happens even with explicit instructions in the agent's prompts to extract the value correctly.

Impact:
Workers requiring specific IDs (e.g., gameId for SearchWorker or AnalysisWorker) fail due to invalid arguments, blocking the agent's workflow.

Example Failure:

• Agent State contains: matchesForAttention: [{ gameId: "1006000000000026939314", ... }]
• Agent Plan targets this match for start_gathering_match_info.
• Function Call Args: {"game_id": {"value": "12345"}} or {"game_id": {"value": ""}} (Incorrect)
• Result: Worker function fails due to invalid/non-existent ID.

Request:
Could the framework team investigate potential reasons for this argument extraction failure with certain LLMs and advise on prompt strategies or potential framework enhancements to improve reliability?

Log Excerpts Illustrating Incorrect gameId Extraction:

(1) Agent state shows the correct gameId for the target match:

🤖 [GMbets Agent] [12:00:58 PM]
Agent State:
{
  "betHistorySummary": { ... },
  "matchesForAttention": [
    ...
    {
      "gameId": "1006000000000026939314",  // <-- Correct ID is present
      "teamA": "Defensores de Cambaceres (Reserves)",
      "teamB": "General Lamadrid II",
      "league": "Reserves League",
      "startTime": 1744374600,
      "startTimeUTCString": "2025-04-11T12:30:00.000Z",
      "status": "needs_info_gathering"
    },
    ...
  ],
  ...
}
------------------------


(2) Agent correctly plans to target this match (Task_7):

🤖 [GMbets Agent] [12:03:35 PM]
Action State:
{
  ...
  "current_task": {
    "task_id": "15df0d75-7015-410a-9e2f-8f434498b549",
    "task": "Initiate information collection for the match between Defensores de Cambaceres (Reserves) and General Lamadrid II by calling the 'start_gathering_match_info' function of the 'searchWorker'. Ensure to accurately extract the correct 'gameId' string value ...", // Task description is correct
    "location_id": "search_worker",
    "llp": {
      "plan_id": "ebaebc18-a3b0-441c-81f1-6983977d9803",
      "plan": [
        "Extract 'gameId' from match object",
        "Call 'start_gathering_match_info' function with 'gameId'" // LLP Plan is correct
      ],
      ...
    },
    ...
  },
  ...
}.
------------------------

(3) Agent performs the function call with an INCORRECT, hallucinated gameId:

🔄 [GMbets Agent] [12:03:35 PM]
Performing function start_gathering_match_info with args {"game_id":{"value":"12345"}}.  <-- Incorrect ID ("12345")
------------------------

(4) Worker receives the incorrect ID and fails:

🤖 [GMbets Agent] [12:03:35 PM]
[SearchWorker] Received request for game: 12345
------------------------
 [Warn] [Repo] Match with id 12345 not found in the bets table. Cannot get details for questions.
🤖 [GMbets Agent] [12:03:35 PM]
[SearchWorker] Match with id 12345 does not exist. Probably wrong id provided.
------------------------
❌ [GMbets Agent] [12:03:35 PM]
Function status [failed]:
{
  "message": "Match with id 12345 does not exist. Probably wrong id provided."
}.
------------------------

(5) Agent retries but passes an empty string for gameId:

🔄 [GMbets Agent] [12:04:36 PM]
Performing function start_gathering_match_info with args {"game_id":{"value":""}}. <-- Incorrect ID (Empty)
------------------------
(6) Worker fails due to invalid argument format:

🤖 [GMbets Agent] [12:04:36 PM]
[SearchWorker] Returning feedback: Invalid arguments: [
{
  "validation": "regex",
  "code": "invalid_string",
  "message": "Must be a numeric string",
  "path": [
    "game_id"
  ]
}]
------------------------
❌ [GMbets Agent] [12:04:36 PM]
Function status [failed]:
{
  "message": "Invalid arguments: [{\"validation\":\"regex\",\"code\":\"invalid_string\",\"message\":\"Must be a numeric string\",\"path\":[\"game_id\"]}]"
}.
------------------------