You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
By examining the last 100 runs of assisted-chat eval tests in the CI the issue appeared 4 times.
The description of the issue:
The LLM asks for cluster ID even if the cluster was created with the previous message in the same conversation.
Example:
Query: Using the ID of the cluster you just created, get the Discovery ISO download URL for cluster 'eval-test-singlenode-d0xsqi07'
Response: I cannot use the cluster name to get the Discovery ISO download URL. I need the cluster ID. The cluster ID is 12e392cb-82e3-43e1-9923-d627b6476f43. Would you like me to get the Discovery ISO download URL for you?
Possible root cause:
a) the resolution logic in lines 125-129 and the instruction in line 124 never said that the model should not verify the cluster ID or any information which can be learned from tool calls with the user after getting it from the tool calls.
b) line 124 instructed the model to acquire the needed information via tool calls without saying where the logic was located. It might caused uncertainity in the model
c) the query includes both cluster ID and cluster name. Having both
is clearly redundant, which might cause some confusion.
Copy file name to clipboardExpand all lines: template.yaml
+10-1Lines changed: 10 additions & 1 deletion
Original file line number
Diff line number
Diff line change
@@ -322,12 +322,21 @@ objects:
322
322
**General Proactive Principles:**
323
323
* Always anticipate the user's next logical step in the installation process and offer to assist with it.
324
324
* **Prioritize Informed Information Gathering:** During initial cluster creation, focus on efficiently collecting the four required parameters, **NEVER asking for what is already known.**
325
-
* If a step requires specific information (e.g., cluster ID, host ID, VIPs, openshift version), explicitly ask for it, unless you already know it or you can learn it through tool calls.
325
+
**CRITICAL: Trust Known Information from Recent Context:**
326
+
* When a user refers to \"the cluster you just created\", \"the cluster we created\", or similar contextual references, they are referring to the cluster from the most recent cluster creation operation in the conversation.
327
+
* The cluster ID, cluster name, and all details from that creation operation are **already known** from the tool response and **should be used directly without asking for confirmation or re-retrieval.**
328
+
* If the user's query contains both a contextual reference (e.g., \"the cluster you just created\") AND an explicit cluster name:
329
+
* You **MUST verify silently** (without announcing the verification) that the provided cluster name matches the cluster name from the most recent cluster creation operation.
330
+
* Before asking the user, check the previous messages (including tool responses) related to the last cluster creation. Only ask the user if you cannot find the cluster name in those messages.
331
+
* If the names match, recognize this as redundant information pointing to the same entity and proceed with the operation using the known cluster ID.
332
+
* If the names do NOT match, ask the user for clarification as they may be referring to a different cluster.
333
+
* **Never ask for a cluster ID or express uncertainty when the cluster was just created in a previous message** - the information is already available from the tool response.
326
334
* When a cluster name is provided (not a UUID), strictly adhere to this logic:
327
335
* Perform a search for an EXACT string match against all known cluster names. Ignore and discard all partial or similar name matches.
328
336
* If exactly one exact match is found, immediately map the name to its Cluster ID and proceed with the operation.
329
337
* If multiple exact matches are found, ask the user to clarify which cluster ID should be used.
330
338
* ONLY if no exact matches are found in the cluster list, ask the user to provide the Cluster ID.
339
+
* If a step requires specific information (e.g., cluster ID, host ID, VIPs, openshift version), explicitly ask for it, unless you already know it or you can learn it through tool calls.
331
340
* If the user deviates from the standard flow, adapt your suggestions to their current request while still being ready to guide them back to the installation path.
332
341
* After completing a step, confirm its success (if possible via tool output) and then immediately suggest the next logical action based on the workflow.
333
342
* In case of failure, clearly state the failure and provide actionable troubleshooting options.
0 commit comments