-
Notifications
You must be signed in to change notification settings - Fork 1
Open
Labels
bugSomething isn't workingSomething isn't working
Description
Problem Summary:
The assistant's web tool calls are malformed which causes the web tool to return parsing errors. Symptoms: the web tool returns "Unknown subcommand 'safety_valve'" and later errors showing assistant-generated narration was concatenated with the JSON payload, causing the tool to receive invalid input.
Evidence:
- feedback/4KGeSJsSjB/message_100_14.txt: assistant attempted a web call with arguments containing "safety_valve":"10k" which the web tool does not accept.
- feedback/4KGeSJsSjB/messages.json (ftm_num 15): tool returned: "error: Unknown subcommand 'safety_valve', tool supports 'search' and 'open'".
- feedback/4KGeSJsSjB/message_100_16.txt and message_100_17.txt: show the web call JSON followed immediately by plaintext narration (e.g. '...}],"}]}The web tool returned an error in the previous call; retrying without the unsupported parameter...'), indicating extraneous text was appended to function-call arguments.
- feedback/4KGeSJsSjB/message_100_20.txt: final assistant reply notes web/tool search attempts and content, confirming the flow of events.
Root cause:
The assistant model produced invalid function-call payloads by:
- Including unsupported parameter keys (e.g. safety_valve) in the web tool arguments.
- Appending human-readable narration/debugging text directly into the serialized function-call arguments, producing malformed JSON/CLI input that the tool rejects.
This indicates either the function-call serialization layer did not strictly validate/clean the model output before sending to the tool, or the model was allowed to emit freeform text within a function call context.
Recommended next steps:
- Enforce strict schema validation on function-call arguments before invoking tools. Reject or sanitize any unknown keys (e.g. safety_valve).
- Strip any assistant-generated narration from function-call payloads; only allow structured JSON matching the tool's parameter schema.
- Add unit/integration tests that simulate model outputs containing trailing narration or unsupported fields to ensure the caller sanitizes or rejects them.
- Optionally, if 'safety_valve' is a desired option, extend the web tool interface to accept it (but prefer input validation first).
- Investigate why the assistant produced repetitive apology/narration loops; consider adjusting persona or adding guardrails preventing verbose narration inside tool call contexts.
Attachments:
- feedback/4KGeSJsSjB/messages.json
- feedback/4KGeSJsSjB/message_100_14.txt
- feedback/4KGeSJsSjB/message_100_16.txt
- feedback/4KGeSJsSjB/message_100_17.txt
- feedback/4KGeSJsSjB/message_100_20.txt
- feedback/4KGeSJsSjB/logs.txt
Please triage and assign to the runtime/tools-invocation team.
@flexus-bob please take a look
Metadata
Metadata
Assignees
Labels
bugSomething isn't workingSomething isn't working