Description
codex-acp rollouts land in acp_trajectory.jsonl with every execute tool call reduced to its title and every edit call reduced to "Editing files": no command output, no patch. The viewer then shows tool cards with nothing under the header.
The information is on the wire, the capture drops it:
acp/session.py handle_update, tool_call branch: the record is built from toolCallId, title, and kind only. The notification's content and rawInput are discarded. codex-acp 1.6.0 sends its file-change patches as content on the initial tool_call (createFileChangeUpdate), so edits lose their diffs here.- Same function,
tool_call_update branch: only content is merged. codex-acp sends the command as rawInput: {command, cwd} on the tool_call and the output as rawOutput: {formatted_output, exit_code} on the completing tool_call_update (createCommandExecutionUpdate, createCommandExecutionCompleteUpdate); neither field is read anywhere in src/benchflow.
trajectories/_capture.py _events_to_trajectory writes content only, so even a captured raw payload would not reach the JSONL.
Checked on the 540-rollout FrontierPhysics GPT batch (benchflow 0.7.6.dev2102): 0 of 64 tool calls in a sample rollout carry content; the same rollout's LiteLLM log has all 45 commands, 11 patches, and every output.
Proposal
ToolCallRecord keeps raw_input and raw_output; the tool_call branch stores the initial content, rawInput, and rawOutput, the tool_call_update branch merges rawInput / rawOutput when present (later wins)._events_to_trajectory emits raw_input / raw_output only when set, so trajectories of agents that never send them stay byte-identical.- The viewer payload falls back to the raw fields when a tool call has no content blocks:
execute renders the command followed by the formatted output and exit code, other kinds render the JSON.
I will open the PR; tests will guard each of the three points.
Description
codex-acp rollouts land in
acp_trajectory.jsonlwith everyexecutetool call reduced to its title and everyeditcall reduced to "Editing files": no command output, no patch. The viewer then shows tool cards with nothing under the header.The information is on the wire, the capture drops it:
acp/session.pyhandle_update,tool_callbranch: the record is built fromtoolCallId,title, andkindonly. The notification'scontentandrawInputare discarded. codex-acp 1.6.0 sends its file-change patches ascontenton the initialtool_call(createFileChangeUpdate), so edits lose their diffs here.tool_call_updatebranch: onlycontentis merged. codex-acp sends the command asrawInput: {command, cwd}on thetool_calland the output asrawOutput: {formatted_output, exit_code}on the completingtool_call_update(createCommandExecutionUpdate,createCommandExecutionCompleteUpdate); neither field is read anywhere insrc/benchflow.trajectories/_capture.py_events_to_trajectorywritescontentonly, so even a captured raw payload would not reach the JSONL.Checked on the 540-rollout FrontierPhysics GPT batch (benchflow 0.7.6.dev2102): 0 of 64 tool calls in a sample rollout carry content; the same rollout's LiteLLM log has all 45 commands, 11 patches, and every output.
Proposal
ToolCallRecordkeepsraw_inputandraw_output; thetool_callbranch stores the initialcontent,rawInput, andrawOutput, thetool_call_updatebranch mergesrawInput/rawOutputwhen present (later wins)._events_to_trajectoryemitsraw_input/raw_outputonly when set, so trajectories of agents that never send them stay byte-identical.executerenders the command followed by the formatted output and exit code, other kinds render the JSON.I will open the PR; tests will guard each of the three points.