Feature Request: Expand ToolNode support for content_and_artifact response format so that it can be used in create_react_agent

[dovstern]

Feature Request

Add support for tools with response_format="content_and_artifact" in LangGraph's ToolNode and create_react_agent, enabling multimodal tools that return both text (for LLM consumption) and images/artifacts (for downstream usage).

Current Behavior

When using tools with LangGraph's create_react_agent, tool artifacts in ToolMessages aren't fully supported. While LangChain tools can return both content (text) and artifact (images, data structures), only the content portion is properly handled in the ToolNode workflow.

Current code in tool_node.py only processes the content:

elif isinstance(response, ToolMessage):
    response.content = cast(
        Union[str, list], msg_content_output(response.content)
    )
    return response

The artifact remains in the ToolMessage but isn't integrated into the agent workflow beyond basic storage.

Requested Enhancement

Enhance ToolNode to properly support tools with the @tool(response_format="content_and_artifact") decorator, ensuring:

1. Tools can return both text (for LLM consumption) and structured data/images as artifacts
2. Artifacts are accessible to downstream nodes in the workflow
3. The agent can leverage multimodal tools that generate both human-readable text and machine-usable data structures

Example Use Case

@tool(response_format="content_and_artifact")
def generate_chart(data: str) -> ToolMessage:
    """Generate a chart from data and return both text description and image."""
    # Create chart image
    chart_image = create_chart(data)
    
    # Return both text description for LLM and image for display/downstream processing
    return ToolMessage(
        content="I've analyzed the data and created a bar chart showing the top categories",
        artifact=chart_image,  # This could be bytes, base64, or other image format
        tool_call_id="..."
    )

With proper support, the agent would:

• Send the text content to the LLM for reasoning
• Make the chart image available for UI display or other agent components

Benefits

1. Multimodal Agents: Create agents that work with images, charts, and other non-text data
2. Better UX: Generate visualization artifacts while providing textual explanations
3. Composability: Build more complex workflows where artifacts are passed between tools

Potential Implementation Approach

The enhancement could:

1. Expand ToolNode to maintain both content and artifact throughout the agent execution
2. Ensure artifacts are accessible in the agent state for downstream processing
3. Add helper methods to extract and utilize artifacts from tool responses

Thank you for considering this feature request!

[vbarda]

@dovstern the messages are always accessible to the downstream nodes in any graph that uses messages, so you should be able to access the artifacts through the tool messages in the graph state. is that not enough for your use case?