UI tools by torchiaf · Pull Request #178 · rancher/rancher-ai-agent

torchiaf · 2026-03-30T07:40:47Z

Related issue: rancher/rancher-ai-ui#190
Related issue: #179

UI Tools Feature

Overview

This PR introduces UI Tools, It enables the AI agent to dynamically select Ui tools declared by the UI. The agent can recommend appropriate UI tools based on the conversation context, and the LLM intelligently selects which tools to invoke with their respective parameters.

Key Features

Kubernetes-Native Configuration: UI tools are defined as UIToolsConfig Custom Resource Definitions (CRDs) in Kubernetes
Per-Config Tool Isolation: Tools are scoped to their configuration, preventing namespace collisions across multiple tool sets
Configurable Tool Limits: Each configuration can specify a maximum number of unique tools the LLM can select per response

Architecture

Components

1. Registry (`app/services/ui_tools/registry.py`)

Central registry for managing UI tools with config-scoped namespacing.

Key Classes:

UITool: Represents a single UI tool with metadata and schema
UIToolSchema: Defines input validation schema for tools
UIToolCall: Represents a tool invocation by the agent
UIToolsConfig: Configuration for a set of UI tools
UIToolsRegistry: Manages tools in a config-scoped dictionary tools_by_config[config_name][tool_name]

2. Loader (`app/services/ui_tools/loader.py`)

Loads UI tool definitions from Kubernetes CRDs into the registry.

Key Functions:

reload_ui_tools_config(): Reload the tools from watcher updates.

3. Selector (`app/services/ui_tools/selector.py`)

Uses LLM to intelligently select appropriate UI tools for a given context.

Key Classes:

UIToolsSelector: Main selector class that invokes LLM for tool selection

Features:

Works with the LLM provider
Includes tool schema information in LLM prompt for accurate selection
Deduplicates tool calls
Enforces maxTools limit by unique tool name:
- LLM can select up to maxTools different tools (by unique name)
- Same tool can be called multiple times with different parameters
- Example with maxTools=2: tool_A(params1), tool_A(params2), tool_B ✓ but tool_C ✗

Configuration

Kubernetes CRD: UIToolsConfig

apiVersion: ai.cattle.io/v1alpha1
kind: UIToolsConfig
metadata:
  name: my-tools
spec:
  config:
    enabled: true                    # Enable/disable this tool set
    revision: 1                      # For change detection/gating
    systemPrompt: "..."              # Custom instructions for tool selection
    maxTools: 2                       # Max unique tools per response (0 = unlimited)
  
  tools:
    - name: chart_tool
      category: visualization
      description: "Renders data charts"
      enabled: true
      schema:
        type: object
        properties:
          chart_type:
            type: string
            description: "Type of chart (bar, line, pie)"
          data:
            type: object
            description: "Chart data"
        required: [chart_type, data]
      metadata:
        icon: "chart"
      revision: 1
    
    - name: table_tool
      category: data-display
      description: "Renders data in table format"
      enabled: true
      schema:
        type: object
        properties:
          columns:
            type: array
            description: "Column definitions"
          rows:
            type: array
            description: "Row data"
        required: [columns, rows]
      metadata:
        icon: "table"
      revision: 1

Configuration Fields

Config-Level Settings

Field	Type	Default	Description
`enabled`	boolean	true	Whether this tool set is active
`revision`	integer	0	Revision number for change tracking
`systemPrompt`	string	null	Custom instructions for tool selection LLM
`maxTools`	integer	5	Maximum unique tools per response (0 = unlimited)

Tool-Level Settings

Field	Type	Required	Description
`name`	string	✓	Unique tool identifier
`category`	string	✓	Tool classification (e.g., "visualization", "data-display")
`description`	string	✓	What the tool does (UI usage)
`prompt`	string	✓	System Prompt for the llm to select the tool
`schema`	object	✓	JSON Schema for input validation
`enabled`	boolean	✓	Enable/disable specific tool
`metadata`	object		Custom metadata (e.g., icon, display hints)
`revision`	integer		Tool definition version

Usage

1. Define Tools in Kubernetes

kubectl apply -f - <<EOF
apiVersion: ai.cattle.io/v1alpha1
kind: UIToolsConfig
metadata:
  name: dashboard
spec:
  config:
    enabled: true
    revision: 1
    maxTools: 2
    systemPrompt: "Select tools to visualize data effectively"
  
  tools:
    - name: bar_chart
      category: chart
      description: "Visualize data with a bar chart"
      enabled: true
      schema:
        type: object
        properties:
          title: { type: string }
          values: { type: array, items: { type: number } }
        required: [title, values]
EOF

2. Agent Uses Tools

The agent automatically:

Loads available tools for the configuration
Calls the LLM with tool descriptions
LLM selects appropriate tools with parameters
Agent validates inputs against schemas
Sends tool calls to frontend for rendering

Implementation Details

Tool Selection Flow

┌─────────────────────┐
│  Agent Response     │
└──────────┬──────────┘
           │
           ▼
┌─────────────────────────────────────┐
│  Load Tools from Registry           │
│  (config-scoped retrieval)          │
└──────────┬──────────────────────────┘
           │
           ▼
┌─────────────────────────────────────┐
│  Format Tools for LLM Prompt        │
│  (include schemas and descriptions) │
└──────────┬──────────────────────────┘
           │
           ▼
┌──────────────────────────────────────┐
│  LLM Selection                       │
│  - Analyze context                   │
│  - Select at most maxTools unique    │
│    tools (same tool allowed with     │
│    different params)                 │
└──────────┬───────────────────────────┘
           │
           ▼
┌──────────────────────────────────────┐
│  Extract Selections       │
│  - Parse JSON tool calls from LLM    │
│  - Validate inputs vs schema         │
│  - Deduplicate identical calls       │
│  - Enforce maxTools limit by         │
│    unique tool name                  │
└──────────┬───────────────────────────┘
           │
           ▼
┌──────────────────────────────────────┐
│  Return to Frontend for Rendering    │
└──────────────────────────────────────┘

Config-Scoped Namespacing

Problem Solved: When multiple UIToolsConfig resources exist, tools with the same name could collide.

Solution: Nested dictionary structure

registry.tools_by_config = {
  "config_a": {
    "chart_tool": UITool(...),
    "table_tool": UITool(...),
  },
  "config_b": {
    "chart_tool": UITool(...),  # Different from config_a's chart_tool
    "custom_tool": UITool(...),
  }
}

Revision-Gated Updates

Prevents race conditions when multiple clients update configurations:

if provided_revision > current_revision:
    # Update all revision-gated fields
    resource['spec']['config']['systemPrompt'] = provided_system_prompt
    resource['spec']['config']['enabled'] = provided_enabled
    resource['spec']['config']['maxTools'] = provided_max_tools
else:
    # Preserve current values
    resource['spec']['config']['systemPrompt'] = current_system_prompt
    resource['spec']['config']['enabled'] = current_enabled
    resource['spec']['config']['maxTools'] = current_max_tools

resource['spec']['config']['revision'] = provided_revision

Integration Points

1. Agent State (base.py)

Calls _dispatch_ui_tools_event() to select tools
Passes context and available tools to selector
Retrieves max_tools from registry config
Saves tool calls to LangGraph state (TODO)

2. WebSocket Handler (websocket.py)

Sends tool calls to connected clients in real-time
Clients render tools based on tool_name and input parameters

3. Memory Service (memory.py)

TODO

4. Webhook validation

TODO

Testing

TODO

Future Enhancements

Convert existing built-in tools events (mcp-response, confirmation, suggestions) into built-in UI tools for better integration.Those will not use LLM selection.
Tool dependency management (tool A requires tool B)
Dynamic tool plugin loading from external sources
Tool execution feedback loop to LLM
Tool invocation history and analytics

raulcabello · 2026-04-01T08:31:17Z

app/routers/ui_tools.py

+
+        # Fetch the resource from Kubernetes
+        try:
+            resource = api.get_namespaced_custom_object(


we should check the user have permissions to fetch UITools

maybe even consider modifying the CRD from the UI instead of adding this new endpoints? what do you think?

I moved the logic for the CRD updates to the UI and removed the endpoints. Let's postpone the Webhook validation to a dedicate PR.

raulcabello · 2026-04-01T08:31:51Z

app/routers/ui_tools.py

+        api = _init_k8s_client()
+
+        try:
+            current_resource = api.get_namespaced_custom_object(


same here. We should check user can update UITools

raulcabello · 2026-04-01T08:38:35Z

app/routers/ui_tools.py

+            detail="Missing required field: 'tools'",
+        )
+
+    tools = body.get("tools")


all these validation should be moved to a webhook. Otherwise, a user can create an invalid UITool with kubectl

raulcabello · 2026-04-01T08:48:52Z

app/services/agent/root.py

                "continue": "tools",
-                "summarize_conversation": "summarize_conversation",
-                "end": END,
+                "summarize_conversation": "ui_tools",  # Route all endings to ui_tools


summarize_conversation won't always be called as it depends on the number of messages. We should change this to make sure ui_tools is always called.

I fixed the workflow in parent and root nodes.
Now the ui-tools happens always before the summarize switch.

raulcabello · 2026-04-01T08:49:57Z

app/services/agent/root.py

            {
                "continue": "agent",
-                "end": END,
+                "end": "ui_tools",  # Route to ui_tools before summarize


this will be called when human validation is rejected, so we should END

Signed-off-by: Francesco Torchia <francesco.torchia@suse.com>

torchiaf · 2026-04-02T17:01:21Z

Thanks for the review @raulcabello .

I addressed reported issues, please take a look.
We removed the API ui-tools API -> when updating the UI tools the registry is no more reloaded. I created a watcher for the ui-tools-config for this reason. The alternative is to fetch the ui-tools from the cluster on each user request, for this reason the watcher is more efficent.

I still keep this in draft as the tests are TODO.

raulcabello · 2026-04-06T15:51:16Z

chart/agent/templates/crds/ai.cattle.io_uitoolsconfig.yaml

+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: v0.17.1


was this generated with kubebuilder? Can you add the go code like we do for AIAgentConfig https://github.com/rancher/rancher-ai-agent/tree/main/crd-generation?

No it was generated manually. I just pushed the uitoolsconfig_types.go generator.

raulcabello · 2026-04-06T15:55:08Z

app/services/ui_tools/registry.py

+
+        Args:
+            tool: The UITool to register
+            config_name: The config this tool belongs to (default: 'default')


why do we want multiple configs?

This allow us to specify multiple ui-tools configs - not required now, but in the future we want to define different tools and dynamically select them when sending the requests from the UI, basically it's a flexible approach.
It's also possible to define multiple tools for different UIs...

raulcabello · 2026-04-06T15:58:45Z

chart/agent/templates/crds/ai.cattle.io_uitoolsconfig.yaml

+            description: UIToolsConfigSpec defines the desired state of UITools
+            properties:
+              tools:
+                description: List of UI tools


what are the benefits of having a list instead of a single tool per UIToolConfig resource? Have you consider only having one tool per UIToolConfig resource? I beleive that would make it easier to manage all tools

Yes, my first approach was to have a list of UITools crd, this seems reasonable code-wise, but:

The UI needs to check if the ui tools definition has changes before initiating a new chat. When the definition has changes compared with the values in the cluster (this can happen when you install a new UI version), they need to be updated in atomic way.

Fetching only 1 resource it's more efficient.

raulcabello · 2026-04-06T16:01:12Z

app/services/ui_tools/selector.py

+4. Include all REQUIRED fields marked with (REQUIRED) in your input
+5. Optional fields can be omitted if not needed
+6. Output a valid JSON array at the very end with this format:
+[


not sure if all models would be able to create a valid JSON array. We've seen in the past that small models like gpt-oss:20b fails to create a valid json for creating k8s resources.
Have you consider using something like MCP UI to prevent this problem? https://mcpui.dev/guide/introduction

raulcabello · 2026-04-06T16:02:41Z

app/services/ui_tools/selector.py

+                                else:
+                                    logging.debug(f"UI tool '{tool_name}' not found in available tools: {[t.name for t in available_tools]}")
+                except json.JSONDecodeError as e:
+                    logging.debug(f"Failed to parse JSON array: {json_str[:100]}... Error: {e}")


I would log an error if it is not a valid JSON. That would mean the UITools won't work

Done, thanks

raulcabello · 2026-04-07T09:10:18Z

app/services/ui_tools/registry.py

+        self.last_updated = datetime.now()
+        self._register_default_tools()
+
+    def _register_default_tools(self) -> None:


why empty? can we remove it ?

Typo from previous implementation - removed

raulcabello · 2026-04-07T09:11:21Z

app/services/ui_tools/registry.py

+            config=self.config,
+        )
+
+    def validate_tool_input(self, tool_name: str, input_data: Dict[str, Any], config_name: str = None) -> tuple[bool, Optional[str]]:


not used, can we remove it?

Typo from previous implementation - removed

raulcabello · 2026-04-07T09:22:44Z

app/services/agent/base.py

+
+            # Dispatch UI tools before the interrupt, so they're available to the client
+            if config is not None:
+                logging.info(f"Dispatching UI tools before confirmation interrupt")


why do we want to dispatch UI tools before confirming? We will dispatch UI tools at the end, so wouldn't this dispatch it twice?

The confirmation process generates 2 different messages in the UI:

Confirmation ask: The agent sends a confirmation message - <confirmation-response> - to the UI; in this context we are dispatching the ui-tools -> the code you posted is executed.

The user confirms/cancels the action: The agent performs the action and return a message in case of confirmation - There are no duplicates, the agent drops the ongoing LangGraph workflow after confirmation so the UI tools node is not executed. So we are not currently sending any ui-tools after confirmation/cancel, but we should. Notice that there are no ui tools elements in the second confirmation chunk:

Signed-off-by: Francesco Torchia <francesco.torchia@suse.com>

torchiaf requested a review from raulcabello March 30, 2026 07:40

torchiaf changed the title ~~temp~~ UI tools Mar 30, 2026

torchiaf force-pushed the feature-ui-tools branch 3 times, most recently from 9c75205 to 34a54c2 Compare March 31, 2026 10:03

raulcabello reviewed Apr 1, 2026

View reviewed changes

torchiaf added 2 commits April 2, 2026 18:51

Add ui-tools node; Add the ui-tools registry; Add Helm templates

d9f5274

Signed-off-by: Francesco Torchia <francesco.torchia@suse.com>

Implement ui-tools watcher to keep the registry up-to-date

cc335df

Signed-off-by: Francesco Torchia <francesco.torchia@suse.com>

torchiaf force-pushed the feature-ui-tools branch from 8e09136 to cc335df Compare April 2, 2026 16:52

torchiaf requested a review from raulcabello April 2, 2026 17:00

raulcabello reviewed Apr 7, 2026

View reviewed changes

raulcabello mentioned this pull request Apr 7, 2026

Spike: Investigate Agent to Agent #168

Closed

torchiaf added 2 commits April 8, 2026 10:51

Add uitoolsconfig generator

48f7e8d

Signed-off-by: Francesco Torchia <francesco.torchia@suse.com>

Remove typos and improve tools parsing errors logging

bb3ac83

Signed-off-by: Francesco Torchia <francesco.torchia@suse.com>

torchiaf requested a review from raulcabello April 8, 2026 09:37

Conversation

torchiaf commented Mar 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

UI Tools Feature

Overview

Key Features

Architecture

Components

1. Registry (app/services/ui_tools/registry.py)

2. Loader (app/services/ui_tools/loader.py)

3. Selector (app/services/ui_tools/selector.py)

Configuration

Kubernetes CRD: UIToolsConfig

Configuration Fields

Config-Level Settings

Tool-Level Settings

Usage

1. Define Tools in Kubernetes

2. Agent Uses Tools

Implementation Details

Tool Selection Flow

Config-Scoped Namespacing

Revision-Gated Updates

Integration Points

1. Agent State (base.py)

2. WebSocket Handler (websocket.py)

3. Memory Service (memory.py)

4. Webhook validation

Testing

Future Enhancements

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

torchiaf commented Apr 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

torchiaf Apr 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

torchiaf Apr 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

torchiaf commented Mar 30, 2026 •

edited

Loading

1. Registry (`app/services/ui_tools/registry.py`)

2. Loader (`app/services/ui_tools/loader.py`)

3. Selector (`app/services/ui_tools/selector.py`)

torchiaf commented Apr 2, 2026 •

edited

Loading

torchiaf Apr 8, 2026 •

edited

Loading

torchiaf Apr 8, 2026 •

edited

Loading

torchiaf Apr 8, 2026 •

edited

Loading