Skip to content

Suppport computer use and sync with new typespec #36101

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Suppport computer use and sync with new typespec #36101

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
4811b87
sync with new type spec
bobogogo Sep 27, 2025
354feed
add support for computer use
bobogogo Oct 1, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions sdk/ai/ai-agents/CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,11 @@
# Release History

## 1.2.0-beta.3 (2025-10-02)

### Features Added

- Add `ToolUtility.createComputerUseTool` to support computer use tool in agent

## 1.2.0-beta.2 (2025-09-26)

### Features Added
Expand Down
Binary file added sdk/ai/ai-agents/data/cua_screenshot.jpg
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added sdk/ai/ai-agents/data/cua_screenshot_next.jpg
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
2 changes: 1 addition & 1 deletion sdk/ai/ai-agents/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "@azure/ai-agents",
"version": "1.2.0-beta.2",
"version": "1.2.0-beta.3",
"description": "Azure AI Agents client library.",
"engines": {
"node": ">=20.0.0"
Expand Down
184 changes: 175 additions & 9 deletions sdk/ai/ai-agents/review/ai-agents-node.api.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -107,7 +107,7 @@ export interface AgentsNamedToolChoice {
}

// @public
export type AgentsNamedToolChoiceType = "function" | "code_interpreter" | "file_search" | "bing_grounding" | "fabric_dataagent" | "sharepoint_grounding" | "azure_ai_search" | "bing_custom_search" | "connected_agent" | "deep_research" | "mcp";
export type AgentsNamedToolChoiceType = "function" | "code_interpreter" | "file_search" | "bing_grounding" | "fabric_dataagent" | "sharepoint_grounding" | "azure_ai_search" | "bing_custom_search" | "connected_agent" | "deep_research" | "mcp" | "computer_use_preview";

// @public
export interface AgentsResponseFormat {
Expand Down Expand Up @@ -276,6 +276,14 @@ export interface BrowserAutomationToolParameters {
connection: BrowserAutomationToolConnectionParameters;
}

// @public
export interface ClickAction extends ComputerUseAction {
button: MouseButton;
type: "click";
x: number;
y: number;
}

// @public
export interface CodeInterpreterToolDefinition extends ToolDefinition {
type: "code_interpreter";
Expand All @@ -287,6 +295,44 @@ export interface CodeInterpreterToolResource {
fileIds?: string[];
}

// @public
export interface ComputerScreenshot {
fileId?: string;
imageUrl?: string;
type: "computer_screenshot";
}

// @public
export interface ComputerToolOutput extends StructuredToolOutput {
acknowledgedSafetyChecks?: SafetyCheck[];
output: ComputerScreenshot;
type: "computer_call_output";
}

// @public
export interface ComputerUseAction {
type: string;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We usually use kind for these types of discriminators in TS - is this autogenerated or hand authored?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it is generated

}

// @public
export type ComputerUseActionUnion = ClickAction | DoubleClickAction | DragAction | KeyPressAction | MoveAction | ScreenshotAction | ScrollAction | TypeAction | WaitAction | ComputerUseAction;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does the base ComputerUseAction belong here? And if we have to add type to every object in the union do we even need the base type? What value does it add?

I would say this main type should be ComputerUseAction (without the union) and the ComputerUseAction interface can be removed. If each member of the union has a type or kind discriminator typescript can narrow the types down without any sort of inheritance hierarchy


// @public
export type ComputerUseEnvironment = "windows" | "mac" | "linux" | "browser";

// @public
export interface ComputerUseToolDefinition extends ToolDefinition {
computerUsePreview: ComputerUseToolParameters;
type: "computer_use_preview";
}

// @public
export interface ComputerUseToolParameters {
displayHeight: number;
displayWidth: number;
environment: ComputerUseEnvironment;
}

// @public
export interface ConnectedAgentDetails {
description: string;
Expand Down Expand Up @@ -314,6 +360,12 @@ export type ContinuablePage<TElement, TPage = TElement[]> = TPage & {
continuationToken?: string;
};

// @public
export interface CoordinatePoint {
x: number;
y: number;
}

// @public
export interface CreateAgentOptionalParams extends OperationOptions {
description?: string | null;
Expand Down Expand Up @@ -372,6 +424,19 @@ export enum DoneEvent {
Done = "done"
}

// @public
export interface DoubleClickAction extends ComputerUseAction {
type: "double_click";
x: number;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not coordinates: CoordinatePoint ?

y: number;
}

// @public
export interface DragAction extends ComputerUseAction {
path: CoordinatePoint[];
type: "drag";
}

// @public
export enum ErrorEvent {
Error = "error"
Expand Down Expand Up @@ -522,6 +587,12 @@ export function isOutputOfType<T extends {
type: string;
}>(output: RequiredAction | RequiredToolCall | ToolDefinitionUnion, type: string): output is T;

// @public
export interface KeyPressAction extends ComputerUseAction {
keys: string[];
type: "keypress";
}

// @public
export enum KnownVersions {
V1 = "v1",
Expand Down Expand Up @@ -890,6 +961,16 @@ export interface MicrosoftFabricToolDefinition extends ToolDefinition {
type: "fabric_dataagent";
}

// @public
export type MouseButton = "left" | "right" | "wheel" | "back" | "forward";

// @public
export interface MoveAction extends ComputerUseAction {
type: "move";
x: number;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not use coordinates: CoordinatePoint ?

y: number;
}

// @public
export interface OpenApiAnonymousAuthDetails extends OpenApiAuthDetails {
type: "anonymous";
Expand Down Expand Up @@ -985,6 +1066,18 @@ export interface RequiredAction {
// @public
export type RequiredActionUnion = SubmitToolOutputsAction | SubmitToolApprovalAction | RequiredAction;

// @public
export interface RequiredComputerUseToolCall extends RequiredToolCall {
computerUsePreview: RequiredComputerUseToolCallDetails;
type: "computer_use_preview";
}

// @public
export interface RequiredComputerUseToolCallDetails {
action: ComputerUseActionUnion;
pendingSafetyChecks: SafetyCheck[];
}

// @public
export interface RequiredFunctionToolCall extends RequiredToolCall {
function: RequiredFunctionToolCallDetails;
Expand Down Expand Up @@ -1012,7 +1105,7 @@ export interface RequiredToolCall {
}

// @public
export type RequiredToolCallUnion = RequiredFunctionToolCall | RequiredMcpToolCall | RequiredToolCall;
export type RequiredToolCallUnion = RequiredFunctionToolCall | RequiredMcpToolCall | RequiredComputerUseToolCall | RequiredToolCall;

// @public
export type ResponseFormat = "text" | "json_object";
Expand Down Expand Up @@ -1091,15 +1184,15 @@ export interface RunsOperations {
createThreadAndRun: (assistantId: string, options?: CreateThreadAndRunOptionalParams) => AgentRunResponse;
get: (threadId: string, runId: string, options?: RunsGetRunOptionalParams) => Promise<ThreadRun>;
list: (threadId: string, options?: RunsListRunsOptionalParams) => PagedAsyncIterableIterator<ThreadRun>;
submitToolOutputs: (threadId: string, runId: string, toolOutputs: ToolOutput[], options?: RunsSubmitToolOutputsToRunOptionalParams) => AgentRunResponse;
submitToolOutputs: (threadId: string, runId: string, toolOutputs: StructuredToolOutputUnion[], options?: RunsSubmitToolOutputsToRunOptionalParams) => AgentRunResponse;
update: (threadId: string, runId: string, options?: RunsUpdateRunOptionalParams) => Promise<ThreadRun>;
}

// @public
export interface RunsSubmitToolOutputsToRunOptionalParams extends OperationOptions {
stream?: boolean | null;
toolApprovals?: ToolApproval[];
toolOutputs?: ToolOutput[];
toolOutputs?: StructuredToolOutputUnion[];
}

// @public
Expand Down Expand Up @@ -1205,6 +1298,20 @@ export interface RunStepCompletionUsage {
totalTokens: number;
}

// @public
export interface RunStepComputerUseToolCall extends RunStepToolCall {
computerUsePreview: RunStepComputerUseToolCallDetails;
type: "computer_use_preview";
}

// @public
export interface RunStepComputerUseToolCallDetails {
acknowledgedSafetyChecks?: SafetyCheck[];
action: ComputerUseActionUnion;
output: ComputerScreenshot;
pendingSafetyChecks: SafetyCheck[];
}

// @public
export interface RunStepConnectedAgent {
agentId?: string;
Expand Down Expand Up @@ -1301,6 +1408,20 @@ export interface RunStepDeltaCodeInterpreterToolCall extends RunStepDeltaToolCal
type: "code_interpreter";
}

// @public
export interface RunStepDeltaComputerUseDetails {
acknowledgedSafetyChecks?: SafetyCheck[];
action?: ComputerUseActionUnion;
output?: ComputerScreenshot;
pendingSafetyChecks?: SafetyCheck[];
}

// @public
export interface RunStepDeltaComputerUseToolCall extends RunStepDeltaToolCall {
computerUsePreview?: RunStepDeltaComputerUseDetails;
type: "computer_use_preview";
}

// @public
export interface RunStepDeltaConnectedAgentToolCall extends RunStepDeltaToolCall {
connectedAgent: RunStepConnectedAgent;
Expand Down Expand Up @@ -1408,7 +1529,7 @@ export interface RunStepDeltaToolCallObject extends RunStepDeltaDetail {
}

// @public
export type RunStepDeltaToolCallUnion = RunStepDeltaMcpToolCall | RunStepDeltaOpenAPIToolCall | RunStepDeltaConnectedAgentToolCall | RunStepDeltaFunctionToolCall | RunStepDeltaFileSearchToolCall | RunStepDeltaCodeInterpreterToolCall | RunStepDeltaBingGroundingToolCall | RunStepDeltaCustomBingGroundingToolCall | RunStepDeltaAzureFunctionToolCall | RunStepDeltaDeepResearchToolCall | RunStepDeltaAzureAISearchToolCall | RunStepDeltaMicrosoftFabricToolCall | RunStepDeltaSharepointToolCall | RunStepDeltaToolCall;
export type RunStepDeltaToolCallUnion = RunStepDeltaMcpToolCall | RunStepDeltaOpenAPIToolCall | RunStepDeltaConnectedAgentToolCall | RunStepDeltaFunctionToolCall | RunStepDeltaFileSearchToolCall | RunStepDeltaCodeInterpreterToolCall | RunStepDeltaBingGroundingToolCall | RunStepDeltaCustomBingGroundingToolCall | RunStepDeltaAzureFunctionToolCall | RunStepDeltaDeepResearchToolCall | RunStepDeltaAzureAISearchToolCall | RunStepDeltaComputerUseToolCall | RunStepDeltaMicrosoftFabricToolCall | RunStepDeltaSharepointToolCall | RunStepDeltaToolCall;

// @public
export interface RunStepDetails {
Expand Down Expand Up @@ -1553,7 +1674,7 @@ export interface RunStepToolCallDetails extends RunStepDetails {
}

// @public
export type RunStepToolCallUnion = RunStepCodeInterpreterToolCall | RunStepFileSearchToolCall | RunStepBingGroundingToolCall | RunStepAzureAISearchToolCall | RunStepBrowserAutomationToolCall | RunStepMcpToolCall | RunStepSharepointToolCall | RunStepMicrosoftFabricToolCall | RunStepBingCustomSearchToolCall | RunStepAzureFunctionToolCall | RunStepFunctionToolCall | RunStepOpenAPIToolCall | RunStepDeepResearchToolCall | RunStepConnectedAgentToolCall | RunStepToolCall;
export type RunStepToolCallUnion = RunStepCodeInterpreterToolCall | RunStepFileSearchToolCall | RunStepBingGroundingToolCall | RunStepAzureAISearchToolCall | RunStepBrowserAutomationToolCall | RunStepMcpToolCall | RunStepComputerUseToolCall | RunStepSharepointToolCall | RunStepMicrosoftFabricToolCall | RunStepBingCustomSearchToolCall | RunStepAzureFunctionToolCall | RunStepFunctionToolCall | RunStepOpenAPIToolCall | RunStepDeepResearchToolCall | RunStepConnectedAgentToolCall | RunStepToolCall;

// @public
export type RunStepType = "message_creation" | "tool_calls" | "activities";
Expand All @@ -1577,6 +1698,27 @@ export interface RunsUpdateRunOptionalParams extends OperationOptions {
metadata?: Record<string, string> | null;
}

// @public
export interface SafetyCheck {
code?: string;
id: string;
message?: string;
}

// @public
export interface ScreenshotAction extends ComputerUseAction {
type: "screenshot";
}

// @public
export interface ScrollAction extends ComputerUseAction {
scrollX: number;
scrollY: number;
type: "scroll";
x: number;
y: number;
}

// @public
export interface SharepointGroundingToolParameters {
connectionList?: ToolConnection[];
Expand All @@ -1588,6 +1730,15 @@ export interface SharepointToolDefinition extends ToolDefinition {
type: "sharepoint_grounding";
}

// @public
export interface StructuredToolOutput {
toolCallId?: string;
type: string;
}

// @public
export type StructuredToolOutputUnion = ToolOutput | ComputerToolOutput | StructuredToolOutput;

// @public
export interface SubmitToolApprovalAction extends RequiredAction {
submitToolApproval: SubmitToolApprovalDetails;
Expand Down Expand Up @@ -1736,12 +1887,12 @@ export interface ToolDefinition {
}

// @public
export type ToolDefinitionUnion = CodeInterpreterToolDefinition | FileSearchToolDefinition | FunctionToolDefinition | BingGroundingToolDefinition | MicrosoftFabricToolDefinition | SharepointToolDefinition | AzureAISearchToolDefinition | OpenApiToolDefinition | BingCustomSearchToolDefinition | ConnectedAgentToolDefinition | DeepResearchToolDefinition | MCPToolDefinition | AzureFunctionToolDefinition | BrowserAutomationToolDefinition | ToolDefinition;
export type ToolDefinitionUnion = CodeInterpreterToolDefinition | FileSearchToolDefinition | FunctionToolDefinition | BingGroundingToolDefinition | MicrosoftFabricToolDefinition | SharepointToolDefinition | AzureAISearchToolDefinition | OpenApiToolDefinition | BingCustomSearchToolDefinition | ConnectedAgentToolDefinition | DeepResearchToolDefinition | MCPToolDefinition | ComputerUseToolDefinition | AzureFunctionToolDefinition | BrowserAutomationToolDefinition | ToolDefinition;

// @public
export interface ToolOutput {
export interface ToolOutput extends StructuredToolOutput {
output?: string;
toolCallId?: string;
type: "function_call_output";
}

// @public
Expand Down Expand Up @@ -1806,6 +1957,10 @@ export class ToolUtility {
definition: CodeInterpreterToolDefinition;
resources: ToolResources;
};
// (undocumented)
static createComputerUseTool(displayWidth: number, displayHeight: number, env: ComputerUseEnvironment): {
definition: ComputerUseToolDefinition;
};
static createConnectedAgentTool(id: string, name: string, description: string): {
definition: ConnectedAgentToolDefinition;
};
Expand Down Expand Up @@ -1838,6 +1993,12 @@ export interface TruncationObject {
// @public
export type TruncationStrategy = "auto" | "last_messages";

// @public
export interface TypeAction extends ComputerUseAction {
text: string;
type: "type";
}

// @public
export interface UpdateAgentOptionalParams extends OperationOptions {
description?: string | null;
Expand Down Expand Up @@ -2126,6 +2287,11 @@ export interface VectorStoreStaticChunkingStrategyResponse extends VectorStoreCh
// @public
export type VectorStoreStatus = "expired" | "in_progress" | "completed";

// @public
export interface WaitAction extends ComputerUseAction {
type: "wait";
}

// (No @packageDocumentation comment for this package)

```
Loading
Loading