WIP: [Agents] Computer Use Tool #42483
Conversation
glharper
requested review from
dargilco,
jhakulin,
trangevi,
nick863 and
howieleung
as code owners
There was a problem hiding this comment.
Pull Request Overview
This PR implements support for a Computer Use tool in the Azure AI Agents SDK, adding functionality for agents to perform computer automation actions like taking screenshots, clicking, typing, and other interactions. The changes update the generated models and operations to support this new tool type.
Key changes:
- Add Computer Use tool support with various action types (click, type, screenshot, etc.)
- Update error handling to use HttpResponse objects directly instead of pre-parsed JSON
- Add comprehensive model definitions for computer automation actions and safety checks
Reviewed Changes
Copilot reviewed 10 out of 10 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| tsp-location.yaml | Updates commit hash to latest version for type spec generation |
| sample_agents_computer_use.py | Adds comprehensive sample demonstrating Computer Use tool functionality |
| _operations.py | Fixes error deserialization to pass HttpResponse objects instead of JSON |
| _patch.py | Adds ComputerUseTool class implementation with proper tool definitions |
| _models.py | Extensive additions for Computer Use actions, tool outputs, and safety checks |
| _enums.py | Adds enums for computer environments and mouse button types |
| init.py | Exports all new Computer Use related classes and enums |
| aio/_operations.py | Applies same error handling fixes as sync version |
| _utils/model_base.py | Updates failsafe deserialization to handle HttpResponse objects |
| apiview-properties.json | Maps new model classes to their API view representations |
Comments suppressed due to low confidence (1)
| @@ -1368,7 +1368,7 @@ def create( | |||
| except (StreamConsumedError, StreamClosedError): | |||
| pass | |||
| map_error(status_code=response.status_code, response=response, error_map=error_map) | |||
| error = _failsafe_deserialize(_models.AgentV1Error, response.json()) | |||
| error = _failsafe_deserialize(_models.AgentV1Error, response) | |||
There was a problem hiding this comment.
The _failsafe_deserialize function now expects an HttpResponse object, but the original implementation expected response.json(). This change could break existing functionality if the _failsafe_deserialize function signature hasn't been updated accordingly.
| error = _failsafe_deserialize(_models.AgentV1Error, response) | |
| error = _failsafe_deserialize(_models.AgentV1Error, response.json()) |
Copilot uses AI. Check for mistakes.
| """ | ||
| return ToolResources() | ||
|
|
||
| def execute(self, tool_call: Any) -> Any: # noqa: D401 - client-side execution not applicable |
There was a problem hiding this comment.
[nitpick] The execute method has a noqa comment indicating client-side execution is not applicable, but the method body is empty with just 'pass'. Consider adding a docstring or comment explaining why this method is not implemented or what the expected behavior should be.
| def execute(self, tool_call: Any) -> Any: # noqa: D401 - client-side execution not applicable | |
| def execute(self, tool_call: Any) -> Any: # noqa: D401 - client-side execution not applicable | |
| """ | |
| This method is intentionally not implemented because client-side execution is not applicable | |
| for ComputerUseTool. The agent does not execute this tool on the client side. | |
| """ |
Copilot uses AI. Check for mistakes.
| module: typing.Optional[str] = None, | ||
| rf: typing.Optional["_RestField"] = None, | ||
| format: typing.Optional[str] = None, | ||
| ) -> typing.Any: | ||
| try: | ||
| return _deserialize(deserializer, value, module, rf, format) | ||
| return _deserialize(deserializer, response.json(), module, rf, format) |
There was a problem hiding this comment.
The function is calling response.json() but needs to handle potential exceptions if the response doesn't contain valid JSON. Consider adding error handling for cases where the response might not be JSON parseable.
| return _deserialize(deserializer, response.json(), module, rf, format) | |
| try: | |
| json_data = response.json() | |
| except (ValueError, json.JSONDecodeError) as e: | |
| raise DeserializationError() from e | |
| return _deserialize(deserializer, json_data, module, rf, format) |
Copilot uses AI. Check for mistakes.
API Change CheckAPIView identified API level changes in this PR and created the following API reviews |
Description
Please add an informative description that covers that changes made by the pull request and link all relevant issues.
If an SDK is being regenerated based on a new API spec, a link to the pull request containing these API spec changes should be included above.
All SDK Contribution checklist:
General Guidelines and Best Practices
Testing Guidelines