Skip to content

Conversation

@dot-agi
Copy link
Member

@dot-agi dot-agi commented May 26, 2025

📥 Pull Request

📘 Description
Closes #974
This PR adds new attributes to get the tool calls, save the user prompt and system instructions under the gen_ai semantic conventions and adds support for tracking Handoff.

🧪 Testing
Tested against the Customer Service example from OpenAI Agents SDK.

@dot-agi dot-agi added bug Something isn't working do not merge labels May 26, 2025
@dot-agi dot-agi requested review from Dwij1704 and tcdent and removed request for Dwij1704 May 27, 2025 17:14
@Dwij1704
Copy link
Member

@the-praxs Tests Failing

@codecov
Copy link

codecov bot commented May 27, 2025

Codecov Report

Attention: Patch coverage is 52.50000% with 76 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
...instrumentation/openai_agents/attributes/common.py 48.83% 66 Missing ⚠️
agentops/instrumentation/openai_agents/exporter.py 16.66% 5 Missing ⚠️
...tops/instrumentation/openai_agents/instrumentor.py 79.16% 5 Missing ⚠️

📢 Thoughts on this report? Let us know!

@dot-agi dot-agi self-assigned this May 27, 2025
@dot-agi
Copy link
Member Author

dot-agi commented May 27, 2025

@the-praxs Tests Failing

made them pass

@dot-agi dot-agi requested a review from Copilot May 27, 2025 18:53
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR updates several telemetry attribute definitions and associated tests to support new semantic conventions for tool calls, prompt responses, and handoffs.

  • Updates attribute keys in tests and attribute extraction functions to use new naming (e.g. LLM_OPENAI_RESPONSE_INSTRUCTIONS, gen_ai.prompt.*).
  • Adjusts instrumentation logic for the OpenAI Agents SDK with improved logging and span handling.
  • Updates version configuration in instrumentation initialization.

Reviewed Changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
tests/unit/instrumentation/openai_core/test_response_attributes.py Replaces LLM_PROMPTS with LLM_OPENAI_RESPONSE_INSTRUCTIONS in response tests.
tests/unit/instrumentation/openai_agents/test_openai_agents_attributes.py Updates tests for agent and tool attributes to reflect new naming conventions.
agentops/semconv/span_attributes.py Introduces a new semantic attribute for OpenAI response instructions.
agentops/instrumentation/openai_agents/instrumentor.py Improves instrumentation setup with additional flag checks and logging.
agentops/instrumentation/openai_agents/exporter.py Refines span update and error handling logic and logs warnings when creating new spans unexpectedly.
agentops/instrumentation/openai_agents/attributes/common.py Enhances attribute extraction with new functions to process prompt messages and tool call data.
agentops/instrumentation/openai/attributes/response.py Updates attribute mapping to use the new instruction attribute key.
agentops/instrumentation/init.py Adjusts the minimal version requirement from "0.1.0" to "0.0.1".

@dot-agi dot-agi requested a review from Copilot May 27, 2025 19:00
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR fixes issues related to tool calls, prompt responses, and handoff tracking by introducing new attributes and updating the semantic conventions for the OpenAI Agents SDK instrumentation. Key changes include renaming attributes from LLM_PROMPTS to LLM_OPENAI_RESPONSE_INSTRUCTIONS, updating span attribute extraction for tool calls and prompt messages, and revising instrumentation exporter and instrumentor behavior.

  • Updated tests to use new attribute names and verify agent and tool span properties.
  • Enhanced exporters and instrumentors with additional logging and span handling logic.
  • Adjusted semantic conventions and attribute mappings to support handoffs and improved prompt processing.

Reviewed Changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
tests/unit/instrumentation/openai_core/test_response_attributes.py Updated assertions to check for LLM_OPENAI_RESPONSE_INSTRUCTIONS attribute.
tests/unit/instrumentation/openai_agents/test_openai_agents_attributes.py Modified attribute assertions for agent/tool span properties.
agentops/semconv/span_attributes.py Added new attribute constant for openai instructions.
agentops/instrumentation/openai_agents/instrumentor.py Introduced instrumentation flags and improved logging.
agentops/instrumentation/openai_agents/exporter.py Revised span export/update logic and logging for ended spans.
agentops/instrumentation/openai_agents/attributes/common.py Extended attribute mappings and prompt message processing.
agentops/instrumentation/openai/attributes/response.py Updated attribute mapping to use the new instructions constant.
agentops/instrumentation/init.py Adjusted minimum version configuration.
Comments suppressed due to low confidence (1)

agentops/instrumentation/openai_agents/exporter.py:371

  • [nitpick] Consider using consistent and less emphatic capitalization (e.g., 'already ended' instead of 'ALREADY ENDED') for logging messages to maintain clarity and readability.
logger.warning( f"[Exporter] SDK span_id: {span_id} (END event) - Attempting to end an ALREADY ENDED span: {span_lookup_key}. Creating a new one instead." )

@dot-agi dot-agi requested a review from Copilot May 27, 2025 19:23
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR enhances semantic conventions and attribute extraction for OpenAI Agents, adds system instructions and prompt details under gen_ai, and supports tracking handoffs.

  • Introduce new semantic constant for OpenAI response instructions.
  • Expand attribute mappings and helpers to serialize prompts, messages, and tool calls.
  • Update instrumentation logic for idempotent instrument/uninstrument and improve span lifecycle handling.

Reviewed Changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
tests/unit/instrumentation/openai_core/test_response_attributes.py Updated test to assert new LLM_OPENAI_RESPONSE_INSTRUCTIONS.
tests/unit/instrumentation/openai_agents/test_openai_agents_attributes.py Adjusted tests to check span kind and tool-specific attributes.
agentops/semconv/span_attributes.py Added LLM_OPENAI_RESPONSE_INSTRUCTIONS semantic constant.
agentops/instrumentation/openai_agents/instrumentor.py Made instrumentation idempotent and improved logging.
agentops/instrumentation/openai_agents/exporter.py Enhanced span end logic with warnings and is_already_ended.
agentops/instrumentation/openai_agents/attributes/common.py Added LLMM message extraction helpers and updated mappings.
agentops/instrumentation/openai/attributes/response.py Swapped LLM_PROMPTS mapping to new instruction constant.
agentops/instrumentation/init.py Lowered openai_agents min_version from 0.1.0 to 0.0.1.
Comments suppressed due to low confidence (3)

tests/unit/instrumentation/openai_agents/test_openai_agents_attributes.py:189

  • The new handoffs and agent_tools attributes added in get_agent_span_attributes aren’t covered by existing tests. Consider adding assertions for AgentAttributes.HANDOFFS and AgentAttributes.AGENT_TOOLS in test_agent_span_attributes.
# Verify extracted attributes

agentops/instrumentation/openai_agents/attributes/common.py:236

  • [nitpick] Building the key as f"{AgentAttributes.AGENT}.calling_tool.name" can lead to inconsistent naming. Define a dedicated semantic constant for calling tool name (e.g., AgentAttributes.CALLING_TOOL_NAME) and use that here.
attributes[f"{AgentAttributes.AGENT}.calling_tool.name"] = str(span_data.from_agent)

agentops/semconv/span_attributes.py:80

  • New code references SpanAttributes.AGENTOPS_SPAN_KIND, but there’s no such constant defined in this file. Please add AGENTOPS_SPAN_KIND = "agentops.span.kind" to SpanAttributes to avoid AttributeError.
LLM_OPENAI_RESPONSE_INSTRUCTIONS = "gen_ai.openai.instructions"

Copy link
Member

@Dwij1704 Dwij1704 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

W PR

@Dwij1704 Dwij1704 enabled auto-merge (squash) May 27, 2025 20:37
@Dwij1704 Dwij1704 merged commit fca6472 into main May 27, 2025
10 checks passed
@Dwij1704 Dwij1704 deleted the fix/agents-tool-calls branch May 27, 2025 20:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: OpenAI Agents SDK is not correctly instrumenting calls

3 participants