-
Notifications
You must be signed in to change notification settings - Fork 509
fix: Agents SDK tool calls, prompt responses and other data #1005
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
|
@the-praxs Tests Failing |
Codecov ReportAttention: Patch coverage is 📢 Thoughts on this report? Let us know! |
made them pass |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
This PR updates several telemetry attribute definitions and associated tests to support new semantic conventions for tool calls, prompt responses, and handoffs.
- Updates attribute keys in tests and attribute extraction functions to use new naming (e.g. LLM_OPENAI_RESPONSE_INSTRUCTIONS, gen_ai.prompt.*).
- Adjusts instrumentation logic for the OpenAI Agents SDK with improved logging and span handling.
- Updates version configuration in instrumentation initialization.
Reviewed Changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| tests/unit/instrumentation/openai_core/test_response_attributes.py | Replaces LLM_PROMPTS with LLM_OPENAI_RESPONSE_INSTRUCTIONS in response tests. |
| tests/unit/instrumentation/openai_agents/test_openai_agents_attributes.py | Updates tests for agent and tool attributes to reflect new naming conventions. |
| agentops/semconv/span_attributes.py | Introduces a new semantic attribute for OpenAI response instructions. |
| agentops/instrumentation/openai_agents/instrumentor.py | Improves instrumentation setup with additional flag checks and logging. |
| agentops/instrumentation/openai_agents/exporter.py | Refines span update and error handling logic and logs warnings when creating new spans unexpectedly. |
| agentops/instrumentation/openai_agents/attributes/common.py | Enhances attribute extraction with new functions to process prompt messages and tool call data. |
| agentops/instrumentation/openai/attributes/response.py | Updates attribute mapping to use the new instruction attribute key. |
| agentops/instrumentation/init.py | Adjusts the minimal version requirement from "0.1.0" to "0.0.1". |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
This PR fixes issues related to tool calls, prompt responses, and handoff tracking by introducing new attributes and updating the semantic conventions for the OpenAI Agents SDK instrumentation. Key changes include renaming attributes from LLM_PROMPTS to LLM_OPENAI_RESPONSE_INSTRUCTIONS, updating span attribute extraction for tool calls and prompt messages, and revising instrumentation exporter and instrumentor behavior.
- Updated tests to use new attribute names and verify agent and tool span properties.
- Enhanced exporters and instrumentors with additional logging and span handling logic.
- Adjusted semantic conventions and attribute mappings to support handoffs and improved prompt processing.
Reviewed Changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| tests/unit/instrumentation/openai_core/test_response_attributes.py | Updated assertions to check for LLM_OPENAI_RESPONSE_INSTRUCTIONS attribute. |
| tests/unit/instrumentation/openai_agents/test_openai_agents_attributes.py | Modified attribute assertions for agent/tool span properties. |
| agentops/semconv/span_attributes.py | Added new attribute constant for openai instructions. |
| agentops/instrumentation/openai_agents/instrumentor.py | Introduced instrumentation flags and improved logging. |
| agentops/instrumentation/openai_agents/exporter.py | Revised span export/update logic and logging for ended spans. |
| agentops/instrumentation/openai_agents/attributes/common.py | Extended attribute mappings and prompt message processing. |
| agentops/instrumentation/openai/attributes/response.py | Updated attribute mapping to use the new instructions constant. |
| agentops/instrumentation/init.py | Adjusted minimum version configuration. |
Comments suppressed due to low confidence (1)
agentops/instrumentation/openai_agents/exporter.py:371
- [nitpick] Consider using consistent and less emphatic capitalization (e.g., 'already ended' instead of 'ALREADY ENDED') for logging messages to maintain clarity and readability.
logger.warning( f"[Exporter] SDK span_id: {span_id} (END event) - Attempting to end an ALREADY ENDED span: {span_lookup_key}. Creating a new one instead." )
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
This PR enhances semantic conventions and attribute extraction for OpenAI Agents, adds system instructions and prompt details under gen_ai, and supports tracking handoffs.
- Introduce new semantic constant for OpenAI response instructions.
- Expand attribute mappings and helpers to serialize prompts, messages, and tool calls.
- Update instrumentation logic for idempotent instrument/uninstrument and improve span lifecycle handling.
Reviewed Changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/unit/instrumentation/openai_core/test_response_attributes.py | Updated test to assert new LLM_OPENAI_RESPONSE_INSTRUCTIONS. |
| tests/unit/instrumentation/openai_agents/test_openai_agents_attributes.py | Adjusted tests to check span kind and tool-specific attributes. |
| agentops/semconv/span_attributes.py | Added LLM_OPENAI_RESPONSE_INSTRUCTIONS semantic constant. |
| agentops/instrumentation/openai_agents/instrumentor.py | Made instrumentation idempotent and improved logging. |
| agentops/instrumentation/openai_agents/exporter.py | Enhanced span end logic with warnings and is_already_ended. |
| agentops/instrumentation/openai_agents/attributes/common.py | Added LLMM message extraction helpers and updated mappings. |
| agentops/instrumentation/openai/attributes/response.py | Swapped LLM_PROMPTS mapping to new instruction constant. |
| agentops/instrumentation/init.py | Lowered openai_agents min_version from 0.1.0 to 0.0.1. |
Comments suppressed due to low confidence (3)
tests/unit/instrumentation/openai_agents/test_openai_agents_attributes.py:189
- The new
handoffsandagent_toolsattributes added inget_agent_span_attributesaren’t covered by existing tests. Consider adding assertions forAgentAttributes.HANDOFFSandAgentAttributes.AGENT_TOOLSintest_agent_span_attributes.
# Verify extracted attributes
agentops/instrumentation/openai_agents/attributes/common.py:236
- [nitpick] Building the key as
f"{AgentAttributes.AGENT}.calling_tool.name"can lead to inconsistent naming. Define a dedicated semantic constant for calling tool name (e.g.,AgentAttributes.CALLING_TOOL_NAME) and use that here.
attributes[f"{AgentAttributes.AGENT}.calling_tool.name"] = str(span_data.from_agent)
agentops/semconv/span_attributes.py:80
- New code references
SpanAttributes.AGENTOPS_SPAN_KIND, but there’s no such constant defined in this file. Please addAGENTOPS_SPAN_KIND = "agentops.span.kind"toSpanAttributesto avoid AttributeError.
LLM_OPENAI_RESPONSE_INSTRUCTIONS = "gen_ai.openai.instructions"
Dwij1704
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
W PR
📥 Pull Request
📘 Description
Closes #974
This PR adds new attributes to get the tool calls, save the user prompt and system instructions under the
gen_aisemantic conventions and adds support for trackingHandoff.🧪 Testing
Tested against the Customer Service example from OpenAI Agents SDK.