DSPy Callback Handler

[michi-okahata]

📥 Pull Request

📘 Description
Addresses #88
Implemented callbacks, instrumentation is outstanding.

🧪 Testing
Tested on toy examples, documentation is outstanding.

---

EntelligenceAI PR Summary

This PR adds DSPy integration and tracing support to AgentOps, expands test coverage, and removes a deprecated span kind.

• Introduced DSPyCallbackHandler and integration module (agentops/integration/callbacks/dspy/)
• Added DSPy calculator example (examples/dspy/dspy_calculator.py)
• Updated integration test workflow for new example (.github/workflows/examples-integration-test.yml)
• Removed 'TEAM' span kind (agentops/semconv/span_kinds.py)

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[areibman]

Review Summary

🏷️ Draft Comments (6)

Skipped posting 6 drafted comments based on your review threshold. Feel free to update them here.

.github/workflows/examples-integration-test.yml (1)

111-111: examples/dspy/dspy_calculator.py is added to the test matrix, but if this file does not exist or is not executable, the workflow will fail for all PRs/merges, causing false negatives.

Scores:

• Production Impact: 4
• Fix Specificity: 4
• Urgency Impact: 4
• Total Score: 12

Reason for filtering: This is a valid concern about CI/CD reliability. The comment correctly identifies that line 111 adds a DSPy example to the test matrix, and if this file doesn't exist or isn't executable, it will cause the entire workflow to fail. This is a legitimate infrastructure issue that could block all PRs and merges.

Analysis: The comment accurately identifies line 111 where the DSPy example is added to the matrix. While we cannot verify if the file exists from the provided context, the concern is technically sound - GitHub Actions workflows will fail if they try to execute non-existent files. High production impact because it could block all CI/CD processes. High fix specificity with clear actionable guidance. High urgency because broken CI blocks development workflow.

---

agentops/integration/callbacks/dspy/callback.py (2)

144-146: self.session_span is overwritten in _create_span if not set, causing subsequent spans to have incorrect parent and breaking session trace hierarchy.

Scores:

• Production Impact: 0
• Fix Specificity: 0
• Urgency Impact: 0
• Total Score: 0

Reason for filtering: While the bug description correctly identifies a real issue with self.session_span being overwritten, the commitable suggestion would break the existing code flow. The suggestion doesn't account for the existing else block at lines 147-150 that also sets parent_ctx, creating a logical inconsistency where parent_ctx would be set twice in different ways.

Analysis: The comment identifies a legitimate architectural issue with session span management, but the proposed fix is incomplete and would introduce new bugs. The suggestion fails to consider the full context of the existing conditional logic structure.

---

200-200: span.update_name('test') in _end_span changes the span's name to 'test' before ending, corrupting trace data and making span names incorrect in production.

Scores:

• Production Impact: 4
• Fix Specificity: 5
• Urgency Impact: 3
• Total Score: 12

Reason for filtering: This comment correctly identifies a real bug where span.update_name('test') corrupts trace data by changing span names to 'test' before ending. The line numbers are accurate, the technical analysis is correct, and the suggested fix is safe and appropriate.

Analysis: High production impact (4) because corrupted span names make distributed tracing difficult to debug and analyze. Perfect fix specificity (5) with a clear, direct replacement that removes the problematic call. Medium urgency (3) as this affects observability but doesn't crash the system - should be fixed soon but not critical.

---

agentops/semconv/span_kinds.py (1)

13-46: TEAM span kind was removed from both AgentOpsSpanKindValues and SpanKind, which will break any code relying on TEAM for span classification and cause runtime errors (e.g., AttributeError or KeyError).

Scores:

• Production Impact: 0
• Fix Specificity: 0
• Urgency Impact: 0
• Total Score: 0

Reason for filtering: This comment should be removed because it reports a non-existent issue. The comment claims that 'TEAM' span kind was removed from both classes, but examining the structured_context shows there is no 'TEAM' member present in either AgentOpsSpanKindValues or SpanKind classes. The line numbers 13-46 span nearly the entire file but don't point to any specific location where TEAM should exist. The commitable_suggestion attempts to add TEAM members that were never there in the first place, making this a false positive based on incorrect assumptions about the code's previous state.

Analysis: The comment fails all three verification checks: (1) Line numbers are too broad and don't correspond to any actual TEAM-related code, (2) The suggestion would add new functionality rather than fix existing code, and (3) The technical premise is incorrect - there is no evidence that TEAM was ever part of these classes. This appears to be a hallucinated issue rather than a real code problem.

---

examples/dspy/dspy_calculator.py (2)

31-31: agentops.validate_trace_spans(trace_context=None) will always raise a ValueError because neither trace_id nor a valid trace_context is provided, violating the function's contract and causing a runtime failure.

Scores:

• Production Impact: 0
• Fix Specificity: 0
• Urgency Impact: 0
• Total Score: 0

Reason for filtering: The comment makes a very specific claim that agentops.validate_trace_spans(trace_context=None) will 'always raise a ValueError', but this cannot be verified from the provided context. The function might legitimately accept None as a valid parameter value or handle it gracefully. Without access to the actual agentops library implementation, this assertion about the function's contract and behavior is unverifiable and potentially incorrect.

Analysis: This comment should be removed because it makes unverifiable claims about external library behavior. The assertion that passing trace_context=None will 'always' cause a ValueError is too definitive without being able to inspect the actual function implementation. The code appears to be a working example, which suggests the current usage may be intentional and functional.

---

16-16: dspy.LM('openai/gpt-4o-mini', temperature=0.5) is instantiated on every run, causing repeated model loading and network initialization, which can significantly slow down startup and increase resource usage for frequent executions.

Scores:

• Production Impact: 0
• Fix Specificity: 0
• Urgency Impact: 0
• Total Score: 0

Reason for filtering: The commitable suggestion would break the code. The suggestion if 'lm' not in globals(): lm = dspy.LM(...) would create a local variable 'lm' inside the if block, making it inaccessible to the subsequent dspy.configure(lm=lm, callbacks=[handler]) call on line 17, which would result in a NameError.

Analysis: While the performance concern about repeated model instantiation is technically valid, the proposed fix is fundamentally flawed. The suggested code would create a scoping issue where 'lm' becomes undefined outside the if block, breaking the existing functionality. This violates the safety check requirement that suggestions must not break the code when applied.

---

[areibman]

Walkthrough

This PR introduces comprehensive DSPy integration into AgentOps, enabling detailed tracing and observability for DSPy workflows. It adds a new DSPy callback handler, updates the integration test workflow to cover a new DSPy example, and provides a sample calculator script demonstrating the integration. Additionally, it removes the deprecated 'TEAM' span kind from the span kinds definition, streamlining the codebase.

Changes

File(s)	Summary
.github/workflows/examples-integration-test.yml	Added 'DSPy ReAct Agent' entry to the test matrix for integration testing of the new DSPy example.
agentops/integration/callbacks/dspy/init.py	New module initializer for DSPy integration; imports and exports DSPyCallbackHandler.
agentops/integration/callbacks/dspy/callback.py	Introduced DSPyCallbackHandler class implementing DSPy callback hooks for OpenTelemetry tracing and AgentOps integration.
agentops/semconv/span_kinds.py	Removed 'TEAM' span kind from AgentOpsSpanKindValues Enum and SpanKind class.
examples/dspy/dspy_calculator.py	Added DSPy-based calculator example demonstrating LLM, custom tool, and AgentOps tracing integration.

Sequence Diagram

This diagram shows the interactions between components:

sequenceDiagram
    title DSPy ReAct Agent Integration Flow

    actor User
    participant "Documentation System" as Docs
    participant "CI/CD Pipeline" as CI
    participant "Examples Registry" as Registry
    participant "OpenAI Agents" as OpenAI
    participant "DSPy ReAct Agent" as DSPy

    Note over User, DSPy: New DSPy ReAct Agent example added to the system

    User->>Docs: Access documentation
    Docs->>Registry: Request available examples
    Registry-->>Docs: Return example list (including new DSPy example)
    
    alt User selects DSPy example
        User->>DSPy: Run DSPy ReAct Agent example
        activate DSPy
        Note right of DSPy: dspy_calculator.py execution
        DSPy-->>User: Return calculation results
        deactivate DSPy
    else User selects OpenAI example
        User->>OpenAI: Run OpenAI Agent example
        activate OpenAI
        Note right of OpenAI: agents_tools.py or customer_service_agent.py
        OpenAI-->>User: Return agent response
        deactivate OpenAI
    end
    
    Note over CI, Registry: CI/CD Pipeline Integration
    User->>CI: Trigger build/test
    CI->>Registry: Load example configurations
    Registry-->>CI: Return updated example list
    CI->>DSPy: Execute DSPy example tests
    DSPy-->>CI: Return test results
    CI-->>User: Report build status

Loading

▶️ ⚡ AI Code Reviews for VS Code, Cursor, Windsurf
Install the extension

Note for Windsurf

Please change the default marketplace provider to the following in the windsurf settings:

Marketplace Extension Gallery Service URL: https://marketplace.visualstudio.com/_apis/public/gallery

Marketplace Gallery Item URL: https://marketplace.visualstudio.com/items

Entelligence.ai can learn from your feedback. Simply add 👍 / 👎 emojis to teach it your preferences. More shortcuts below

Emoji Descriptions:

• ⚠️ Potential Issue - May require further investigation.
• 🔒 Security Vulnerability - Fix to ensure system safety.
• 💻 Code Improvement - Suggestions to enhance code quality.
• 🔨 Refactor Suggestion - Recommendations for restructuring code.
• ℹ️ Others - General comments and information.

Interact with the Bot:

• Send a message or request using the format:
  @entelligenceai + *your message*

Example: @entelligenceai Can you suggest improvements for this code?

• Help the Bot learn by providing feedback on its responses.
  @entelligenceai + *feedback*

Example: @entelligenceai Do not comment on `save_auth` function !

Also you can trigger various commands with the bot by doing
@entelligenceai command

The current supported commands are

1. config - shows the current config
2. retrigger_review - retriggers the review

More commands to be added soon.

[areibman]

LGTM 👍