[Bug]: Summary of Issues with OBSERVE SDK Integration

[techm-2025]

Issue Description

Summary of Issues with OBSERVE SDK Integration

1. Missing Usage Example in README for ioa_observe_sdk

The installation step in the README specifies pip install ioa_observe_sdk, but there are no examples or documentation showing the correct usage, such as from ioa_observe.sdk .... Please consider adding a usage example in the README to help developers understand the correct import and usage pattern.

2. Missing Implementations

Metrics task_delegation and collaboration_success_rate are documented but not implemented.

3. Decorator Limitation and Poor Documentation

Documentation claims all metrics can be captured via decorators, but only three actually work out of the box:

• execution_success_rate
• uptime_and_availability
• failure_rate

Other metrics require passing undocumented attributes to decorators.
Direct code implementation works but requires extensive engineering effort.

4. Incorrect Metric Population (Bug)

token_usage and agent_error_count are incorrectly mapped to tool_call_count and number_active_agents.

5. Incorrect or Unclear Metric Calculation

interpretation_score appears to be miscalculated (possible bug).
Metrics such as interpretation_score and connection_reliability lack documentation and require complex custom logic to capture.

Expected Behavior

• Fixing identified issues will lead to expected behavior.

Affected Version

v1.0.0

Steps to Reproduce

These Issues found during observe integration for the healthcare MAS app.

Checklist

• I have read the contributing guidelines
• I have verified this does not duplicate an existing issue

[91pavan]

Hello, thanks for raising an important issue.

Incorrect Import Path in Documentation

Can you point me to where in the documentation we have this incorrect path specified?

Missing Implemetations
Metrics task_delegation_accuracy and collaboration_success_rate are documented but not implemented.

That is correct. We have this documented in our internal wiki, but no externally. The logic is complicated, and can't be done at the SDK level i.e., before emitting the telemetry to Otel.

Other metrics require passing undocumented attributes to decorators.
Direct code implementation works but requires extensive engineering effort.

May I know which metrics or decorators are you referring to? Some metrics just need the app to be long running like availability or uptime for instance where these are computed at time intervals based on previous runs metadata.

Incorrect Metric Population (Bug)
token_usage and agent_error_count are incorrectly mapped to tool_call_count and number_active_agents.
Incorrect or Unclear Metric Calculation

I couldn't find this metric. Can you point me to the code where you are seeing the disrepency?

interpretation_score appears to be miscalculated (possible bug).

Yes, this is hard to implement as well, maybe we should just remove it altogether.

[techm-2025]

@91pavan thanks for the follow-up. I have clarified specific items below. We can take these discussions offline to triage and close them.

1. It’s not explicitly shown as an incorrect path in the documentation, but the README’s installation step (pip install ioa_observe_sdk) doesn’t align with any example demonstrating how to import the package. For instance, developers might assume usage like from ioa_observe_sdk import ..., which can cause much pain in troubleshooting. Adding a short usage snippet in README would help clarify this. I revised the Issue with this description.

2. Undocumented attributes such as the ones below were added to get metrics collection.
   return {
   "goto": "triage_service_agent",
   "success": True,
   "task_id" : result['id'],
   "context_id":result['contextId'],
   "action":"triage_started"
   }
   Also we have not pursued direct implementation as this was not the suggested path.

3. We’re trying to reproduce this issue for further analysis. Essentially, the undocumented metrics tool_call_count and number_active_agents appeared instead of the documented ones token_usage and agent_error_count, leading to the interpretation that they might be incorrectly mapped.

For issues 2 and 5, you’ve already acknowledged the complexity involved. If there are no plans to implement these, please update the documentation accordingly to avoid confusion.

[91pavan]

Thanks @techm-2025. I've updated the docs with your feedback.

[haboy52589]

it is fixed for the mapping issues