Lazy Loading of LLM Libraries to Prevent Duplicate Instrumentation

[Dwij1704]

We've been facing issues with duplicate LLM instrumentation when multiple instrumentors (openai, crewai etc.) try to wrap the same LLM calls. This leads to redundant spans and potential conflicts in tracing.
Refactoring instrumentation module to be independent of each other is eating me up for the past week :)

The Solution

Me, @the-praxs and @fenilfaldu were discussing approaches to fix tightly coupled behavior and make it independent of each other. Thanks to @the-praxs (🐐) who came up with this solution from V3, we've implemented a lazy loading approach for LLM libraries. Instead of eagerly importing and instrumenting all possible LLM providers and agentic frameworks, we now check sys.modules for direct imports and prioritize Agentic Libararis over providers to eliminate redundant spans.

Why This is Better

• Prevents duplicate instrumentation of the same LLM calls
• Reduces overhead by only instrumenting what's actually used
• More reliable than depending on other instrumentors' LLM call spans
• Cleaner traces with single spans per LLM call

Required Changes to Agentic Libraries

With this change, we need to update our agentic library integrations to record their own LLM spans:

• CrewAI: Need to modify to record spans for its internal LLM calls
• AutoGen: Update to track LLM interactions within agent conversations
• Agents SDK: Add span recording for direct LLM calls

This ensures that even when these frameworks make LLM calls internally, we'll have proper visibility into their operations without duplicate instrumentation.

Current Limitations

There's one known limitation: if a provider/agentic framework is imported in a separate file and then imported into the main file, we miss it. For example:

# llm_caller.py (helper file)
from anthropic import Anthropic  # This import might be missed

def call_anthropic():
    client = Anthropic()
    return client.messages.create(...)

# main.py
from llm_caller import call_anthropic  # We detect this import but miss the underlying Anthropic import

Possible fix is to scan imported modules' source files for additional imports

Testing

I've done some testing with this approach and found it to be more reliable than our previous method. It successfully prevents duplicate spans but we still need to update crewai, autogen and agents instrumentations.

[codecov]

Codecov Report

Attention: Patch coverage is 79.06977% with 18 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
agentops/instrumentation/__init__.py	79.06%	18 Missing ⚠️

📢 Thoughts on this report? Let us know!

[Dwij1704]

Import Hooking Magic:
Hook into Python's import system, when someone imports an LLM provider or agentic library, we automatically detect it and set up the instrumentation.

We handle the tricky case of provider vs agentic library conflicts. For example, if someone's using CrewAI (which uses litellm, which uses OpenAI under the hood), we automatically uninstrument the direct OpenAI provider to avoid double-counting.

The implementation uses a singleton InstrumentationManager that:

• Monitors Python imports using a custom import hook
• Maintains a list of active instrumentors
• Handles the provider vs agentic library dance

@the-praxs @tcdent @areibman Thoughts? checkout the PR, I have made changes with this approach and it works!

[dot-agi]

The dance of instrumentation is a tedious one.

But I will review thy PR. Time for some Easter eggs in the codebase :)

[tcdent]

Does this now incorporate a fix for modules imported in sub-modules? Super common use case to have a project span multiple modules.

[tcdent]

Just left a couple comments on code style.

[Dwij1704]

Does this now incorporate a fix for modules imported in sub-modules? Super common use case to have a project span multiple modules.

Yes it does, at first it checks using sys.modules to instrument existing direct imports, then relies on import hook to instrument indirect imports

[Dwij1704]

@tcdent Try this out

#Example.py

from dotenv import load_dotenv
load_dotenv()

import os
from openai import OpenAI
import agentops

agentops.init(os.getenv("AGENTOPS_API_KEY"))
from llm_caller import generate_content
client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))

print("Hello, world!")
res1 = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Hello, world!"}]
)
print(res1.choices[0].message.content)


response = generate_content("Hello, world!")
print(response)

#llm_caller.py

from anthropic import Anthropic
from dotenv import load_dotenv
import os
load_dotenv()
client = Anthropic(api_key=os.getenv("ANTHROPIC_API_KEY"))
def generate_content(prompt):
    # Make a completion request - AgentOps will track it automatically
    message = client.messages.create(
        model="claude-3-opus-20240229",
        max_tokens=1000,
        messages=[
            {"role": "user", "content": prompt}
        ]
    )

    return message