Ability to report "metadata" to open telemetry per agent run

[galfrishman]

Description

The use-case in hand is multiple relatively short agent runs, each run should be tagged with some "business attributes" so later traces can be filtered/searched by them, for example a run associated with specific customer, specific Jira issue, etc.

I think the natural place to provide this information is as an argument to agent.run* methods.
Thanks

References

No response

[DouweM]

@galfrishman Thanks, good idea. We don't currently have any instrumentation-specific fields on the run/iter methods, and adding one implies adding it in 10 or so places, but we do have instrument=InstrumentationSettings(...) on Agent, so before possibly deciding an agent.run argument makes most sense, I'd like to consider if it could make sense to leverage InstrumentationSettings for this feature. Potentially, it could take an span_attributes argument that in addition to a dict could take take a callable that takes RunContext, just like the agent tool and instructions functions. The business attributes could then be stored on deps and transformed into span attributes. Would that work for you? One advantage is that you wouldn't need to remember to do this on every run-call, just specify it once on the agent and make sure your deps type requires the relevant fields to be set.

[galfrishman]

@DouweM I guess that could technically work, but from software engineering perspective it sounds like it doesn't fit naturally into the Dependencies abstraction. In my mind, ideally the code that implements the agent shouldn't even know it is being instrumented, let alone have access to the instrumentation settings or have an ability to modify or override them. In addition, the dynamic nature of callables may inject unnecessary complexities into the instrumentation layer. I wouldn't want end users to be troubled with "did it run and there is an issue with tracing or did it not run"? the tracing should just work once configured.

I understand it may be a big lift to add support in 10+ places, but for the sake of the project I'd prefer that it would have the right architecture and abstractions in place rather than one more feature with a leaky abstraction.

[DouweM]

for the sake of the project I'd prefer that it would have the right architecture and abstractions in place rather than one more feature with a leaky abstraction.

@galfrishman I agree completely.

In my mind, ideally the code that implements the agent shouldn't even know it is being instrumented, let alone have access to the instrumentation settings or have an ability to modify or override them.

Currently instrument: InstrumentationSettings is only an argument on Agent, but I agree it should be on agent.run (and derivatives) as well as the appropriate settings are often determined at call site. I'd then be inclined to make attributes (or agent_run_attributes or something else) an attribute on that object.

In addition, the dynamic nature of callables may inject unnecessary complexities into the instrumentation layer. I wouldn't want end users to be troubled with "did it run and there is an issue with tracing or did it not run"? the tracing should just work once configured.

I agree the tracing should just work; you're thinking about the callable potentially raising an exception?

A RunContext-based callable would have the advantage of being able to be called at the end to get attributes from the agent run result as well, rather than the attributes needing to be known ahead of the agent run. We could potentially implement that with a different hook that takes AgentRunResult, as RunContext does not currently actually contain the output.

[galfrishman]

I agree the tracing should just work; you're thinking about the callable potentially raising an exception?

Either an exception or just failure to associate the appropriate business attributes so the trace ends up in a place no one is looking at.

A RunContext-based callable would have the advantage of being able to be called at the end to get attributes from the agent run result as well, rather than the attributes needing to be known ahead of the agent run. We could potentially implement that with a different hook that takes AgentRunResult, as RunContext does not currently actually contain the output.

I agree structured tracing of certain outputs may be useful but it was not the intention behind this use-case.

[DouweM]

Either an exception or just failure to associate the appropriate business attributes so the trace ends up in a place no one is looking at.

@galfrishman Couldn't that second failure scenario be managed by making the relevant attributes required on the deps type, which is specified on the Agent as well? Note that I agree that shouldn't be the only way to do this, but at least that way you're forced to pass in those attributes at agent-run time via deps, instead of being able to forget to pass the custom attributes dictionary.

Either way, I agree it should also be possible to set these attribute at run-time, and I'm leaning towards doing that by adding the same instrument: InstrumentationSettings argument to the run methods that currently exists on Agent -- since you may also want to tweak other instrumentation settings on a per-run basis -- and letting you set attributes via a field on that object.

Would that work for you?

[galfrishman]

It makes sense yes. Thanks.

[DouweM]

@galfrishman Ok great. Would you be up for implementing this in a PR?

[galfrishman]

If I would we would not be having this conversation :)

[DouweM]

@galfrishman Heh, I'll have a look next month.

[DouweM]

@galfrishman Have you seen Baggage by the way? https://logfire.pydantic.dev/docs/reference/advanced/baggage/#basic-usage That could also be a solution here that doesn't require us to add any new fields.