Add Agent and agent run metadata and expose it on result objects and span attributes #3370

sstanovnik · 2025-11-07T15:33:21Z

This allows users to add extra metadata to the agent's span, either as a direct string or dict, or a callable that takes the RunContext. The attributes are added/computed after the agent finishes. The metadata is under the logfire.agent.metadata attribute.

sstanovnik · 2025-11-07T15:34:42Z

pydantic_ai_slim/pydantic_ai/agent/__init__.py

    def _run_span_end_attributes(
        self,
        settings: InstrumentationSettings,
-        usage: _usage.RunUsage,
-        message_history: list[_messages.ModelMessage],
-        new_message_index: int,
+        graph_ctx: GraphRunContext[_agent_graph.GraphAgentState, _agent_graph.GraphAgentDeps[AgentDepsT, OutputDataT]],
    ):


I reworked what attributes are passed to this because the existing ones are already available in the RunContext. I hope that's okay.

DouweM · 2025-11-07T20:54:55Z

@sstanovnik Have you seen Baggage https://logfire.pydantic.dev/docs/reference/advanced/baggage/#basic-usage That could also be a solution here that doesn't require us to add any new fields.

DouweM · 2025-11-07T20:58:47Z

docs/logfire.md

+from pydantic_ai.models.instrumented import InstrumentationSettings
+
+
+def span_attribute_callback(ctx: RunContext[None]) -> dict[str, str]:


This is useful, but not type safe unfortunately, as InstrumentationSettings is not generic in the deps type (and probably shouldn't be), meaning that you could create a span_attribute_callback that takes RunContext[Foo] and use it on an Agent[Bar] without any typing issues, which would then fail at runtime if you try to read attrs that exist on Foo but not Bar...

The obvious ways to solve that would be to make InstrumentationSettings generic or to make Agent and agent.run take the span_attributes directly, but I don't like either of those :) If baggage works for you, I'd be inclined to not add any new feature to Pydantic AI and just document that instead.

Yeah, I noticed that this couldn't be made type-safe without making the carrier generic, but that seems like a huge undertaking that also changes the interface in a way that, I would think, isn't desirable until v2.

Baggage would work for my immediate usecase, even though it applies to all child spans, which I don't really want it to - conceptually, I'm wanting to add data to the one specific span.

What about only accepting a dict for now, and leaving a callable taking RunContext[TDeps] for whenever InstrumentationSettings can be adapted to take a generic parameter?

@sstanovnik Maybe instrumentation settings are not the best place for this:

In line with #3263, what do you think about arbitrary metadata: dict[str, Any] on the Agent, that's also set on the agent run span under a metadata attribute? That way you wouldn't be setting span attributes directly (assuming that's not an issue), but they would be queryable etc, and it'd feel more natural to add this on both Agent and agent.run directly, with support for RunContext-taking callables.

That's also great - any way of adding information to the span would be great. What do you mean by

That way you wouldn't be setting span attributes directly (assuming that's not an issue), but they would be queryable [...]

I see that the linked PR does add the metadata into the span.

I'm up for closing this PR and opening another with the same approach using metadata, if you'd like :)

@sstanovnik I just mean that the metadata dict keys will not directly become span attributes, as they'd be nested under metadata, but if that's fine for your use case, feel free to update this PR (with force push possibly), or create a new one!

Metadata is attached to the logfire.agent.metadata span attribute. It can either be a string, dict, or a callable taking the RunContext and returning a string or a dict.

sstanovnik

Force-pushed this to retain the relevant conversation history of the previous implementation. I also added some comments on particulars I'm not certain about.

A major difference from your suggestion is that I did not add this to the Agent.run method. I felt the change was large enough as it is without having to figure out how to wire/override/merge that.

pydantic_ai_slim/pydantic_ai/agent/__init__.py

docs/logfire.md

pydantic_ai_slim/pydantic_ai/agent/__init__.py

github-actions · 2025-11-25T14:00:41Z

This PR is stale, and will be closed in 3 days if no reply is received.

sstanovnik · 2025-11-27T14:09:53Z

Apologies for the delay in responding to your review. Thank you for taking the time for this.

I believe I've addressed all comments:

The metadata type is changed from str | dict[str, str] to dict[str, Any].
logfire.agent.metadata is now agent.metadata.
Metadata is now available on run results.
Metadata is fully computed even if the agent run fails.
Both run and iter now have the metadata attribute and work with .override().

DouweM · 2025-11-28T19:18:55Z

docs/agents.md

 You can retrieve usage statistics (tokens, requests, etc.) at any time from the [`AgentRun`][pydantic_ai.agent.AgentRun] object via `agent_run.usage()`. This method returns a [`RunUsage`][pydantic_ai.usage.RunUsage] object containing the usage data.

 Once the run finishes, `agent_run.result` becomes a [`AgentRunResult`][pydantic_ai.agent.AgentRunResult] object containing the final output (and related metadata).
+You can inspect [`agent_run.metadata`][pydantic_ai.agent.AgentRun] or [`agent_run.result.metadata`][pydantic_ai.agent.AgentRunResult] after the run completes to read any metadata configured on the agent.


This deserves a separate section that also explains how to set it. I think much of what's currently in logfire.md can be moved here, and then those docs can link here.

I put most of the documentation here and only mentioned the relevant parts in logfire.md, linking here for details.

DouweM · 2025-11-28T19:20:00Z

docs/logfire.md

+    'openai:gpt-5',
+    instrument=True,
+    metadata=lambda ctx: {'deployment': 'staging', 'tenant': ctx.deps.tenant},
+)


It'd be nice to expand the example to show running the agent with some deps.tenant, and then printing result.metadata at the end

Example expanded - I included a complete combined example for the "Accessing usage and final output" section in the docs.

pydantic_ai_slim/pydantic_ai/agent/__init__.py

DouweM · 2025-11-28T19:23:35Z

pydantic_ai_slim/pydantic_ai/agent/__init__.py

            usage: Optional usage to start with, useful for resuming a conversation or agents used in tools.
+            metadata: Optional metadata to attach to this run. Accepts a dictionary or a callable taking
+                [`RunContext`][pydantic_ai.tools.RunContext]. The resolved dictionary is shallow merged into the
+                agent's metadata (or any [`Agent.override`][pydantic_ai.agent.Agent.override]) with run-level keys


Existing agent.overrides always cause the agent.run value to be ignored, fully overwriting the agent + agent run original. We should have the same behavior here

DouweM · 2025-11-28T19:24:29Z

pydantic_ai_slim/pydantic_ai/agent/__init__.py

+                    try:
+                        yield agent_run
+                    finally:
+                        resolve_run_metadata()


Does this need to be a function or could it be inlined?

With the change that metadata is computed both at the start and end of the run, this now needs to be a function, but I placed it as a class method, not inline as it was before.

pydantic_ai_slim/pydantic_ai/agent/__init__.py

DouweM · 2025-11-28T19:27:14Z

pydantic_ai_slim/pydantic_ai/agent/__init__.py

+            base_config = self._metadata
+
+        base_metadata = self._resolve_metadata_config(base_config, ctx)
+        run_metadata = self._resolve_metadata_config(run_metadata_config, ctx)


As mentioned above, overridden metadata fully overrides run metadata

Done, override now completely overrides both agent-constructor and run-level metadata.

pydantic_ai_slim/pydantic_ai/agent/__init__.py

DouweM · 2025-11-28T19:31:23Z

pydantic_ai_slim/pydantic_ai/agent/abstract.py

            usage: Optional usage to start with, useful for resuming a conversation or agents used in tools.
+            metadata: Optional metadata to attach to this run. Accepts a dictionary or a callable taking
+                [`RunContext`][pydantic_ai.tools.RunContext]. The resolved dictionary is shallow merged into the
+                agent's metadata (or any [`Agent.override`][pydantic_ai.agent.Agent.override]) with run-level keys


See above, this behavior is inconsistent with existing expectations, so the behavior and docstring will need updating

Docstrings updated to match the new override behaviour.

pydantic_ai_slim/pydantic_ai/result.py

github-actions · 2025-12-07T14:00:39Z

This PR is stale, and will be closed in 3 days if no reply is received.

sstanovnik · 2025-12-09T08:33:33Z

Thank you again bearing with my intermittent responses and for taking the time to review this code. I've addressed all comments:

The .override() now completely overrides metadata, agent-level and run-level settings are ignored.
Run-level settings are still shallow-merged into agent-level metadata.
Metadata is now computed twice: first at the beginning of the run, then upon successful completion.
Documentation was updated to reflect the new behaviour, and placed in agents.md, with links in logfire.md.

DouweM · 2025-12-10T18:46:49Z

docs/agents.md

 You can retrieve usage statistics (tokens, requests, etc.) at any time from the [`AgentRun`][pydantic_ai.agent.AgentRun] object via `agent_run.usage()`. This method returns a [`RunUsage`][pydantic_ai.usage.RunUsage] object containing the usage data.

 Once the run finishes, `agent_run.result` becomes a [`AgentRunResult`][pydantic_ai.agent.AgentRunResult] object containing the final output (and related metadata).
+You can inspect [`agent_run.metadata`][pydantic_ai.agent.AgentRun] or [`agent_run.result.metadata`][pydantic_ai.agent.AgentRunResult] after the run completes to read any metadata configured on the [`Agent`][pydantic_ai.agent.Agent].


Let's make this a separate section that specifically documents the Metadata feature, and shows that it can be set on the agent as well as the run.

DouweM · 2025-12-10T18:48:19Z

docs/logfire.md

+
+See the [usage and metadata example in the agents guide](agents.md#accessing-usage-and-final-output) for an example that derives metadata from `deps` and accesses [`AgentRunResult.metadata`][pydantic_ai.agent.AgentRunResult].
+
+Resolved metadata is available after the run completes on


I think we just need this on the main doc. If we link to there from here, this section only needs to mention that the metadata ends up on the metadata span attribute

DouweM · 2025-12-10T18:50:56Z

pydantic_ai_slim/pydantic_ai/_run_context.py

    """Whether the output passed to an output validator is partial."""
    run_id: str | None = None
    """"Unique identifier for the agent run."""
+    metadata: dict[str, Any] | None = None


Have a look at TemporalRunContext; we should add the field there to be included by default, and also update the docstring and Temporal docs to mention that.

Maybe the logic there can be changed to only list those fields that are excluded rather than the ones that are included, as new keys should typically be included unless the values are very large.

DouweM · 2025-12-10T18:51:42Z

pydantic_ai_slim/pydantic_ai/agent/__init__.py

            model_settings: Optional settings to use for this model's request.
            usage_limits: Optional limits on model request count or token usage.
            usage: Optional usage to start with, useful for resuming a conversation or agents used in tools.
+            metadata: Optional metadata to attach to this run. Accepts a dictionary or a callable taking


We should add it to the methods on UIAdapter as well

DouweM · 2025-12-10T18:55:09Z

pydantic_ai_slim/pydantic_ai/agent/__init__.py

+
+                    final_result = agent_run.result
+                    if instrumentation_settings and run_span.is_recording():
+                        if instrumentation_settings.include_content and final_result is not None:


Let's combine these 2 ifs

DouweM · 2025-12-10T18:56:16Z

pydantic_ai_slim/pydantic_ai/agent/__init__.py

+    ) -> dict[str, Any] | None:
+        run_context = build_run_context(graph_run_ctx)
+        resolved_metadata = self._get_metadata(run_context, metadata)
+        run_context.metadata = resolved_metadata


I don't think this is needed as the run_context var is discarded

sstanovnik commented Nov 7, 2025

View reviewed changes

DouweM requested changes Nov 7, 2025

View reviewed changes

DouweM self-assigned this Nov 7, 2025

DouweM added the awaiting author revision label Nov 7, 2025

Add metadata to the Agent class.

11687ce

Metadata is attached to the logfire.agent.metadata span attribute. It can either be a string, dict, or a callable taking the RunContext and returning a string or a dict.

sstanovnik force-pushed the extra-agent-span-attributes branch from c57e1bb to 11687ce Compare November 14, 2025 20:21

sstanovnik changed the title ~~Add span_attributes to InstrumentationSettings.~~ Add metadata to the Agent class. Nov 14, 2025

Misplaced doc line.

cf001b5

sstanovnik commented Nov 14, 2025

View reviewed changes

pydantic_ai_slim/pydantic_ai/agent/__init__.py Outdated Show resolved Hide resolved

pydantic_ai_slim/pydantic_ai/agent/__init__.py Outdated Show resolved Hide resolved

pydantic_ai_slim/pydantic_ai/agent/__init__.py Show resolved Hide resolved

Coverage.

150777a

DouweM requested changes Nov 17, 2025

View reviewed changes

github-actions bot added the Stale label Nov 25, 2025

sstanovnik added 7 commits November 27, 2025 08:52

Merge branch 'main' into extra-agent-span-attributes

9a37466

Remove string, add dict-str-any.

2faab1e

Remove logfire prefix from agent metadata.

6fe93df

Add metadata to run contexts and results.

44553ca

Compute run metadata even on failure.

6db0f1f

Add metadata to run and iter.

d05887c

Doc fixes.

0c5b782

sstanovnik requested a review from DouweM November 27, 2025 14:09

github-actions bot removed the Stale label Nov 28, 2025

DouweM requested changes Nov 28, 2025

View reviewed changes

github-actions bot added the Stale label Dec 7, 2025

sstanovnik added 3 commits December 8, 2025 13:57

Merge branch 'refs/heads/main' into extra-agent-span-attributes

c5560ba

Agent metadata without the value.

5666b0e

Consolidate docs under agents.md.

0ea1157

sstanovnik added 4 commits December 8, 2025 15:48

Override to completely override agent and run metadata.

c707bfa

Rename metadata computation functions for consistency.

af3640b

Rename agent.metadata to metadata.

333dfe5

Compute metadata both at start and finish of a run.

149f1ae

sstanovnik requested a review from DouweM December 9, 2025 08:33

github-actions bot removed the Stale label Dec 9, 2025

DouweM requested changes Dec 10, 2025

View reviewed changes

DouweM changed the title ~~Add metadata to the Agent class.~~ Add Agent and agent run metadata and expose it on result objects and span attributes Dec 10, 2025

		from pydantic_ai.models.instrumented import InstrumentationSettings


		def span_attribute_callback(ctx: RunContext[None]) -> dict[str, str]:


		See the [usage and metadata example in the agents guide](agents.md#accessing-usage-and-final-output) for an example that derives metadata from `deps` and accesses [`AgentRunResult.metadata`][pydantic_ai.agent.AgentRunResult].

		Resolved metadata is available after the run completes on

Add Agent and agent run metadata and expose it on result objects and span attributes #3370

Are you sure you want to change the base?

Add Agent and agent run metadata and expose it on result objects and span attributes #3370

Uh oh!

Conversation

sstanovnik commented Nov 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

DouweM commented Nov 7, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sstanovnik left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Nov 25, 2025

Uh oh!

sstanovnik commented Nov 27, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actions bot commented Dec 7, 2025

Uh oh!

sstanovnik commented Dec 9, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

sstanovnik commented Nov 7, 2025 •

edited

Loading