Skip to content

Commit 479370f

Browse files
alexmojakialexmojaki
authored
Document MetricsOptions(collect_in_spans=True) configuration and the logfire.metrics attribute (#1437)
1 parent 026d8a6 commit 479370f

File tree

3 files changed

+208
-0
lines changed

3 files changed

+208
-0
lines changed

‎docs/images/logfire-screenshot-metrics-in-spans-live-view.png‎

77.6 KB
Loading

‎docs/reference/advanced/metrics-in-spans.md‎

Lines changed: 207 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,207 @@
1+
# Aggregating Metrics in Spans
2+
3+
Logfire lets you aggregate counter and histogram metrics within the current active span and its ancestors. This is particularly useful for calculating totals for things like LLM token usage or costs on higher-level operations, making it easier and more efficient to query this data without processing individual child spans.
4+
5+
This guide will walk you through how to enable this feature and use it with both custom metrics and automated instrumentation.
6+
7+
!!! note
8+
This is an experimental feature. The API and data format may change in future releases. We welcome your feedback to help us prioritize improvements and stability.
9+
10+
This only works for metrics recorded within the same process. It does not aggregate metrics across distributed traces.
11+
12+
## Enabling Metric Aggregation
13+
14+
This feature is _always_ enabled for any metric named `operation.cost`, which in particular is recorded by [Pydantic AI](https://ai.pydantic.dev/) (since [v1.0.11](https://github.com/pydantic/pydantic-ai/releases/tag/v1.0.11)) when [instrumented](../../integrations/llms/pydanticai.md) for LLM costs.
15+
16+
To enable this feature for all metrics, you need to configure Logfire with the `collect_in_spans` option in [`MetricsOptions`][logfire.MetricsOptions]. This should be done once when your application starts.
17+
18+
```py
19+
import logfire
20+
21+
logfire.configure(metrics=logfire.MetricsOptions(collect_in_spans=True))
22+
```
23+
24+
Once enabled, any counters or histograms recorded within a span will be aggregated into a `logfire.metrics` attribute on that span.
25+
26+
## Example: A Custom Counter
27+
28+
Let's start with a simple example of tracking some cumulative amount.
29+
30+
First, we define a metric counter. Then, within a span, we add to it multiple times.
31+
32+
```py
33+
import logfire
34+
35+
logfire.configure(metrics=logfire.MetricsOptions(collect_in_spans=True))
36+
37+
counter = logfire.metric_counter('my_amount')
38+
39+
with logfire.span('doing stuff'):
40+
counter.add(1)
41+
counter.add(2)
42+
```
43+
44+
The `doing stuff` span will now contain a `logfire.metrics` attribute that holds the aggregated total of the `my_amount` counter, i.e. `3`.
45+
Running the following query in the Explore view:
46+
47+
```sql
48+
SELECT attributes->>'logfire.metrics'->>'my_amount'->>'total' AS total_amount
49+
FROM records
50+
WHERE span_name = 'doing stuff'
51+
```
52+
53+
will return one row with `total_amount` equal to `3`.
54+
55+
## Example: LLM Token Usage and Costs with Pydantic AI
56+
57+
Generative AI instrumentations like [Pydantic AI](../../integrations/llms/pydanticai.md) or [OpenAI](../../integrations/llms/openai.md) that follow OpenTelemetry conventions will record a metric called `gen_ai.client.token.usage`. You can use metric aggregation to get a total of tokens used in a higher-level operation that may involve multiple LLM calls. Pydantic AI specifically also records an `operation.cost` metric, which is always aggregated.
58+
59+
Here’s an example:
60+
61+
```py
62+
from pydantic_ai import Agent
63+
import logfire
64+
65+
logfire.configure(metrics=logfire.MetricsOptions(collect_in_spans=True))
66+
logfire.instrument_pydantic_ai()
67+
68+
agent = Agent('gpt-4o')
69+
70+
@agent.tool_plain
71+
async def get_random_number() -> int:
72+
return 4
73+
74+
with logfire.span('span'):
75+
agent.run_sync('Give me one random number')
76+
agent.run_sync('Generate two random numbers')
77+
```
78+
79+
The calls to `agent.run_sync` create child spans named `agent run`. The outer `span` aggregates the token metrics from these children, as shown in the Live View:
80+
81+
![Token and cost aggregation in live view](../../images/logfire-screenshot-metrics-in-spans-live-view.png)
82+
83+
### Understanding the Span Data
84+
85+
The outer `'span'` now has a `logfire.metrics` attribute containing the aggregated token data from the two `agent.run_sync` calls. The JSON structure looks like this:
86+
87+
```json
88+
{
89+
"gen_ai.client.token.usage": {
90+
"details": [
91+
{
92+
"attributes": {
93+
"gen_ai.operation.name": "chat",
94+
"gen_ai.request.model": "gpt-4o",
95+
"gen_ai.response.model": "gpt-4o-2024-08-06",
96+
"gen_ai.system": "openai",
97+
"gen_ai.token.type": "input"
98+
},
99+
"total": 224
100+
},
101+
{
102+
"attributes": {
103+
"gen_ai.operation.name": "chat",
104+
"gen_ai.request.model": "gpt-4o",
105+
"gen_ai.response.model": "gpt-4o-2024-08-06",
106+
"gen_ai.system": "openai",
107+
"gen_ai.token.type": "output"
108+
},
109+
"total": 73
110+
}
111+
],
112+
"total": 297
113+
},
114+
"operation.cost": {
115+
"details": [
116+
{
117+
"attributes": {
118+
"gen_ai.operation.name": "chat",
119+
"gen_ai.request.model": "gpt-4o",
120+
"gen_ai.response.model": "gpt-4o-2024-08-06",
121+
"gen_ai.system": "openai",
122+
"gen_ai.token.type": "input"
123+
},
124+
"total": 0.00056
125+
},
126+
{
127+
"attributes": {
128+
"gen_ai.operation.name": "chat",
129+
"gen_ai.request.model": "gpt-4o",
130+
"gen_ai.response.model": "gpt-4o-2024-08-06",
131+
"gen_ai.system": "openai",
132+
"gen_ai.token.type": "output"
133+
},
134+
"total": 0.00073
135+
}
136+
],
137+
"total": 0.00129
138+
}
139+
}
140+
```
141+
142+
Note:
143+
144+
1. Each `total` value (both at the top level and inside `details`) in the JSON matches a value in the token badge in the UI.
145+
2. Each of these metrics (but not metrics in general) has separate entries for `input` and `output` in the `details`, each with its own total. In general, each combination of values of `attributes` has its own entry in `details`. Here, the only differing attribute is `gen_ai.token.type`.
146+
147+
The general structure of `logfire.metrics` is:
148+
149+
```json
150+
{
151+
"<metric_name>": {
152+
"details": [
153+
{
154+
"attributes": { ... },
155+
"total": <number>
156+
},
157+
...
158+
],
159+
"total": <number>
160+
},
161+
...
162+
}
163+
```
164+
165+
### Querying Nested Token Data
166+
167+
To query these nested details, you need a more complex SQL query to "un-nest" the JSON data into a flat, table-like structure.
168+
169+
```sql
170+
WITH
171+
with_span_metric_name AS (SELECT unnest(json_keys(attributes->>'logfire.metrics')::text[]) AS span_metric_name, * FROM records),
172+
with_span_metric AS (SELECT attributes->>'logfire.metrics'->>span_metric_name AS span_metric, * FROM with_span_metric_name),
173+
with_span_metric_detail AS (SELECT span_metric->>'details'->>unnest(generate_series((json_length(span_metric->>'details') - 1)::int)) AS span_metric_detail, * FROM with_span_metric)
174+
SELECT
175+
span_name,
176+
span_metric_name,
177+
span_metric_detail->>'total' AS total,
178+
span_metric_detail->>'attributes'->>'gen_ai.token.type' AS token_type
179+
FROM with_span_metric_detail
180+
WHERE span_metric_name = 'gen_ai.client.token.usage'
181+
```
182+
183+
**How this query works:**
184+
185+
* The `WITH` clauses progressively expand the nested JSON in the `logfire.metrics` attribute.
186+
* `with_span_metric_name` unnests the metric names (e.g. `gen_ai.client.token.usage` and `operation.cost`).
187+
* `with_span_metric` extracts the JSON object for each metric.
188+
* `with_span_metric_detail` unnests the `details` array, creating a separate row for each item (one for `input` and one for `output` in our example).
189+
190+
You can copy the `WITH` clauses as a reusable prefix for any query that needs to analyze aggregated metrics. The final `SELECT` statement then easily extracts the totals.
191+
192+
The result of this query will look like this, showing token counts and costs broken down by span and type:
193+
194+
| span_name | span_metric_name | total | token_type |
195+
|-----------|--------------------------------|-------|------------|
196+
| span | gen_ai.client.token.usage | 224 | input |
197+
| span | gen_ai.client.token.usage | 73 | output |
198+
| span | operation.cost | 0.00056 | input |
199+
| span | operation.cost | 0.00073 | output |
200+
| agent run | gen_ai.client.token.usage | 95 | input |
201+
| agent run | gen_ai.client.token.usage | 19 | output |
202+
| agent run | operation.cost | 0.0002375 | input |
203+
| agent run | operation.cost | 0.00019 | output |
204+
| agent run | gen_ai.client.token.usage | 129 | input |
205+
| agent run | gen_ai.client.token.usage | 54 | output |
206+
| agent run | operation.cost | 0.0003225 | input |
207+
| agent run | operation.cost | 0.00054 | output |

‎mkdocs.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -173,6 +173,7 @@ nav:
173173
- Baggage: reference/advanced/baggage.md
174174
- Generators: reference/advanced/generators.md
175175
- Testing: reference/advanced/testing.md
176+
- Aggregating Metrics in Spans: reference/advanced/metrics-in-spans.md
176177
- Examples: reference/examples.md
177178
- Configuration: reference/configuration.md
178179
- Data Regions: reference/data-regions.md

0 commit comments

Comments
 (0)