Skip to content

Commit 3dbf9c4

Browse files
committed
chore(docs): add documentation for LLM Analytics Errors tab
Add documentation for the Errors tab feature including: - Beta callout - How error normalization works - Metrics displayed in the tab - How to capture and customize errors - Normalization rules reference Part of PostHog/posthog#44323
1 parent 9468df9 commit 3dbf9c4

File tree

2 files changed

+114
-0
lines changed

2 files changed

+114
-0
lines changed
Lines changed: 104 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,104 @@
1+
---
2+
title: Errors
3+
---
4+
5+
import CalloutBox from 'components/Docs/CalloutBox'
6+
7+
<CalloutBox icon="IconInfo" title="Errors is in beta" type="fyi">
8+
9+
The Errors tab is currently in beta. We'd love to [hear your feedback](https://app.posthog.com/llm-analytics/errors#panel=support%3Afeedback%3A%3Alow%3Atrue) as we develop this feature.
10+
11+
</CalloutBox>
12+
13+
The Errors tab groups and aggregates error messages from your LLM application, helping you identify patterns and prioritize which errors to fix.
14+
15+
## How it works
16+
17+
When your LLM application captures an error event (with `$ai_is_error` set to `true`), PostHog automatically normalizes the error message by replacing dynamic values like IDs, timestamps, and numbers with placeholders. This groups similar errors together, even when they contain different specific values.
18+
19+
For example, these two errors:
20+
- `"Request req_abc123 failed at 2025-01-11T14:25:51Z with status 429"`
21+
- `"Request req_xyz789 failed at 2025-01-10T09:15:30Z with status 429"`
22+
23+
Are normalized to:
24+
- `"Request <ID> failed at <TIMESTAMP> with status <N>"`
25+
26+
This allows you to see that both errors are the same underlying issue, rather than treating them as separate problems.
27+
28+
## Error metrics
29+
30+
The Errors tab displays the following metrics for each normalized error:
31+
32+
| Metric | Description |
33+
|--------|-------------|
34+
| **error** | The normalized error message |
35+
| **traces** | Number of unique traces containing this error |
36+
| **generations** | Number of generation events with this error |
37+
| **spans** | Number of span events with this error |
38+
| **embeddings** | Number of embedding events with this error |
39+
| **sessions** | Number of unique sessions affected |
40+
| **users** | Number of unique users who encountered this error |
41+
| **days_seen** | Number of distinct days the error occurred |
42+
| **first_seen** | When the error first appeared |
43+
| **last_seen** | Most recent occurrence |
44+
45+
## Investigating errors
46+
47+
Click on any error row to drill down into the [Traces](/docs/llm-analytics/traces) tab, filtered to show only traces containing that specific error. This helps you:
48+
49+
- See the full context of what led to the error
50+
- Examine the inputs and outputs around the failure
51+
- Identify patterns across affected traces
52+
53+
## Capturing errors
54+
55+
To have errors appear in the Errors tab, set `$ai_is_error` to `true` and include the error message in `$ai_error` when capturing LLM events.
56+
57+
Most PostHog LLM integrations automatically capture errors when API calls fail. If you're using manual capture, include these properties:
58+
59+
```python
60+
posthog.capture(
61+
distinct_id="user_123",
62+
event="$ai_generation",
63+
properties={
64+
"$ai_trace_id": "trace_abc",
65+
"$ai_model": "gpt-4",
66+
"$ai_is_error": True,
67+
"$ai_error": "Rate limit exceeded: 429 Too Many Requests"
68+
}
69+
)
70+
```
71+
72+
## Custom error normalization
73+
74+
If you want more control over how errors are grouped, you can set the `$ai_error_normalized` property yourself. When this property is provided, PostHog uses your value instead of auto-normalizing.
75+
76+
```python
77+
posthog.capture(
78+
distinct_id="user_123",
79+
event="$ai_generation",
80+
properties={
81+
"$ai_trace_id": "trace_abc",
82+
"$ai_is_error": True,
83+
"$ai_error": "OpenAI API error: rate_limit_exceeded for org-abc123",
84+
"$ai_error_normalized": "OpenAI rate limit exceeded" # Custom grouping
85+
}
86+
)
87+
```
88+
89+
## Normalization rules
90+
91+
PostHog normalizes errors by replacing:
92+
93+
| Pattern | Replacement | Example |
94+
|---------|-------------|---------|
95+
| UUIDs and request IDs | `<ID>` | `req_abc123``<ID>` |
96+
| ISO timestamps | `<TIMESTAMP>` | `2025-01-11T14:25:51Z``<TIMESTAMP>` |
97+
| Tool call IDs | `<TOOL_CALL_ID>` | `toolu_01ABC...``<TOOL_CALL_ID>` |
98+
| Function call IDs | `<CALL_ID>` | `call_abc123``call_<CALL_ID>` |
99+
| User IDs | `<USER_ID>` | `user_abc123``user_<USER_ID>` |
100+
| Token counts | `<TOKEN_COUNT>` | `"tokenCount":5000``"tokenCount":<TOKEN_COUNT>` |
101+
| Large numbers (9+ digits) | `<ID>` | `1234567890``<ID>` |
102+
| Other numbers | `<N>` | `429`, `5000``<N>` |
103+
104+
Errors are truncated to 1000 characters before normalization.

src/navs/index.js

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4814,6 +4814,16 @@ export const docsMenu = {
48144814
color: 'blue',
48154815
featured: true,
48164816
},
4817+
{
4818+
name: 'Errors',
4819+
url: '/docs/llm-analytics/errors',
4820+
icon: 'IconWarning',
4821+
color: 'red',
4822+
badge: {
4823+
title: 'Beta',
4824+
className: 'uppercase !bg-blue/10 !text-blue !dark:text-white !dark:bg-blue/50',
4825+
},
4826+
},
48174827
{
48184828
name: 'Embeddings',
48194829
url: '/docs/llm-analytics/embeddings',

0 commit comments

Comments
 (0)