|
| 1 | +--- |
| 2 | +title: Errors |
| 3 | +--- |
| 4 | + |
| 5 | +import CalloutBox from 'components/Docs/CalloutBox' |
| 6 | + |
| 7 | +<CalloutBox icon="IconInfo" title="Errors is in beta" type="fyi"> |
| 8 | + |
| 9 | +The Errors tab is currently in beta. We'd love to [hear your feedback](https://app.posthog.com/llm-analytics/errors#panel=support%3Afeedback%3A%3Alow%3Atrue) as we develop this feature. |
| 10 | + |
| 11 | +</CalloutBox> |
| 12 | + |
| 13 | +The Errors tab groups and aggregates error messages from your LLM application, helping you identify patterns and prioritize which errors to fix. |
| 14 | + |
| 15 | +## How it works |
| 16 | + |
| 17 | +When your LLM application captures an error event (with `$ai_is_error` set to `true`), PostHog automatically normalizes the error message by replacing dynamic values like IDs, timestamps, and numbers with placeholders. This groups similar errors together, even when they contain different specific values. |
| 18 | + |
| 19 | +For example, these two errors: |
| 20 | +- `"Request req_abc123 failed at 2025-01-11T14:25:51Z with status 429"` |
| 21 | +- `"Request req_xyz789 failed at 2025-01-10T09:15:30Z with status 429"` |
| 22 | + |
| 23 | +Are normalized to: |
| 24 | +- `"Request <ID> failed at <TIMESTAMP> with status <N>"` |
| 25 | + |
| 26 | +This allows you to see that both errors are the same underlying issue, rather than treating them as separate problems. |
| 27 | + |
| 28 | +## Error metrics |
| 29 | + |
| 30 | +The Errors tab displays the following metrics for each normalized error: |
| 31 | + |
| 32 | +| Metric | Description | |
| 33 | +|--------|-------------| |
| 34 | +| **error** | The normalized error message | |
| 35 | +| **traces** | Number of unique traces containing this error | |
| 36 | +| **generations** | Number of generation events with this error | |
| 37 | +| **spans** | Number of span events with this error | |
| 38 | +| **embeddings** | Number of embedding events with this error | |
| 39 | +| **sessions** | Number of unique sessions affected | |
| 40 | +| **users** | Number of unique users who encountered this error | |
| 41 | +| **days_seen** | Number of distinct days the error occurred | |
| 42 | +| **first_seen** | When the error first appeared | |
| 43 | +| **last_seen** | Most recent occurrence | |
| 44 | + |
| 45 | +## Investigating errors |
| 46 | + |
| 47 | +Click on any error row to drill down into the [Traces](/docs/llm-analytics/traces) tab, filtered to show only traces containing that specific error. This helps you: |
| 48 | + |
| 49 | +- See the full context of what led to the error |
| 50 | +- Examine the inputs and outputs around the failure |
| 51 | +- Identify patterns across affected traces |
| 52 | + |
| 53 | +## Capturing errors |
| 54 | + |
| 55 | +To have errors appear in the Errors tab, set `$ai_is_error` to `true` and include the error message in `$ai_error` when capturing LLM events. |
| 56 | + |
| 57 | +Most PostHog LLM integrations automatically capture errors when API calls fail. If you're using manual capture, include these properties: |
| 58 | + |
| 59 | +```python |
| 60 | +posthog.capture( |
| 61 | + distinct_id="user_123", |
| 62 | + event="$ai_generation", |
| 63 | + properties={ |
| 64 | + "$ai_trace_id": "trace_abc", |
| 65 | + "$ai_model": "gpt-4", |
| 66 | + "$ai_is_error": True, |
| 67 | + "$ai_error": "Rate limit exceeded: 429 Too Many Requests" |
| 68 | + } |
| 69 | +) |
| 70 | +``` |
| 71 | + |
| 72 | +## Custom error normalization |
| 73 | + |
| 74 | +If you want more control over how errors are grouped, you can set the `$ai_error_normalized` property yourself. When this property is provided, PostHog uses your value instead of auto-normalizing. |
| 75 | + |
| 76 | +```python |
| 77 | +posthog.capture( |
| 78 | + distinct_id="user_123", |
| 79 | + event="$ai_generation", |
| 80 | + properties={ |
| 81 | + "$ai_trace_id": "trace_abc", |
| 82 | + "$ai_is_error": True, |
| 83 | + "$ai_error": "OpenAI API error: rate_limit_exceeded for org-abc123", |
| 84 | + "$ai_error_normalized": "OpenAI rate limit exceeded" # Custom grouping |
| 85 | + } |
| 86 | +) |
| 87 | +``` |
| 88 | + |
| 89 | +## Normalization rules |
| 90 | + |
| 91 | +PostHog normalizes errors by replacing: |
| 92 | + |
| 93 | +| Pattern | Replacement | Example | |
| 94 | +|---------|-------------|---------| |
| 95 | +| UUIDs and request IDs | `<ID>` | `req_abc123` → `<ID>` | |
| 96 | +| ISO timestamps | `<TIMESTAMP>` | `2025-01-11T14:25:51Z` → `<TIMESTAMP>` | |
| 97 | +| Tool call IDs | `<TOOL_CALL_ID>` | `toolu_01ABC...` → `<TOOL_CALL_ID>` | |
| 98 | +| Function call IDs | `<CALL_ID>` | `call_abc123` → `call_<CALL_ID>` | |
| 99 | +| User IDs | `<USER_ID>` | `user_abc123` → `user_<USER_ID>` | |
| 100 | +| Token counts | `<TOKEN_COUNT>` | `"tokenCount":5000` → `"tokenCount":<TOKEN_COUNT>` | |
| 101 | +| Large numbers (9+ digits) | `<ID>` | `1234567890` → `<ID>` | |
| 102 | +| Other numbers | `<N>` | `429`, `5000` → `<N>` | |
| 103 | + |
| 104 | +Errors are truncated to 1000 characters before normalization. |
0 commit comments