feat(llma): Graduate Trace/Event Summarization to GA

[andrewm4894]

Overview

Graduate the LLM analytics trace/event summarization feature from alpha to beta. This feature allows users to generate AI-powered summaries of LLM traces and events, helping them quickly understand complex AI interactions.

Current State:

• Backend: Full implementation with OpenAI + Gemini providers
• Rate limits: 50/min burst, 200/hour sustained, 500/day cap
• Access: Gated by is_ai_data_processing_approved org flag
• Backend event tracking exists (llma summarization generated)

Scope: UI-focused user-initiated summarization. Batch processing (Temporal workflows) is out of scope as it's part of a separate clustering workstream.

---

Phase 1: Tracking & Instrumentation

• Add data-attr to trace view tabs for autocapture (feat(llma): add data-attr to trace view tabs for autocapture #44734)
• Add feedback UI (thumbs up/down) to SummaryViewDisplay.tsx (feat(llma): add feedback UI for summarization #44736)
• Track feedback events: llma summarization feedback (feat(llma): add feedback UI for summarization #44736)

PRs:

• feat(llma): add data-attr to trace view tabs for autocapture #44734 - Add data-attr to trace view tabs for autocapture
• feat(llma): add feedback UI for summarization #44736 - Add feedback UI for summarization

---

Phase 2: Documentation

• Create summarization docs page in posthog.com repo (chore(docs): add trace summarization documentation posthog.com#14350)
• Update to beta status (chore(docs): add trace summarization documentation posthog.com#14350)

PRs:

• chore(docs): add trace summarization documentation posthog.com#14350 - Add trace summarization documentation

---

Phase 3: Pre-GA Review

• Security audit: Error responses verified - no sensitive data leaks
• Review rate limits for GA scale - keeping current limits (500/day sufficient, OpenAI is BYOK)
• Test with fresh org (AI consent flow)

---

Phase 4: Beta Release

• Remove feature flag gating (feat(llma): move summarization to beta #44740)
• Update UI tag from Alpha to Beta (feat(llma): move summarization to beta #44740)
• Update docs to beta status (chore(docs): add trace summarization documentation posthog.com#14350)
• Create monitoring dashboard
• Fix pseudo-trace focus regression (fix(llma): Don't auto-focus first event in pseudo-traces #44741)

PRs:

• feat(llma): move summarization to beta #44740 - Move summarization to beta (remove FF, update tag)
• fix(llma): Don't auto-focus first event in pseudo-traces #44741 - Fix pseudo-trace initial focus (UX regression from feat(llma): focus on the next event if in a pseudo-trace #43810)

Dashboard: LLM Analytics Summarization - GA Monitoring

Includes:

• Daily summarizations trend
• Average duration
• Summarizations by mode (minimal/detailed)
• Summarizations by type (trace/event)
• Unique users

---

Phase 5: GA Release (Future)

• Remove beta tag from UI
• Remove beta callout from docs
• Add to PostHog changelog

---

Success Metrics

• Weekly active users using summarization
• Summarizations per active user
• Feedback positive ratio
• Error rate < 5%

---

PRs & Assets

Item	Description	Status
#44734	Frontend data-attr tracking	Open
#44736	Feedback UI & tracking	Open
#44740	Move to beta (remove FF, update tag)	Open
#44741	Fix pseudo-trace focus regression	Open
PostHog/posthog.com#14350	Documentation (beta)	Open
Dashboard	Monitoring	Done

---

Out of Scope

• Batch processing / Temporal workflows (separate clustering workstream)
• BYOK changes (keeping current approach - rate limits sufficient)
• Rate limit changes (review post-GA based on usage)