diff --git a/agents/monitoring/enterprise/SKILL.md b/agents/monitoring/enterprise/SKILL.md index 2cf15ce..39c2403 100644 --- a/agents/monitoring/enterprise/SKILL.md +++ b/agents/monitoring/enterprise/SKILL.md @@ -1,67 +1,255 @@ --- name: enterprise-monitor -description: Monitors enterprise account health, usage, and contract compliance. Alerts on low engagement, quota breaches, or churn risk. -tags: [monitoring, enterprise, accounts, contracts] +description: "Monitor enterprise account health, usage, and contract compliance. Detects low engagement, quota breaches, and churn risk. Use when: (1) detecting enterprise account usage drops, (2) alerting on inactive accounts, (3) investigating power user engagement changes." +tags: [monitoring, enterprise, accounts, contracts, churn-risk] --- # Enterprise Monitor -## When to use - -- "Monitor enterprise account usage" -- "Monitor enterprise churn risk" -- "Alert when enterprise account is inactive" - -## What it monitors - -- **Account usage**: DAU, WAU, MAU per enterprise org -- **Token consumption**: Usage vs contracted quota, historical consumption trends -- **User activation**: % of seats active -- **Engagement**: Video generations, image generations, downloads per org -- **Churn signals**: Declining usage, inactive users - -## Steps - -1. **Gather requirements from user:** - - Which enterprise org(s) to monitor (or all) - - Alert threshold based on historical usage of each account (e.g., "usage drops > 30% vs their baseline", "MAU below their 30-day average", "< 50% of contracted quota") - - Time window (weekly, monthly) - - Notification channel (Slack, email, Linear issue) - -2. **Read shared files:** - - `shared/product-context.md` — LTX products, enterprise business model, user types - - `shared/bq-schema.md` — Enterprise user segmentation queries - - `shared/metric-standards.md` — Enterprise metrics, quota tracking - - `shared/event-registry.yaml` — Feature events (if analyzing engagement) - - `shared/gpu-cost-query-templates.md` — GPU cost queries (if analyzing infrastructure costs) - - `shared/gpu-cost-analysis-patterns.md` — Cost analysis patterns (if analyzing infrastructure costs) - -3. **Identify enterprise users:** - - Use enterprise segmentation CTE from bq-schema.md (lines 441-461) - - Apply McCann split (McCann_NY vs McCann_Paris) - - Exclude Lightricks and Popular Pays - -4. **Write monitoring SQL:** - - Query org-level usage metrics - - Set baseline for each org based on their historical usage (e.g., 30-day average, 90-day trend) - - Compare current usage against org-specific baseline or contracted quota - - Flag orgs below threshold or showing decline - - Flag meaningful drops for power users (users with top usage within each org) - -5. **Present to user:** - - Show SQL query - - Show example alert format with org name and metrics - - Confirm threshold values and alert logic - -6. **Set up alert** (manual for now): - - Document SQL - - Configure notification to customer success team - -## Rules - -- DO use EXACT enterprise segmentation CTE from bq-schema.md without modification -- DO apply McCann split (McCann_NY vs McCann_Paris) -- DO exclude Lightricks and Popular Pays from enterprise orgs -- DO break out pilot vs contracted accounts -- DO NOT alert on free/self-serve users — this agent is enterprise-only -- DO include org name in alert for easy customer success follow-up +## 1. Overview (Why?) + +Enterprise accounts (contract and pilot) represent high-value customers with negotiated quotas and specific engagement patterns. Unlike self-serve users, enterprise usage should be monitored per-organization with org-specific baselines, since each has different team sizes, use cases, and contract terms. + +This skill provides **autonomous enterprise account monitoring** that detects declining usage, underutilization of quotas, inactive periods, and power user drops — all of which signal churn risk or engagement problems that require customer success intervention. + +**Problem solved**: Identify enterprise churn risk early through usage signals — before contracts end or accounts go completely inactive — with org-level root cause analysis. + +## 2. Requirements (What?) + +Monitor these outcomes autonomously: + +- [ ] DAU/WAU/MAU drops per enterprise org (> 30% vs org baseline) +- [ ] Token consumption vs contracted quota (underutilization < 50%) +- [ ] User activation (% of seats active) +- [ ] Video/image generation engagement per org +- [ ] Power user drops within org (> 20% decline) +- [ ] Zero activity for 7+ consecutive days +- [ ] Alerts include org name for customer success follow-up +- [ ] Pilot vs contract accounts separated +- [ ] McCann split applied (McCann_NY vs McCann_Paris) +- [ ] Lightricks and Popular Pays excluded + +## 3. Progress Tracker + +* [ ] Read shared knowledge (enterprise segmentation, schema, metrics) +* [ ] Identify enterprise users with segmentation CTE +* [ ] Write monitoring SQL with org-specific baselines +* [ ] Execute query for target date range +* [ ] Analyze results by org and account type +* [ ] Identify power user drops within each org +* [ ] Present findings with org-level details +* [ ] Route alerts to customer success team + +## 4. Implementation Plan + +### Phase 1: Read Alert Thresholds + +**Generic thresholds** (data-driven analysis pending): +- DoD/WoW usage drops > 30% vs org's baseline +- MAU below org's 30-day average +- Token consumption < 50% of contracted quota (underutilization) +- Power user drops > 20% within org +- Zero activity for 7+ consecutive days + +[!IMPORTANT] These are generic thresholds. Consider creating production thresholds based on org-specific analysis. + +### Phase 2: Read Shared Knowledge + +Before writing SQL, read: +- **`shared/product-context.md`** — LTX products, enterprise business model, user types +- **`shared/bq-schema.md`** — Enterprise user segmentation queries (lines 441-461) +- **`shared/metric-standards.md`** — Enterprise metrics, quota tracking +- **`shared/event-registry.yaml`** — Feature events (if analyzing engagement) + +**Data nuances**: +- Use EXACT enterprise segmentation CTE from bq-schema.md (lines 441-461) without modification +- Apply McCann split: `McCann_NY` vs `McCann_Paris` +- Exclude: `Lightricks`, `Popular Pays`, `None` +- Contract accounts: Indegene, HearWell_BeWell, Novig, Cylndr Studios, Miroma, Deriv, McCann_Paris +- Pilot accounts: All other enterprise orgs + +### Phase 3: Identify Enterprise Users + +✅ **PREFERRED: Use exact segmentation CTE from bq-schema.md** + +```sql +WITH ent_users AS ( + SELECT DISTINCT + lt_id, + CASE + WHEN COALESCE(enterprise_name_at_purchase, current_enterprise_name, organization_name) = 'McCann_NY' THEN 'McCann_NY' + WHEN COALESCE(enterprise_name_at_purchase, current_enterprise_name, organization_name) LIKE '%McCann%' THEN 'McCann_Paris' + ELSE COALESCE(enterprise_name_at_purchase, current_enterprise_name, organization_name) + END AS org + FROM `ltx-dwh-prod-processed.web.ltxstudio_users` + WHERE is_enterprise_user + AND current_customer_plan_type IN ('contract', 'pilot') + AND COALESCE(enterprise_name_at_purchase, current_enterprise_name, organization_name) NOT IN ('Lightricks', 'Popular Pays', 'None') +), +enterprise_users AS ( + SELECT DISTINCT + lt_id, + CASE + WHEN org IN ('Indegene', 'HearWell_BeWell', 'Novig', 'Cylndr Studios', 'Miroma', 'Deriv', 'McCann_Paris') + THEN org + ELSE CONCAT(org, ' Pilot') + END AS org, + CASE + WHEN org IN ('Indegene', 'HearWell_BeWell', 'Novig', 'Cylndr Studios', 'Miroma', 'Deriv', 'McCann_Paris') + THEN 'Contract' + ELSE 'Pilot' + END AS account_type + FROM ent_users + WHERE org NOT IN ('Lightricks', 'Popular Pays', 'None') +) +``` + +### Phase 4: Write Monitoring SQL + +✅ **PREFERRED: Monitor all enterprise orgs with org-specific baselines** + +```sql +WITH org_metrics AS ( + SELECT + a.dt, + e.org, + e.account_type, + COUNT(DISTINCT a.lt_id) AS dau, + SUM(a.num_tokens_consumed) AS tokens, + SUM(a.num_generate_image) AS image_gens, + SUM(a.num_generate_video) AS video_gens + FROM `ltx-dwh-prod-processed.web.ltxstudio_agg_user_date` a + JOIN enterprise_users e ON a.lt_id = e.lt_id + WHERE a.dt >= DATE_SUB(CURRENT_DATE(), INTERVAL 30 DAY) + AND a.dt <= DATE_SUB(CURRENT_DATE(), INTERVAL 1 DAY) + GROUP BY a.dt, e.org, e.account_type +), +metrics_with_baseline AS ( + SELECT + *, + AVG(dau) OVER ( + PARTITION BY org + ORDER BY dt ROWS BETWEEN 30 PRECEDING AND 1 PRECEDING + ) AS dau_baseline_30d, + AVG(tokens) OVER ( + PARTITION BY org + ORDER BY dt ROWS BETWEEN 30 PRECEDING AND 1 PRECEDING + ) AS tokens_baseline_30d + FROM org_metrics +) +SELECT * FROM metrics_with_baseline +WHERE dt = DATE_SUB(CURRENT_DATE(), INTERVAL 1 DAY); +``` + +**Key patterns**: +- **Org-specific baselines**: Each org compared to its own 30-day average +- **McCann split**: Separate McCann_NY and McCann_Paris +- **Account type**: Contract vs Pilot + +### Phase 5: Analyze Results + +**For usage trends**: +- Compare org's current usage vs their 30-day baseline +- Flag orgs with DoD/WoW drops > 30% vs baseline +- Flag orgs with MAU below their 30-day average + +**For quota analysis**: +- Compare token consumption vs contracted quota +- Flag underutilization (< 50% of quota) +- Identify orgs approaching or exceeding quota + +**For engagement**: +- Track power users within each org (top 20% by token usage) +- Flag power user drops > 20% within org +- Flag zero activity for 7+ consecutive days + +### Phase 6: Present Findings + +Format results with: +- **Summary**: Key finding (e.g., "Novig enterprise account inactive for 7 days") +- **Org details**: Org name, account type (Contract/Pilot), baseline usage +- **Metrics**: DAU, tokens, generations vs baseline +- **Recommendation**: Customer success action (reach out, investigate, adjust quota) + +**Alert format**: +``` +⚠️ ENTERPRISE ALERT: + • Org: Novig (Contract) + Metric: Token consumption + Current: 0 tokens | Baseline: 27K/day + Drop: -100% | Zero activity for 7 days + +Recommendation: Contact account manager immediately +``` + +### Phase 7: Route Alert + +For ongoing monitoring: +1. Save SQL query +2. Set up in BigQuery scheduled query or Hex Thread +3. Configure notification for customer success team +4. Include org name and account manager contact in alert + +## 5. Context & References + +### Shared Knowledge +- **`shared/product-context.md`** — LTX products, enterprise business model, user types +- **`shared/bq-schema.md`** — Enterprise user segmentation queries (lines 441-461) +- **`shared/metric-standards.md`** — Enterprise metrics, quota tracking +- **`shared/event-registry.yaml`** — Feature events for engagement analysis + +### Data Sources +- **Users table**: `ltx-dwh-prod-processed.web.ltxstudio_users` +- **Usage table**: `ltx-dwh-prod-processed.web.ltxstudio_agg_user_date` +- Key columns: `lt_id`, `enterprise_name_at_purchase`, `current_enterprise_name`, `organization_name`, `current_customer_plan_type` + +### Enterprise Orgs + +**Contract accounts**: +- Indegene +- HearWell_BeWell +- Novig +- Cylndr Studios +- Miroma +- Deriv +- McCann_Paris + +**Pilot accounts**: All other enterprise orgs (suffixed with " Pilot") + +**Excluded**: Lightricks, Popular Pays, None + +## 6. Constraints & Done + +### DO NOT + +- **DO NOT** modify enterprise segmentation CTE — use exact version from bq-schema.md +- **DO NOT** alert on free/self-serve users — this agent is enterprise-only +- **DO NOT** combine McCann_NY and McCann_Paris — keep them separate +- **DO NOT** include Lightricks or Popular Pays in enterprise monitoring +- **DO NOT** use generic baselines — each org compared to its own historical usage + +### DO + +- **DO** use EXACT enterprise segmentation CTE from bq-schema.md (lines 441-461) without modification +- **DO** apply McCann split (McCann_NY vs McCann_Paris) +- **DO** exclude Lightricks, Popular Pays, and None from enterprise orgs +- **DO** break out pilot vs contracted accounts +- **DO** include org name in alert for customer success follow-up +- **DO** use org-specific baselines (each org's 30-day average) +- **DO** flag DoD/WoW usage drops > 30% vs org baseline +- **DO** flag MAU below org's 30-day average +- **DO** flag token consumption < 50% of contracted quota +- **DO** flag power user drops > 20% within org +- **DO** flag zero activity for 7+ consecutive days +- **DO** route alerts to customer success team with org details +- **DO** validate unusual patterns with customer success before alerting + +### Completion Criteria + +✅ All enterprise orgs monitored (Contract and Pilot) +✅ Org-specific baselines applied +✅ McCann split applied (NY vs Paris) +✅ Lightricks and Popular Pays excluded +✅ Alerts include org name and account type +✅ Usage drops, quota issues, and engagement drops detected +✅ Findings routed to customer success team