Skip to content

feat: add enterprise account monitoring agent #45

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Assaf Hay Eden (AssafHayEden) wants to merge 1 commit into
base: main
Choose a base branch
from
Open

feat: add enterprise account monitoring agent #45

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
310 changes: 249 additions & 61 deletions agents/monitoring/enterprise/SKILL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,67 +1,255 @@
---
name: enterprise-monitor
description: Monitors enterprise account health, usage, and contract compliance. Alerts on low engagement, quota breaches, or churn risk.
tags: [monitoring, enterprise, accounts, contracts]
description: "Monitor enterprise account health, usage, and contract compliance. Detects low engagement, quota breaches, and churn risk. Use when: (1) detecting enterprise account usage drops, (2) alerting on inactive accounts, (3) investigating power user engagement changes."
tags: [monitoring, enterprise, accounts, contracts, churn-risk]
---

# Enterprise Monitor

## When to use

- "Monitor enterprise account usage"
- "Monitor enterprise churn risk"
- "Alert when enterprise account is inactive"

## What it monitors

- **Account usage**: DAU, WAU, MAU per enterprise org
- **Token consumption**: Usage vs contracted quota, historical consumption trends
- **User activation**: % of seats active
- **Engagement**: Video generations, image generations, downloads per org
- **Churn signals**: Declining usage, inactive users

## Steps

1. **Gather requirements from user:**
- Which enterprise org(s) to monitor (or all)
- Alert threshold based on historical usage of each account (e.g., "usage drops > 30% vs their baseline", "MAU below their 30-day average", "< 50% of contracted quota")
- Time window (weekly, monthly)
- Notification channel (Slack, email, Linear issue)

2. **Read shared files:**
- `shared/product-context.md` — LTX products, enterprise business model, user types
- `shared/bq-schema.md` — Enterprise user segmentation queries
- `shared/metric-standards.md` — Enterprise metrics, quota tracking
- `shared/event-registry.yaml` — Feature events (if analyzing engagement)
- `shared/gpu-cost-query-templates.md` — GPU cost queries (if analyzing infrastructure costs)
- `shared/gpu-cost-analysis-patterns.md` — Cost analysis patterns (if analyzing infrastructure costs)

3. **Identify enterprise users:**
- Use enterprise segmentation CTE from bq-schema.md (lines 441-461)
- Apply McCann split (McCann_NY vs McCann_Paris)
- Exclude Lightricks and Popular Pays

4. **Write monitoring SQL:**
- Query org-level usage metrics
- Set baseline for each org based on their historical usage (e.g., 30-day average, 90-day trend)
- Compare current usage against org-specific baseline or contracted quota
- Flag orgs below threshold or showing decline
- Flag meaningful drops for power users (users with top usage within each org)

5. **Present to user:**
- Show SQL query
- Show example alert format with org name and metrics
- Confirm threshold values and alert logic

6. **Set up alert** (manual for now):
- Document SQL
- Configure notification to customer success team

## Rules

- DO use EXACT enterprise segmentation CTE from bq-schema.md without modification
- DO apply McCann split (McCann_NY vs McCann_Paris)
- DO exclude Lightricks and Popular Pays from enterprise orgs
- DO break out pilot vs contracted accounts
- DO NOT alert on free/self-serve users — this agent is enterprise-only
- DO include org name in alert for easy customer success follow-up
## 1. Overview (Why?)

Enterprise accounts (contract and pilot) represent high-value customers with negotiated quotas and specific engagement patterns. Unlike self-serve users, enterprise usage should be monitored per-organization with org-specific baselines, since each has different team sizes, use cases, and contract terms.

This skill provides **autonomous enterprise account monitoring** that detects declining usage, underutilization of quotas, inactive periods, and power user drops — all of which signal churn risk or engagement problems that require customer success intervention.

**Problem solved**: Identify enterprise churn risk early through usage signals — before contracts end or accounts go completely inactive — with org-level root cause analysis.

## 2. Requirements (What?)

Monitor these outcomes autonomously:

- [ ] DAU/WAU/MAU drops per enterprise org (> 30% vs org baseline)
- [ ] Token consumption vs contracted quota (underutilization < 50%)
- [ ] User activation (% of seats active)
- [ ] Video/image generation engagement per org
- [ ] Power user drops within org (> 20% decline)
- [ ] Zero activity for 7+ consecutive days
- [ ] Alerts include org name for customer success follow-up
- [ ] Pilot vs contract accounts separated
- [ ] McCann split applied (McCann_NY vs McCann_Paris)
- [ ] Lightricks and Popular Pays excluded

## 3. Progress Tracker

* [ ] Read shared knowledge (enterprise segmentation, schema, metrics)
* [ ] Identify enterprise users with segmentation CTE
* [ ] Write monitoring SQL with org-specific baselines
* [ ] Execute query for target date range
* [ ] Analyze results by org and account type
* [ ] Identify power user drops within each org
* [ ] Present findings with org-level details
* [ ] Route alerts to customer success team

## 4. Implementation Plan

### Phase 1: Read Alert Thresholds

**Generic thresholds** (data-driven analysis pending):
- DoD/WoW usage drops > 30% vs org's baseline
- MAU below org's 30-day average
- Token consumption < 50% of contracted quota (underutilization)
- Power user drops > 20% within org
- Zero activity for 7+ consecutive days

[!IMPORTANT] These are generic thresholds. Consider creating production thresholds based on org-specific analysis.

### Phase 2: Read Shared Knowledge

Before writing SQL, read:
- **`shared/product-context.md`** — LTX products, enterprise business model, user types
- **`shared/bq-schema.md`** — Enterprise user segmentation queries (lines 441-461)
- **`shared/metric-standards.md`** — Enterprise metrics, quota tracking
- **`shared/event-registry.yaml`** — Feature events (if analyzing engagement)

**Data nuances**:
- Use EXACT enterprise segmentation CTE from bq-schema.md (lines 441-461) without modification
- Apply McCann split: `McCann_NY` vs `McCann_Paris`
- Exclude: `Lightricks`, `Popular Pays`, `None`
- Contract accounts: Indegene, HearWell_BeWell, Novig, Cylndr Studios, Miroma, Deriv, McCann_Paris
- Pilot accounts: All other enterprise orgs

### Phase 3: Identify Enterprise Users

✅ **PREFERRED: Use exact segmentation CTE from bq-schema.md**

```sql
WITH ent_users AS (
SELECT DISTINCT
lt_id,
CASE
WHEN COALESCE(enterprise_name_at_purchase, current_enterprise_name, organization_name) = 'McCann_NY' THEN 'McCann_NY'
WHEN COALESCE(enterprise_name_at_purchase, current_enterprise_name, organization_name) LIKE '%McCann%' THEN 'McCann_Paris'
ELSE COALESCE(enterprise_name_at_purchase, current_enterprise_name, organization_name)
END AS org
FROM `ltx-dwh-prod-processed.web.ltxstudio_users`
WHERE is_enterprise_user
AND current_customer_plan_type IN ('contract', 'pilot')
AND COALESCE(enterprise_name_at_purchase, current_enterprise_name, organization_name) NOT IN ('Lightricks', 'Popular Pays', 'None')
),
enterprise_users AS (
SELECT DISTINCT
lt_id,
CASE
WHEN org IN ('Indegene', 'HearWell_BeWell', 'Novig', 'Cylndr Studios', 'Miroma', 'Deriv', 'McCann_Paris')
THEN org
ELSE CONCAT(org, ' Pilot')
END AS org,
CASE
WHEN org IN ('Indegene', 'HearWell_BeWell', 'Novig', 'Cylndr Studios', 'Miroma', 'Deriv', 'McCann_Paris')
THEN 'Contract'
ELSE 'Pilot'
END AS account_type
FROM ent_users
WHERE org NOT IN ('Lightricks', 'Popular Pays', 'None')
)
```

### Phase 4: Write Monitoring SQL

✅ **PREFERRED: Monitor all enterprise orgs with org-specific baselines**

```sql
WITH org_metrics AS (
SELECT
a.dt,
e.org,
e.account_type,
COUNT(DISTINCT a.lt_id) AS dau,
SUM(a.num_tokens_consumed) AS tokens,
SUM(a.num_generate_image) AS image_gens,
SUM(a.num_generate_video) AS video_gens
FROM `ltx-dwh-prod-processed.web.ltxstudio_agg_user_date` a
JOIN enterprise_users e ON a.lt_id = e.lt_id
WHERE a.dt >= DATE_SUB(CURRENT_DATE(), INTERVAL 30 DAY)
AND a.dt <= DATE_SUB(CURRENT_DATE(), INTERVAL 1 DAY)
GROUP BY a.dt, e.org, e.account_type
),
metrics_with_baseline AS (
SELECT
*,
AVG(dau) OVER (
PARTITION BY org
ORDER BY dt ROWS BETWEEN 30 PRECEDING AND 1 PRECEDING
) AS dau_baseline_30d,
AVG(tokens) OVER (
PARTITION BY org
ORDER BY dt ROWS BETWEEN 30 PRECEDING AND 1 PRECEDING
) AS tokens_baseline_30d
FROM org_metrics
)
SELECT * FROM metrics_with_baseline
WHERE dt = DATE_SUB(CURRENT_DATE(), INTERVAL 1 DAY);
```

**Key patterns**:
- **Org-specific baselines**: Each org compared to its own 30-day average
- **McCann split**: Separate McCann_NY and McCann_Paris
- **Account type**: Contract vs Pilot

### Phase 5: Analyze Results

**For usage trends**:
- Compare org's current usage vs their 30-day baseline
- Flag orgs with DoD/WoW drops > 30% vs baseline
- Flag orgs with MAU below their 30-day average

**For quota analysis**:
- Compare token consumption vs contracted quota
- Flag underutilization (< 50% of quota)
- Identify orgs approaching or exceeding quota

**For engagement**:
- Track power users within each org (top 20% by token usage)
- Flag power user drops > 20% within org
- Flag zero activity for 7+ consecutive days

### Phase 6: Present Findings

Format results with:
- **Summary**: Key finding (e.g., "Novig enterprise account inactive for 7 days")
- **Org details**: Org name, account type (Contract/Pilot), baseline usage
- **Metrics**: DAU, tokens, generations vs baseline
- **Recommendation**: Customer success action (reach out, investigate, adjust quota)

**Alert format**:
```
⚠️ ENTERPRISE ALERT:
• Org: Novig (Contract)
Metric: Token consumption
Current: 0 tokens | Baseline: 27K/day
Drop: -100% | Zero activity for 7 days

Recommendation: Contact account manager immediately
```

### Phase 7: Route Alert

For ongoing monitoring:
1. Save SQL query
2. Set up in BigQuery scheduled query or Hex Thread
3. Configure notification for customer success team
4. Include org name and account manager contact in alert

## 5. Context & References

### Shared Knowledge
- **`shared/product-context.md`** — LTX products, enterprise business model, user types
- **`shared/bq-schema.md`** — Enterprise user segmentation queries (lines 441-461)
- **`shared/metric-standards.md`** — Enterprise metrics, quota tracking
- **`shared/event-registry.yaml`** — Feature events for engagement analysis

### Data Sources
- **Users table**: `ltx-dwh-prod-processed.web.ltxstudio_users`
- **Usage table**: `ltx-dwh-prod-processed.web.ltxstudio_agg_user_date`
- Key columns: `lt_id`, `enterprise_name_at_purchase`, `current_enterprise_name`, `organization_name`, `current_customer_plan_type`

### Enterprise Orgs

**Contract accounts**:
- Indegene
- HearWell_BeWell
- Novig
- Cylndr Studios
- Miroma
- Deriv
- McCann_Paris

**Pilot accounts**: All other enterprise orgs (suffixed with " Pilot")

**Excluded**: Lightricks, Popular Pays, None

## 6. Constraints & Done

### DO NOT

- **DO NOT** modify enterprise segmentation CTE — use exact version from bq-schema.md
- **DO NOT** alert on free/self-serve users — this agent is enterprise-only
- **DO NOT** combine McCann_NY and McCann_Paris — keep them separate
- **DO NOT** include Lightricks or Popular Pays in enterprise monitoring
- **DO NOT** use generic baselines — each org compared to its own historical usage

### DO

- **DO** use EXACT enterprise segmentation CTE from bq-schema.md (lines 441-461) without modification
- **DO** apply McCann split (McCann_NY vs McCann_Paris)
- **DO** exclude Lightricks, Popular Pays, and None from enterprise orgs
- **DO** break out pilot vs contracted accounts
- **DO** include org name in alert for customer success follow-up
- **DO** use org-specific baselines (each org's 30-day average)
- **DO** flag DoD/WoW usage drops > 30% vs org baseline
- **DO** flag MAU below org's 30-day average
- **DO** flag token consumption < 50% of contracted quota
- **DO** flag power user drops > 20% within org
- **DO** flag zero activity for 7+ consecutive days
- **DO** route alerts to customer success team with org details
- **DO** validate unusual patterns with customer success before alerting

### Completion Criteria

✅ All enterprise orgs monitored (Contract and Pilot)
✅ Org-specific baselines applied
✅ McCann split applied (NY vs Paris)
✅ Lightricks and Popular Pays excluded
✅ Alerts include org name and account type
✅ Usage drops, quota issues, and engagement drops detected
✅ Findings routed to customer success team