Genie Usage Tracking

A self-service "My Genie Usage & Cost" dashboard for every user in your Databricks account — without giving anyone access to billing system tables.

Since July 8, 2026, Databricks bills Genie products (Genie Agents, Genie Code, Genie One) pay-as-you-go, with 150 free DBUs of LLM usage per user per month. Paid usage data lands in system.billing.usage — a table you almost certainly don't want to expose to everyone, since it contains the whole account's billing history.

This repo is a Declarative Automation Bundle (formerly Databricks Asset Bundles) that solves the tension: it deploys an AI/BI dashboard that any user can open to see their own Genie consumption and estimated cost — and nothing else. Deployable to any workspace in two commands.

What viewers see

The dashboard (Usage page) shows, always scoped to the person viewing it:

• Overview — billed DBUs and estimated billed cost (list price) for a selectable date range, with daily trends
• Where your usage goes — cost by Genie surface (GENIE_CODE, GENIE_AGENTS, …), by workspace (with names, not just IDs), by Genie agent (which agent burns your budget, with a link to it), and your Genie interactions per day (Agents/One conversations from audit logs, Genie Code messages from system.access.assistant_events)
• This month & your bill — month-to-date billed DBUs and cost (≈ what you pay, since the table records only paid usage — see below), estimated total consumption with the free allowance added back, and a month-end projection at your current pace — an early warning before an admin-set budget blocks you
• History & detail — monthly cost for the last 6 months and an individual usage-records table

A second page (README) explains the security model, the paid-only billing semantics, all data caveats, and a recipe for a personal spend alert — so the dashboard is self-documenting.

How it works

                    deploys & publishes (2 commands)
  ┌─────────────────────────────────────────────────────────────┐
  │  Declarative Automation Bundle                              │
  │                                                             │
  │  UC schema ─── secure views ──── owned by service principal │
  │   (viewer         │                (has system.billing      │
  │    grants)        │                 access; humans don't)   │
  │                   │ WHERE run_as = current_user()           │
  │                   ▼                                         │
  │  AI/BI dashboard, published with VIEWER's OWN credentials   │
  └─────────────────────────────────────────────────────────────┘
                       ▲
        every user opens it, sees only their own rows

Three mechanisms make this safe to share account-wide:

1. Secure views over the system tables (my_genie_usage_with_cost for billing ⨝ prices ⨝ workspace names, my_genie_activity for Genie interaction events from audit logs and assistant_events) filter rows with lower(identity_metadata.run_as) = lower(current_user()), evaluated at query time. Unity Catalog views read their underlying tables with the view owner's rights, so viewers need SELECT on the views only — never on system.billing.* or system.access.*.
2. The dashboard is published with viewer's own credentials (embed_credentials: false, pinned in the bundle config). Queries run as the viewer, so current_user() resolves to them. Publishing with embedded credentials would show the publisher's data to everyone — the bundle makes the safe setting the default and redeploys it on every bundle deploy.
3. Defence-in-depth — the dashboard datasets repeat the current_user() filter in their own WHERE clauses, in case the view definition ever changes.

The views are created and owned by a service principal, so billing-table access never has to be granted to a human.

Quickstart

databricks bundle deploy -t prod          # schema + grants + job + published dashboard
databricks bundle run genie_usage_setup -t prod   # creates the secure views

Share the published dashboard URL with your users. Done — each person's usage appears automatically.

Most workspaces will need a couple of overrides:

databricks bundle deploy -t prod \
  --var="catalog=shared" \
  --var="warehouse_name=My Serverless Warehouse" \
  --var="run_as_service_principal=<your-sp-application-id>"

Variable	Default	Purpose
catalog	main	Existing catalog to hold the schema/view
schema	genie_usage	Schema created and owned by the bundle
viewer_group	account users	UC principal granted access to the views
dashboard_viewer_group	account users	Group granted CAN_READ on the dashboard (falls back: set to users if your workspace rejects account groups in ACLs)
warehouse_name	Serverless Starter Warehouse	Existing SQL warehouse for the dashboard (ID looked up by name)
run_as_service_principal	(required, no default)	Application ID of the SP that runs the setup job and owns the view — pass via --var or export BUNDLE_VAR_run_as_service_principal=...

Prerequisites

• Databricks CLI ≥ 0.281.0 (databricks --version), authenticated to your workspace.
• A service principal for run_as_service_principal. It needs:
  • SELECT on system.billing (an account admin grants this once: GRANT USE SCHEMA, SELECT ON SCHEMA system.billing TO `<sp-application-id>`;),
  • optionally SELECT on system.access — used for workspace names (workspaces_latest) and the per-day interactions chart (audit + assistant_events); each source is probed independently and the views degrade gracefully without it (NULL names, partial or empty activity view),
  • USE CATALOG on the target catalog (schema-level privileges are granted declaratively by the bundle),
  • ideally the right to GRANT USE CATALOG to viewers on that catalog — otherwise the setup job prints the statement for an admin to run.
• The deploying identity needs USE CATALOG + CREATE SCHEMA on the target catalog and the "Service Principal: User" role on the SP (note: Manager alone is not enough to bind run_as).
• Serverless jobs compute enabled (or attach your own cluster to the setup job), and a SQL warehouse for the dashboard.

Dev target

databricks bundle deploy (no -t) uses the dev target: resources are prefixed per-user and isolated in your home folder, so you can iterate safely next to a production deployment. The setup job and dashboard follow the prefixed schema name automatically.

What the numbers mean: paid usage only

The most important subtlety, verified empirically against live billing data at hourly grain:

• Each identified user gets 150 free DBUs of LLM usage per month (≈ $10.50 at US East list price; the dollar value varies by region), resetting on the 1st. See the July 2026 release note.
• system.billing.usage records only paid usage beyond that allowance. Usage inside the free tier is not billed and never appears in the table. We verified this by correlating per-message activity (system.access.assistant_events) with billing at hourly grain: hours of Genie activity at the start of a billing month produce zero billing records, and records switch on mid-day exactly when the allowance runs out.
• Therefore the dashboard's totals are approximately your bill at list price — and the presence of any records in a month means that user's free tier is already exhausted. The "Est. Total Consumption" counter adds the allowance back on top (billed + 150 DBUs × avg unit price; the allowance is a free_dbus dataset parameter, default 150).
• The budgets UI counts the same paid usage. It updates in near real-time while the system table lags a few hours, so small differences between the two are timing, not semantics. A per-user budget threshold applies on top of the free allowance (a $15 budget ≈ $25–28 of total consumption value depending on region); a $0 threshold with Block is effectively a "free tier only" policy.
• Costs are list-price estimates (pricing.effective_list.default); negotiated discounts are not reflected. SQL warehouse compute is billed separately. Service principals get no free allowance — all their usage is billed and recorded.

Personal spend alerts

The dashboard is pull-only; for a push notification each user can create a personal alert on the same secure view (~1 minute). Alerts run as their owner, so the view's current_user() filter makes the alert watch that person's spend only:

1. Workspace → New → Alert, query:

   SELECT COALESCE(SUM(estimated_cost_usd), 0) AS mtd_cost_usd
   FROM <catalog>.<schema>.my_genie_usage_with_cost
   WHERE usage_date >= DATE_TRUNC('MONTH', CURRENT_DATE)

2. Condition — two useful variants, daily schedule, email notification:
   • mtd_cost_usd > 0 — a "my free tier just died" alert: fires the day your first paid record lands. Since usage inside the free allowance is never recorded anywhere, this is the closest thing to a free-tier fuel gauge that exists — not "80/150 used," but "the tank just hit empty," in your inbox within a day.
   • mtd_cost_usd > <threshold> — a spend ceiling (billed cost at list price — roughly what you pay).

The same recipe is on the dashboard's README page, where viewers will actually find it. (One shared alert can't do this per-viewer — that's why it's per-user.)

Truthfulness notes on the insight widgets

• Projections are a linear extrapolation over complete days of the current month — deliberately labeled "rough"; noisy in the first days of a month.
• Interactions per day combines two sources: Genie Agents/One conversation events from audit logs (aibiGenie, genieChat) and Genie Code messages from system.access.assistant_events (Public Preview — one row per user-submitted message; background requests like autocomplete are not logged since May 2026). Genie Code does not log to audit logs, which is why the dedicated table is needed.
• Per-agent breakdown uses usage_metadata.genie.agent_id, which Databricks populates for Genie Agents only; other surfaces show blank agent IDs.

Troubleshooting

Dashboard is empty for a user who definitely used Genie:

1. Still inside the free tier — the most common reason: system.billing.usage records only paid usage, so a user who hasn't exhausted their free 150 DBUs this month has no rows at all. Not a bug — it means they haven't spent anything yet.
2. Billing lag — system.billing.usage can lag up to ~24 h.
3. Nothing before July 8, 2026 — Genie billing records only exist from the pricing change onward.
4. Multiple identities — usage is attributed to the identity the person used Genie with. On Azure especially, one human can map to several identities (e.g. a plain email user@company.com and a guest UPN user_company.com#EXT#@tenant.onmicrosoft.com). current_user() resolves to the login identity, so the dashboard only shows rows recorded under that identity. If a user reports missing data, check which identity their usage is attributed to:

   SELECT identity_metadata.run_as, COUNT(*), SUM(usage_quantity)
   FROM system.billing.usage WHERE billing_origin_product = 'GENIE' GROUP BY ALL

Everyone sees the same (the publisher's) data: the dashboard was republished manually with embedded credentials. Run databricks bundle deploy to restore the safe publish settings, and keep manual republishing on "Don't embed credentials".

Setup job fails reading system.billing.usage: the run-as service principal is missing the billing grant — the job's error message contains the exact GRANT statement for an account admin.

Repository layout

databricks.yml                  # bundle definition, variables, dev/prod targets
resources/
  schema.yml                    # UC schema + declarative viewer/SP grants
  setup_job.yml                 # serverless job that creates the secure views (runs as the SP)
  dashboard.yml                 # dashboard resource (viewer credentials, catalog/schema injection)
src/
  setup_genie_usage_view.py     # parameterized, idempotent setup notebook
  genie_usage.lvdash.json       # dashboard definition (catalog/schema-agnostic)

Customizing

Dashboard: edit the deployed draft dashboard in the UI, then pull changes back into the repo:

databricks bundle generate dashboard --resource genie_usage_dashboard --watch

Keep dataset table references unqualified (FROM my_genie_usage_with_cost) — the bundle injects catalog and schema at deploy time via dataset_catalog/dataset_schema, which is what makes the same JSON portable across workspaces.

Views: the SQL lives in src/setup_genie_usage_view.py. To track other products, adjust the billing_origin_product filter — the per-user security model works for any of them.

Teardown: databricks bundle destroy -t prod removes the schema (including the view), job, and dashboard.

Related docs

• Manage budgets and cost controls for Genie — the query this dashboard is built on, plus how admins enforce (not just observe) spend limits with per-user budget thresholds
• Billable usage system table reference
• Declarative Automation Bundles (formerly Databricks Asset Bundles)

Author & license

Created by Lukasz Maron — LinkedIn · GitHub

MIT licensed — use it, fork it, ship it; just keep the copyright notice.

Contributing

Issues and PRs welcome at github.com/lukaleet/genie-code-usage-tracking. If you deploy this in your workspace and hit an edge case (another cloud, another identity setup, a changed free-tier amount), please open an issue — the goal is that this works everywhere out of the box.