Separate telemetry handling between CLI and the app by fredrikekelund · Pull Request #2780 · Automattic/studio

fredrikekelund · 2026-03-12T14:26:29Z

Related issues

Fixes STU-1365

How AI was used in this PR

For executing smaller subtasks of the refactor and for planning parts of it.

Proposed Changes

This PR makes a few updates to the telemetry handling in the CLI, in anticipation of distributing the CLI through npm:

Separate the StatsGroup and StatsMetric definitions between the CLI and the app so that they're not shared.
Maintain the shared bumpStat and bumpAggregatedUniqueStat logic.
Create type-safe wrappers for those functions in apps/cli and apps/studio.
Tweak the ConfigFileProvider interface in tools/common/lib/bump-stat.ts to prepare for the separation of CLI and app config files (happening in Move CLI site data to dedicated config file #2731 and related PRs).
Added two new bump stats (anonymized, as always): studio-cli-weekly-unq-npm and studio-cli-weekly-unq-app. These two stats report usage per platform.
Added a __ENABLE_CLI_TELEMETRY__ constant definition that makes it so the regular npm run cli:build output disables telemetry altogether. Only npm run cli:package and npm run cli:build:prod will enable telemetry.

Testing Instructions

Apply the following diff

diff --git a/tools/common/lib/bump-stat.ts b/tools/common/lib/bump-stat.ts
index 105a5324..4ec9683e 100644
--- a/tools/common/lib/bump-stat.ts
+++ b/tools/common/lib/bump-stat.ts
@@ -23,7 +23,7 @@ export function __bumpStat( group: string, stat: string, bumpInDev = false ) {
                return false;
        }

-       if ( process.env.E2E || ( process.env.NODE_ENV === 'development' && ! bumpInDev ) ) {
+       if ( true ) {
                console.info( `Would have bumped stat: ${ group }=${ stat }` );
                return false;
        }

Run npm run cli:build:prod
Run node apps/cli/dist/cli/main.js site list
Ensure that the output contains Would have bumped stat: studio-cli-weekly-unq-app=darwin
Ensure that the appdata file contains a studio-cli-weekly-unq-app key in lastBumpStats

Pre-merge Checklist

Have you checked for TypeScript, React or other console errors?

Add a separate Vite config (vite.config.npm.ts) that externalizes all runtime dependencies and adds a Node.js shebang, producing a standalone CLI package publishable to npm as @automattic/studio-cli. The existing Electron-bundled build (vite.config.ts) is unchanged.

- Remove unused `main` field (CLI-only package) - Include source maps in `files` whitelist - Use `npx --no-install` for deterministic postinstall

…ground#3315)

…ized)

… @wordpress/i18n

…ld-pipeline

Reverts the inline patch-package approach which breaks in CI (patch-package errors on missing packages in workspace context). The script detects monorepo workspaces and skips patching there.

The previous check for node_modules/ existence was unreliable because npm can create the directory with hoisted leftovers even in workspaces. Instead, check if the actual patched packages exist locally.

…encies

…ld-pipeline

…ne-cli-telemetry

wpmobilebot · 2026-03-17T08:46:13Z

📊 Performance Test Results

Comparing 7b0a05a vs trunk

app-size

Metric	trunk	`7b0a05a`	Diff	Change
App Size (Mac)	1236.39 MB	1236.37 MB	0.02 MB	⚪ 0.0%

site-editor

Metric	trunk	`7b0a05a`	Diff	Change
load	1636 ms	1878 ms	+242 ms	🔴 14.8%

site-startup

Metric	trunk	`7b0a05a`	Diff	Change
siteCreation	7096 ms	7089 ms	7 ms	⚪ 0.0%
siteStartup	3927 ms	3931 ms	+4 ms	⚪ 0.0%

Results are median values from multiple test runs.

Legend: 🟢 Improvement (faster) | 🔴 Regression (slower) | ⚪ No change (<50ms diff)

fredrikekelund · 2026-03-17T09:27:46Z

apps/cli/lib/appdata.ts

-		lastBumpStats: z
-			.record( z.string(), z.partialRecord( z.enum( StatsMetric ), z.number() ) )
-			.optional(),
+		lastBumpStats: z.record( z.string(), z.record( z.string(), z.number() ) ).optional(),


I previously advocated for the stricter schema here (#2394 (review)), but it actually causes more trouble than it fixes, given the other changes in this PR. Strictness in runtime types makes sense (i.e., for the function arguments to `bumpStat), but strictness in the appdata schema introduces the risk of runtime exceptions if the config file contains unknown bump stats. That isn't really needed, since we only use this object to check when we last bumped a particular stat.

Anyway, with this change, it's safe to land this PR to trunk even before the config file PR train (#2807 etc) has landed.

bcotrim

Confirmed and works as expected!
Nice separation LGTM 👍

bcotrim · 2026-03-17T09:43:59Z

apps/cli/lib/appdata.ts

 import { readFile, writeFile } from 'atomically';
 import { z } from 'zod';
 import { validateAccessToken } from 'cli/lib/api';
+import { StatsMetric } from 'cli/lib/types/bump-stats';


Unused import here, it should be safe to remove.

ivan-ottinger and others added 30 commits March 12, 2026 08:56

Address Copilot review feedback

81d1bcd

- Remove unused `main` field (CLI-only package) - Include source maps in `files` whitelist - Use `npx --no-install` for deterministic postinstall

Remove redundant web WASM pruning (removed upstream in wordpress-play…

773f2ee

…ground#3315)

Remove node_modules pruning from postinstall script

2cf25ef

Extract shared Vite config into vite.config.base.ts

7707212

Remove @studio/common deps from dependencies (bundled by Vite)

fb7be26

Rename package from @automattic/studio-cli to @automattic/wp-studio

57bfb0e

Simplify externalDeps filter (no-op @studio/common exception removed)

535a343

Fail loudly when patch-package fails during postinstall

760b4b2

Use Rollup banner for shebang to preserve sourcemap accuracy

0786acf

Deduplicate Node builtin externals in npm Vite config

2ba7ede

Sync CLI package version with Studio app version

81c92d1

Remove source maps from npm package

07ae336

Simplify postinstall to inline patch-package command

bccdb50

Remove yargs locale copying (no longer needed since yargs is external…

bdce54e

…ized)

Use defineConfig() in base Vite config for consistency

e55cc43

Clarify why @php-wasm/@wp-playground regex externalization is needed

d39b8b4

Move __STUDIO_CLI_VERSION__ define to shared base config

2da2890

Update dependencies: remove unused @wp-playground/common, add zod and…

8455b4d

… @wordpress/i18n

Update package-lock.json to match package.json

6eacd2d

Merge remote-tracking branch 'origin/trunk' into stu-1362-cli-npm-bui…

e060e40

…ld-pipeline

Restore postinstall script for npm consumers

45c5bb1

Reverts the inline patch-package approach which breaks in CI (patch-package errors on missing packages in workspace context). The script detects monorepo workspaces and skips patching there.

Fix postinstall workspace detection by checking for patched packages

b677102

The previous check for node_modules/ existence was unreliable because npm can create the directory with hoisted leftovers even in workspaces. Instead, check if the actual patched packages exist locally.

Add directly imported @php-wasm and @wp-playground packages to depend…

bcb1fa4

…encies

Merge remote-tracking branch 'origin/trunk' into stu-1362-cli-npm-bui…

8301ec6

…ld-pipeline

Separate bump stat handling between CLI and app

6b756c5

Only enable CLI telemetry in production builds

9bbd8bb

Type-safe wrappers in apps/cli

1d02ed6

Fix bad imports

fd74c4d

Cleanup

dbcb0ee

fredrikekelund self-assigned this Mar 12, 2026

fredrikekelund added 20 commits March 12, 2026 15:31

Naming

d902e60

Fix locking regression

0b3b46a

Only copy node_modules when building CLI for prod

81e1f66

Fix package-lock diff

508ef54

Upgrade Playground packages

b140592

Merge branch 'trunk' into stu-1362-cli-npm-build-pipeline

479e8b9

Fixup

0328355

Even more fixups

2684826

Added README and LICENSE

be9c5e7

Update version

cfb3213

npm pkg fix and remove publishConfig

7e77f61

Add a warning in the README

1396884

Install atomically, fix tree-shaking, upgrade version

2515bf2

Move ora dependency

66eab06

Version

ddb03c8

Define external modules the same way in every build

3193be8

Dynamic command imports to avoid barrel file issue

6908bec

Merge branch 'stu-1362-cli-npm-build-pipeline' into stu-1365-standalo…

03e99d0

…ne-cli-telemetry

Fix circular dependency issue

2edd6dd

Fix

197a919

Base automatically changed from stu-1362-cli-npm-build-pipeline to trunk March 17, 2026 08:18

Merge branch 'trunk' into stu-1365-standalone-cli-telemetry

d097274

More permissive lastBumpStats zod schema

888ae05

fredrikekelund commented Mar 17, 2026

View reviewed changes

bcotrim approved these changes Mar 17, 2026

View reviewed changes

Review suggestion

7b0a05a

fredrikekelund merged commit 81d87fc into trunk Mar 17, 2026
10 checks passed

fredrikekelund deleted the stu-1365-standalone-cli-telemetry branch March 17, 2026 12:14

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Separate telemetry handling between CLI and the app#2780

Separate telemetry handling between CLI and the app#2780
fredrikekelund merged 54 commits intotrunkfrom
stu-1365-standalone-cli-telemetry

fredrikekelund commented Mar 12, 2026 •

edited

Loading

Uh oh!

wpmobilebot commented Mar 17, 2026 •

edited

Loading

Uh oh!

fredrikekelund Mar 17, 2026

Uh oh!

bcotrim left a comment

Uh oh!

bcotrim Mar 17, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

fredrikekelund commented Mar 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Related issues

How AI was used in this PR

Proposed Changes

Testing Instructions

Pre-merge Checklist

Uh oh!

wpmobilebot commented Mar 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📊 Performance Test Results

app-size

site-editor

site-startup

Uh oh!

fredrikekelund Mar 17, 2026

Choose a reason for hiding this comment

Uh oh!

bcotrim left a comment

Choose a reason for hiding this comment

Uh oh!

bcotrim Mar 17, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

fredrikekelund commented Mar 12, 2026 •

edited

Loading

wpmobilebot commented Mar 17, 2026 •

edited

Loading