refactor: unify telemetry architecture and use background workers for all telemetry (#3642)

* refactor: unify telemetry architecture and use background workers for all telemetry

This PR refactors the telemetry system to improve performance and consistency:

1. Unified background worker pattern: Both Herokulytics and OTEL/Sentry telemetry
   now use the same background worker process via spawnTelemetryWorker(), preventing
   any telemetry operations from blocking the main CLI process.

2. Consistent class-based architecture: All three telemetry clients
   (BackboardHerokulyticsClient, BackboardOtelClient, SentryClient) now use
   the same class-based pattern with lazy initialization.

3. Improved naming for clarity:
   - analytics.ts → backboard-herokulytics-client.ts (clarifies it sends Herokulytics data)
   - honeycomb-client.ts → backboard-otel-client.ts (clarifies it sends to Backboard, not directly to Honeycomb)
   - Hook files renamed to describe what they do (e.g., collect-and-send-herokulytics.ts)

4. Enhanced debugging: Added comprehensive telemetryDebug logging throughout
   all telemetry clients for easier troubleshooting (enabled via DEBUG=analytics-telemetry).

5. Optimized imports: Moved dynamic imports after telemetry checks to avoid
   loading unnecessary modules when telemetry is disabled.

6. Added isTTY to debug output: Fixed missing isTTY field in OTEL debug logging.

Key architectural improvements:
- Herokulytics (command usage analytics) → Backboard /hamurai endpoint
- OTEL (performance telemetry) → Backboard /otel/v1/traces endpoint → Honeycomb
- Sentry (error reporting) → Sentry.io
- All three systems now use background workers, ensuring zero blocking overhead

* refactor: add explicit type discriminators for telemetry data

Replace implicit type inference based on field presence with explicit _type
discriminators for all telemetry data types. This makes the telemetry worker
more robust and self-documenting.

Changes:
- Add _type: 'herokulytics' to HerokulyticsData interface
- Add _type: 'otel' to Telemetry interface
- Add _type: 'error' to CLIError interface (optional)
- Update telemetry-worker to check _type field instead of inferring from field presence
- Update serializeTelemetryData to add _type: 'error' for Error objects
- Fix OTEL provider to use module-level singleton instead of instance-level
  to avoid global registry conflicts in tests
- Update all test mocks to include _type field

* fix lint

* fix: remove duplicate telemetry send from beforeExit handler

The beforeExit handler in worker-client.ts was causing duplicate telemetry
sends because the postrun hook also sends telemetry for successful command
completion. This resulted in two spans being created for every command.

The fix removes the beforeExit handler for normal command completion, while
keeping the SIGINT/SIGTERM handlers since those bypass the hook lifecycle
and still need to send telemetry.

Now telemetry is sent once via the postrun hook for normal command completion,
and via signal handlers only when the process is interrupted.

* chore: customize telemetry debug namespace and color

Changes the debug namespace from 'analytics-telemetry' to 'heroku:analytics' to match Heroku CLI conventions, and sets a custom color (147) for better visibility in terminal output.

* perf: lazy-load telemetry clients to reduce init hook time

Defers loading of heavy OpenTelemetry and Sentry libraries until they're actually needed (in the background worker process), rather than during CLI initialization. This should significantly reduce the setup-otel-telemetry init hook time.

Before: OpenTelemetry/Sentry libraries loaded during init hook (~199ms)
After: Libraries only loaded when sendTelemetry() is called in worker process

* refactor: convert global-telemetry to singleton class pattern

Refactors the telemetry module to use a singleton class pattern for better encapsulation:
- Renamed global-telemetry.ts to telemetry-manager.ts for clarity
- Converted standalone functions to TelemetryManager class methods
- Exported singleton instance (telemetryManager) for use across codebase
- Updated all imports in hooks and worker to use singleton instance
- Maintained lazy-loading of OpenTelemetry/Sentry clients for performance

This provides better organization and encapsulation of telemetry state while preserving all existing functionality.

* perf: lazy-load heavy dependencies in init path

Optimizes CLI startup time by deferring heavy module loads:

1. Lazy-load fs-extra in terms-of-service hook
   - Only loads when TOS banner needs to be shown (one-time per user)
   - Most users skip this import entirely

2. Lazy-load @heroku-cli/command and @oclif/core/config in getToken()
   - Made getToken() async with dynamic imports
   - These modules only needed in background worker, not during init
   - Updated BackboardOtelClient.ensureInitialized() to be async
   - Updated all call sites and tests

These changes reduce the number of heavy modules loaded during CLI initialization, improving startup performance.

* refactor: convert commands to use lazy-loaded modules

Convert 27 commands to use centralized LazyModuleLoader for improved CLI startup performance:

- Lodash (8 commands): dashboard, apps/info, access, config/edit, apps, releases, ps/type, ps/scale
- Inquirer (8 commands): pipelines/create, pipelines/add, domains/add, apps/transfer, keys/add, certs/add, certs/generate, domains (using @inquirer/prompts)
- Date-fns (5 commands): status, auth/token, certs/auto, data/maintenances/info, data/maintenances/schedule
- Chrono (1 command): data/pg/fork
- Yaml (1 command, from previous session): apps/create

Changes defer heavy npm package imports until command execution time rather than at module parse time, reducing CLI load time.

Skipped data/pg/create and data/pg/update due to complex wrapper prompt patterns.

* refactor: remove deps.ts and rename UserConfig to HerokulyticsConfig

- Delete src/deps.ts (unnecessary indirection layer)
- Move user-config.ts to lib/analytics-telemetry/herokulytics-config.ts
- Rename UserConfig class to HerokulyticsConfig (more descriptive)
- Update backboard-herokulytics-client to import HTTP and HerokulyticsConfig directly
- Update test files to use new import paths and class name
- Fix fork.unit.test.ts to pass chrono parameter to parseRollbackInterval

Since BackboardHerokulyticsClient is already lazy-loaded in the background telemetry worker, deps.ts provided no performance benefit. Direct imports are clearer and simpler.

* feat: add 10 second timeout to telemetry worker process

Add MAX_WORKER_LIFETIME_MS timeout to ensure the background telemetry worker never hangs indefinitely. This prevents the worker from running forever in case of:
- Network request hangs
- OpenTelemetry/Sentry failures
- Other unexpected blocking operations

The worker will now automatically exit after 10 seconds maximum, ensuring no orphaned background processes.

* chore: remove unnecessary lazy-load comments

Remove all 'Lazy-load' comments throughout the codebase. The code is self-documenting with the lazyModuleLoader pattern, making these comments redundant.

* slightly improve version hook

Author: eablack

cspell-dictionary.txt

@@ -115,6 +115,7 @@ hamurai
 herokai
 herokuapp
 herokudns
+herokulytics
 herokumanager
 herokussl
 histfile

package.json

@@ -182,21 +182,21 @@
     "dirname": "heroku",
     "hooks": {
       "command_not_found": [
-        "./dist/hooks/command_not_found/performance_analytics"
+        "./dist/hooks/command_not_found/setup-otel-telemetry"
       ],
       "finally": [
-        "./dist/hooks/finally/sentry"
+        "./dist/hooks/finally/send-otel-and-sentry-errors"
       ],
       "init": [
         "./dist/hooks/init/version",
         "./dist/hooks/init/terms-of-service",
-        "./dist/hooks/init/performance_analytics"
+        "./dist/hooks/init/setup-otel-telemetry"
       ],
       "postrun": [
-        "./dist/hooks/postrun/performance_analytics"
+        "./dist/hooks/postrun/send-otel-telemetry"
       ],
       "prerun": [
-        "./dist/hooks/prerun/analytics"
+        "./dist/hooks/prerun/collect-and-send-herokulytics"
       ],
       "preupdate": [
         "./dist/hooks/preupdate/check-npm-auth"

src/commands/access/index.ts

@@ -3,8 +3,8 @@ import {HerokuAPIError} from '@heroku-cli/command/lib/api-client.js'
 import * as Heroku from '@heroku-cli/schema'
 import {color, hux} from '@heroku/heroku-cli-util'
 import {ux} from '@oclif/core/ux'
-import _ from 'lodash'
 
+import {lazyModuleLoader} from '../../lib/lazy-module-loader.js'
 import {getOwner, isTeamApp} from '../../lib/teamUtils.js'
 
 type AdminWithPermissions = Heroku.TeamMember & {
@@ -28,6 +28,8 @@ export default class AccessIndex extends Command {
   static topic = 'access'
 
   public async run(): Promise<void> {
+    const _ = await lazyModuleLoader.loadLodash()
+
     const {flags} = await this.parse(AccessIndex)
     const {app: appName, json} = flags
     const {body: app} = await this.heroku.get<Heroku.App>(`/apps/${appName}`)
@@ -43,7 +45,7 @@ export default class AccessIndex extends Command {
           admin.permissions = adminPermissions
           return admin
         })
-        collaborators = buildCollaboratorsArray(collaborators, admins)
+        collaborators = buildCollaboratorsArray(collaborators, admins, _)
       } catch (error: any) {
         if (!(error instanceof HerokuAPIError && error.http.statusCode === 403))
           throw error
@@ -53,11 +55,11 @@ export default class AccessIndex extends Command {
     if (json)
       printJSON(collaborators)
     else
-      printAccess(app, collaborators)
+      printAccess(app, collaborators, _)
   }
 }
 
-function buildCollaboratorsArray(collaboratorsRaw: Heroku.TeamAppCollaborator[], admins: Heroku.TeamMember[]) {
+function buildCollaboratorsArray(collaboratorsRaw: Heroku.TeamAppCollaborator[], admins: Heroku.TeamMember[], _: any) {
   const collaboratorsNoAdmins = _.reject(collaboratorsRaw, {role: 'admin'})
   return _.union(collaboratorsNoAdmins, admins)
 }
@@ -82,12 +84,12 @@ function buildTableColumns(showPermissions: boolean) {
   return baseColumns
 }
 
-function printAccess(app: Heroku.App, collaborators: any[]) {
+function printAccess(app: Heroku.App, collaborators: any[], _: any) {
   const showPermissions = isTeamApp(app.owner?.email)
   collaborators = _.chain(collaborators)
-    .sortBy(c => c.email || c.user.email)
-    .reject(c => /herokumanager\.com$/.test(c.user.email))
-    .map(collab => {
+    .sortBy((c: any) => c.email || c.user.email)
+    .reject((c: any) => /herokumanager\.com$/.test(c.user.email))
+    .map((collab: any) => {
       const {email} = collab.user
       const {permissions, role} = collab
       const data: MemberData = {email, role: role || 'collaborator'}

src/commands/apps/create.ts

@@ -9,9 +9,9 @@ import {
 import * as Heroku from '@heroku-cli/schema'
 import {Args, Interfaces, ux} from '@oclif/core'
 import fs from 'fs-extra'
-import {parse} from 'yaml'
 
 import Git from '../../lib/git/git.js'
+import {lazyModuleLoader} from '../../lib/lazy-module-loader.js'
 
 const git = new Git()
 
@@ -193,6 +193,7 @@ ${color.command('heroku apps:create --region eu')}`]
   static hiddenAliases = ['create']
 
   async readManifest() {
+    const {parse} = await lazyModuleLoader.loadYaml()
     const buffer = await fs.readFile('heroku.yml')
     return parse(buffer.toString())
   }

src/commands/apps/index.ts

@@ -3,8 +3,8 @@ import {SpaceCompletion} from '@heroku-cli/command/lib/completions.js'
 import * as Heroku from '@heroku-cli/schema'
 import {color, hux} from '@heroku/heroku-cli-util'
 import {ux} from '@oclif/core/ux'
-import _ from 'lodash'
 
+import {lazyModuleLoader} from '../../lib/lazy-module-loader.js'
 import {App} from '../../lib/types/app.js'
 
 export default class AppsIndex extends Command {
@@ -32,6 +32,8 @@ export default class AppsIndex extends Command {
   static topic = 'apps'
 
   async run() {
+    const _ = await lazyModuleLoader.loadLodash()
+
     const {flags} = await this.parse(AppsIndex)
 
     const teamIdentifier = flags.team
@@ -65,7 +67,7 @@ export default class AppsIndex extends Command {
     if (json) {
       hux.styledJSON(apps)
     } else {
-      print(apps, user, space, team)
+      print(apps, user, space, team, _)
     }
   }
 }
@@ -87,7 +89,7 @@ function listApps(apps: Heroku.App) {
   apps.forEach((app: App) => ux.stdout(regionizeAppName(app)))
 }
 
-function print(apps: Heroku.App, user: Heroku.Account, space?: string, team?: null | string) {
+function print(apps: Heroku.App, user: Heroku.Account, space: string | undefined, team: null | string | undefined, _: any) {
   if (apps.length === 0) {
     if (space) ux.stdout(`There are no apps in space ${color.space(space)}.`)
     else if (team) ux.stdout(`There are no apps in team ${color.team(team)}.`)

src/commands/apps/info.ts

@@ -1,14 +1,96 @@
-import {color, hux} from '@heroku/heroku-cli-util'
 import {Command, flags} from '@heroku-cli/command'
 import * as Heroku from '@heroku-cli/schema'
+import {color, hux} from '@heroku/heroku-cli-util'
 import {Args, ux} from '@oclif/core'
 import {filesize} from 'filesize'
-import _ from 'lodash'
 import * as util from 'util'
 
 import {getGeneration} from '../../lib/apps/generation.js'
+import {lazyModuleLoader} from '../../lib/lazy-module-loader.js'
+
+export default class AppsInfo extends Command {
+  static args = {
+    app: Args.string({hidden: true}),
+  }
+
+  static description = 'show detailed app information'
+  static examples = [
+    color.command('heroku apps:info'),
+    color.command('heroku apps:info --shell'),
+  ]
+
+  static flags = {
+    app: flags.app(),
+    extended: flags.boolean({char: 'x', hidden: true}),
+    json: flags.boolean({char: 'j', description: 'output in json format'}),
+    remote: flags.remote(),
+    shell: flags.boolean({char: 's', description: 'output more shell friendly key/value pairs'}),
+  }
+
+  static help = `$ heroku apps:info
+=== example
+Git URL:   https://git.heroku.com/example.git
+Repo Size: 5M
+...
+
+$ heroku apps:info --shell
+git_url=https://git.heroku.com/example.git
+repo_size=5000000
+...`
+
+  static hiddenAliases = ['info']
+
+  static topic = 'apps'
+
+  async run() {
+    const _ = await lazyModuleLoader.loadLodash()
+
+    const {args, flags} = await this.parse(AppsInfo)
 
-const {countBy, snakeCase} = _
+    const app = args.app || flags.app
+    if (!app) throw new Error('No app specified.\nUSAGE: heroku apps:info --app my-app')
+
+    const info = await getInfo(app, this, flags.extended)
+    const addons = info.addons.map((a: Heroku.AddOn) => a.plan?.name).sort()
+    const collaborators = info.collaborators.map((c: Heroku.Collaborator) => c.user.email)
+      .filter((c: Heroku.Collaborator) => c !== info.app.owner.email)
+      .sort()
+
+    function shell() {
+      function print(k: string, v: string) {
+        ux.stdout(`${_.snakeCase(k)}=${v}`)
+      }
+
+      print('auto_cert_mgmt', info.app.acm)
+      print('addons', addons)
+      print('collaborators', collaborators)
+
+      if (info.app.archived_at) print('archived_at', formatDate(new Date(info.app.archived_at)))
+      if (info.app.cron_finished_at) print('cron_finished_at', formatDate(new Date(info.app.cron_finished_at)))
+      if (info.app.cron_next_run) print('cron_next_run', formatDate(new Date(info.app.cron_next_run)))
+      if (info.app.database_size) print('database_size', filesize(info.app.database_size, {round: 0, standard: 'jedec'}))
+      if (info.app.create_status !== 'complete') print('create_status', info.app.create_status)
+      if (info.pipeline_coupling) print('pipeline', `${info.pipeline_coupling.pipeline.name}:${info.pipeline_coupling.stage}`)
+
+      print('git_url', info.app.git_url)
+      print('web_url', info.app.web_url)
+      print('repo_size', filesize(info.app.repo_size, {round: 0, standard: 'jedec'}))
+      if (getGeneration(info.app) !== 'fir') print('slug_size', filesize(info.app.slug_size, {round: 0, standard: 'jedec'}))
+      print('owner', info.app.owner.email)
+      print('region', info.app.region.name)
+      print('dynos', util.inspect(_.countBy(info.dynos, 'type')))
+      print('stack', info.app.stack.name)
+    }
+
+    if (flags.shell) {
+      shell()
+    } else if (flags.json) {
+      hux.styledJSON(info)
+    } else {
+      print(info, addons, collaborators, flags.extended, _)
+    }
+  }
+}
 
 function formatDate(date: Date) {
   return date.toISOString()
@@ -57,7 +139,7 @@ async function getInfo(app: string, client: Command, extended: boolean) {
   return data
 }
 
-function print(info: Heroku.App, addons: Heroku.AddOn[], collaborators: Heroku.Collaborator[], extended: boolean) {
+function print(info: Heroku.App, addons: Heroku.AddOn[], collaborators: Heroku.Collaborator[], extended: boolean, _: any) {
   const data: Heroku.App = {}
   data.Addons = addons
   data.Collaborators = collaborators
@@ -78,7 +160,7 @@ function print(info: Heroku.App, addons: Heroku.AddOn[], collaborators: Heroku.C
   if (getGeneration(info.app) !== 'fir') data['Slug Size'] = filesize(info.app.slug_size, {round: 0, standard: 'jedec'})
   data.Owner = color.user(info.app.owner.email)
   data.Region = info.app.region.name
-  data.Dynos = countBy(info.dynos, 'type')
+  data.Dynos = _.countBy(info.dynos, 'type')
   data.Stack = (function (app) {
     let stack = info.app.stack.name
     if (app.stack.name !== app.build_stack.name) {
@@ -98,85 +180,3 @@ function print(info: Heroku.App, addons: Heroku.AddOn[], collaborators: Heroku.C
     }
   }
 }
-
-export default class AppsInfo extends Command {
-  static args = {
-    app: Args.string({hidden: true}),
-  }
-
-  static description = 'show detailed app information'
-  static examples = [
-    color.command('heroku apps:info'),
-    color.command('heroku apps:info --shell'),
-  ]
-
-  static flags = {
-    app: flags.app(),
-    extended: flags.boolean({char: 'x', hidden: true}),
-    json: flags.boolean({char: 'j', description: 'output in json format'}),
-    remote: flags.remote(),
-    shell: flags.boolean({char: 's', description: 'output more shell friendly key/value pairs'}),
-  }
-
-  static help = `$ heroku apps:info
-=== example
-Git URL:   https://git.heroku.com/example.git
-Repo Size: 5M
-...
-
-$ heroku apps:info --shell
-git_url=https://git.heroku.com/example.git
-repo_size=5000000
-...`
-
-  static hiddenAliases = ['info']
-
-  static topic = 'apps'
-
-  async run() {
-    const {args, flags} = await this.parse(AppsInfo)
-
-    const app = args.app || flags.app
-    if (!app) throw new Error('No app specified.\nUSAGE: heroku apps:info --app my-app')
-
-    const info = await getInfo(app, this, flags.extended)
-    const addons = info.addons.map((a: Heroku.AddOn) => a.plan?.name).sort()
-    const collaborators = info.collaborators.map((c: Heroku.Collaborator) => c.user.email)
-      .filter((c: Heroku.Collaborator) => c !== info.app.owner.email)
-      .sort()
-
-    function shell() {
-      function print(k: string, v: string) {
-        ux.stdout(`${snakeCase(k)}=${v}`)
-      }
-
-      print('auto_cert_mgmt', info.app.acm)
-      print('addons', addons)
-      print('collaborators', collaborators)
-
-      if (info.app.archived_at) print('archived_at', formatDate(new Date(info.app.archived_at)))
-      if (info.app.cron_finished_at) print('cron_finished_at', formatDate(new Date(info.app.cron_finished_at)))
-      if (info.app.cron_next_run) print('cron_next_run', formatDate(new Date(info.app.cron_next_run)))
-      if (info.app.database_size) print('database_size', filesize(info.app.database_size, {round: 0, standard: 'jedec'}))
-      if (info.app.create_status !== 'complete') print('create_status', info.app.create_status)
-      if (info.pipeline_coupling) print('pipeline', `${info.pipeline_coupling.pipeline.name}:${info.pipeline_coupling.stage}`)
-
-      print('git_url', info.app.git_url)
-      print('web_url', info.app.web_url)
-      print('repo_size', filesize(info.app.repo_size, {round: 0, standard: 'jedec'}))
-      if (getGeneration(info.app) !== 'fir') print('slug_size', filesize(info.app.slug_size, {round: 0, standard: 'jedec'}))
-      print('owner', info.app.owner.email)
-      print('region', info.app.region.name)
-      print('dynos', util.inspect(countBy(info.dynos, 'type')))
-      print('stack', info.app.stack.name)
-    }
-
-    if (flags.shell) {
-      shell()
-    } else if (flags.json) {
-      hux.styledJSON(info)
-    } else {
-      print(info, addons, collaborators, flags.extended)
-    }
-  }
-}

src/commands/apps/transfer.ts

@@ -2,10 +2,10 @@ import {Command, flags} from '@heroku-cli/command'
 import * as Heroku from '@heroku-cli/schema'
 import * as color from '@heroku/heroku-cli-util/color'
 import {Args, ux} from '@oclif/core'
-import inquirer from 'inquirer'
 import tsheredoc from 'tsheredoc'
 
 import {appTransfer} from '../../lib/apps/app-transfer.js'
+import {lazyModuleLoader} from '../../lib/lazy-module-loader.js'
 import ConfirmCommand from '../../lib/confirmCommand.js'
 import {getOwner, isTeamApp, isValidEmail} from '../../lib/teamUtils.js'
 import AppsLock from './lock.js'
@@ -36,7 +36,7 @@ export default class AppsTransfer extends Command {
 
   static topic = 'apps'
 
-  getAppsToTransfer(apps: Heroku.App[]) {
+  getAppsToTransfer(apps: Heroku.App[], inquirer: any) {
     return inquirer.prompt([{
       choices: apps.map(app => ({
         name: `${color.app(app.name ?? '')} (${getOwner(app.owner?.email ?? '')})`, value: {name: app.name, owner: app.owner?.email},
@@ -49,12 +49,14 @@ export default class AppsTransfer extends Command {
   }
 
   public async run() {
+    const inquirer = await lazyModuleLoader.loadInquirer()
+
     const {args, flags} = await this.parse(AppsTransfer)
     const {app, bulk, confirm, locked} = flags
     const {recipient} = args
     if (bulk) {
       const {body: allApps} = await this.heroku.get<Heroku.App[]>('/apps')
-      const selectedApps = await this.getAppsToTransfer(allApps.sort((a, b) => (a.name ?? '').localeCompare(b.name ?? '')))
+      const selectedApps = await this.getAppsToTransfer(allApps.sort((a, b) => (a.name ?? '').localeCompare(b.name ?? '')), inquirer)
       ux.warn(`Transferring applications to ${color.name(recipient)}...\n`)
       for (const app of selectedApps.choices) {
         try {