[WIP] Add anonymous metrics tracking (RFC - API not yet implemented)

[stbenjam]

Mock-up for collecting anonymous usage metrics on AI helper commands. Backend API isn't implemented yet.

Installs a UserPromptSubmit hook to look for slash commands, and sends them to our anonymous metrics usage tracking service.

The user has to opt-in to the plugin to make this work, though.

Example of what's sent:

[2025-10-21T14:33:53+00:00Z] Sending metrics: {"type": "slash_command", "name": "ci:ask-sippy", "engine": "claude", "version": "1.0", "timestamp": "2025-10-21T14:33:53+00:00Z", "session_id": "abbba331-eeff-41d1-89d5-b57f206defb5", "os": "darwin", "mac": "9ff1f8981086488424e994a57a50789e23a996b03294ec44bb5a18b428dce144", "prompt_length": 25}

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: stbenjam

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [stbenjam]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[enxebre]

thanks, this is great! dropped some feedback.

[stbenjam]

The hooks approach works, but requires the user to install the plugin.

[theobarberbany]

The hooks approach works, but requires the user to install the plugin.
Hmm - could we use team plugins?

To set up team plugins:
Add marketplace and plugin configuration to your repository’s .claude/settings.json
Team members trust the repository folder
Plugins install automatically for all team members

[stbenjam]

The hooks approach works, but requires the user to install the plugin.
Hmm - could we use team plugins?

To set up team plugins:
Add marketplace and plugin configuration to your repository’s .claude/settings.json
Team members trust the repository folder
Plugins install automatically for all team members

This sounds like more work than just installing the metrics plugin (more or less a one-liner), as teams would have to clone ai-helpers vs. consuming it from github, setup trust, etc right?

[theobarberbany]

Ah - that wasn't how I was thinking it could be used.

I'm not sure team members would have to clone the ai-helpers repo (I've also not tested this :) ).

I think you can specify plugins e.g in team owned repos, so one person PRs these changes to.claude-plugin/marketplace.json and sections in .claude/settings.json that specify we want to have the metrics tracking plugin?

This way, If I'm reading the docs right you can opt in a repository vs per user?

I'm skimming https://skywork.ai/blog/claude-code-plugin-standardization-team-guide/

but something like

cat .claude/settings.json
{
  "extraKnownMarketplaces": [
    {
      "name": "org-standard",
      "url": "https://raw.githubusercontent.com/openshift-eng/ai-helpers/main/.claude-plugin/marketplace.json",
      "trusted": true
    }
  ],
  "enabledPlugins": [
    "[email protected]",
  ],
  "requireRepoTrust": true,
  "notes": "Pin plugin versions to ensure reproducible behavior across contributors."
}

cat .claude-plugin/marketplace.json
{
  "name": "ai-helpers",
  "description": "Curated, approved Claude Code plugins for org-wide use.",
  "plugins": [
    {
      "id": "ai-helpers.metrcs",
      "version": "1.2.3",
      "source": {
        "type": "git",
        "url": "https://github.com/openshift-eng/main/plugins/metrics",
        "ref": "v1.2.3"
      }
    }
  ]
}

[openshift-merge-robot]

PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.