Epic: Integrate User Metrics and Interaction Tracking

[enrico-kaack-comp]

Understand the Epic

We want to gain insights in:

• What are users doing with our tool
  • think patterns, sessions etc
• How are Features used
  • how do people interact, where do they click to accomplish certain action
• performance of the ui
  • network, rendering, user.. where is time spend
• errors in the frontend
  • crash logs

This user metrics and interaction tracking should be done in a privacy preserving way and HAS to comply with data protection standard etc.

Context / Background

We want to develop the UI along the users request and feedback. Beside direct feedback we can also profit from this kind of tracking to see what parts of the UI we need to focus on.

Our webapp may run in various setups and contexts.
Already today, we are offering service internally, we monitor SAP users, approved tool: Dynatrace (BTP hosted)
As we are offering our tooling open-source, Dynatrace is not possible. We probably want to support open-source tools.

Does a plug-in/mechanism make sense to support different reporting/metrics.

User Stories or tasks

• Define what metrics we want to collect and what purpose they serve
• Define a concept how to integrate this tracking into the open-source project
• Research a suitable storage/analytic solution
• Implement the tracking solution into the frontend
• Setup storage/analytic solution
• Ensure compliance (e.g. privacy policy, user opt out etc.)

What is required to accept the Epic as finished.

• Defined metrics can be extracted from the system

Dependencies of this Epic

No response

Risks of this Epic

No response

Known Stakeholders of this Epic

No response

Milestone Definitions for this Epic.

TBD

Metrics Definition for this Epic.

TBD

Any further valuable resources.

No response

[GenosseOtt]

I am advertising https://plausible.io/ here ;-) Please consider

[andreaskienle]

I think it might make sense to look at analytics as two independent layers

1. Browser-side instrumentation (events, Web Vitals, errors, performance traces, …) running directly as JS in the browser
2. Storage and dashboards centralised somewhere

There are different approaches

• Separate browser-side instrumentation from storage and dashboards:

  • OpenTelemetry (OTel) gives us an open standard for this
  • An OTel collector (self-hosted or SaaS) can forward the data to Dynatrace
    • I think we already have Dynatrace somewhere (?), I remember a presentation by Max – do we want to integrate our data?
  • The UI code stays vendor-agnostic; self-hosters only need to edit a ⁠config to decide where their data goes.

• “All-in-one” service such as Sentry, Plausible, Adobe Analytics (was mentioned by the OSPO)

  • These ship SDK, storage and a dashboard UI in one box, so the implementation is easier
  • But we lock ourselves to that vendor (licence/hosting costs, etc.)
    • could maybe be mitigated with canonical events and separating them a bit from the SDK provider
    • The SDK provider will be part of the frontend coding, however, and supporting a different provider will involve adding another SDK
  • The OSPO considers this acceptable; if other deployments ever want a different provider they can contribute an additional adapter later

[enrico-kaack-comp]

Ideas of metrics to collect:

Metrics

General

• Unique Users per Week: How many users do we have?
• Amount of sessions (= start to end of interaction of user): How often is our UI used? How often do users use our webapp?
• User Client (Mobile/Desktop)
• Active User Session Time (as in tab is active): Average, Median, Percentile etc....

Performance

• Time to List Load: Project List, Workspace List, Workspace List with all MCP
• Time to MCP Detail Load

Behaviour

Should be related to the Amount of Sessions to get a feeling for how our UI is mainly used.

• Use of Create Project Dialog
• Use of Create Workspace
• Use of Create MCP Dialog
• Status View
• Create MCP Button Click
• Kubeconfig Download
• Show Kubeconfig Command Window