[P&S Initiative] Collect user session length and count metrics

[yuvipanda]

Note: This initiative is not complete, the only issue defined should give us more info on what else needs to be done once that is complete.

For https://github.com/2i2c-org/meta/issues/1814, we need to provide an easily explainable set of numbers that BD can use to talk to communities. We currently provide 'active users' only. We would like to provide:

1. Total number of user sessions
2. Median length of each user session
3. 99th percentile length of user sessions

This fundamentally can be expressed as a prometheus histogram, although we don't have the ways to provide that yet.

Problem

This is a little difficult to do currently with the information we have in prometheus, because we don't really have any raw data on sessions. Session presence or absence is calculated by the presence or absence of metrics (like kube_pod_status_phase) which makes this difficult. While we can do things like 'number of running servers at any given instant' easily, it's more difficult to do 'number of running servers today'. It's even more difficult to do things like 'median session length of users today'. A useful way to think of this is that promql excels at 'map' operations and 'filter' operations, but struggles with 'reduce' operations.

It's possible that someone with more promql knowledge than me could do this, but I've asked around and haven't necessarily found solutions we can actually trust.

The last time we tried to do this, we ended up adding these metrics to JupyterHub directly so we could trust the active user counts: jupyterhub/jupyterhub#4214

Possible solutions

1. promql + Grafana based this off our existing metrics. This may include using a Prometheus recording rule to create a new timeseries
2. Add this to JupyterHub as a metric, same as we did last time. However, that was possible because JupyterHub already tracked the one thing that was needed - users and when they were last active. While Spawner (the ORM object) in JupyterHub does have started and last_activity timestamps, these are reset whenever a user's server stops, so we can't really use them here. Also, it doesn't contain historical information - we can only know about the last server start / stop, nothing before
3. Collect new metrics from the user pods. In particular, collect prometheus: Expose 3 activity metrics jupyter-server/jupyter_server#1471 and then do solution 1.
4. Write our own prometheus exporter in python, that can treat prometheus as a source of data, compute the additional info we want, and export it for collection again by prometheus. This we know will work, so if we can't do it through the other options we do this.

Path forward

We'll experiment with 2i2c-org/meta#2544 to figure out if we can get at least active user session counts set up, and if not, what additional work we would need to do.

[yuvipanda]

I just spent some time with @colliand talking about this.

In the interest of faster iteration and keeping our eye on the goals, we have decided to do two things:

1. Time to First Value can be simply measured using our current MAU counts. When the first 5 users log in (for example), the MAU goes to 5.
2. The core way @colliand wants to use this, and the reason for BD to drive on the engagement KPI, is to be able to sit together with a community during renewal conversations, and explore how they have used it using Grafana. The major problem with this is that our Grafana is currently very inward facing - it's focused on diagnostics rather than reporting. So admins find it hard to use and so does BD. The way around this is to have an initiative be focused on improving Grafana to be more admin facing friendly than engineer friendly as it is now.

(1) now unblocks the quarterly goal of measuring TFFV, and (2) sets up a potential goal for next quarter. While the engagement KPI measured out in this issue body is still helpful, because it's no longer blocking the TFFV goal, I recommend we stop working on this initiative inside P&S and focus on other goals.

[colliand]

I agree with what Yuvi just posted. I want to add some nuance.

Yes, the core way I want this is to answer customer questions in the run-up to renewal like "Is there any evidence that people in my community are actually using this?" and to address "Yeah, those are just clicks. Are they actually staying logged in and doing research?". There are powerful ways to answer these kinds of questions using things we've already built up with Prometheus and Grafana. So the conversion from diagnostics to reporting is a good thrust.

I'd also like to share Grafana dashboard(s) designed for direct use by JupyterHub admins (or buyers). I'd like people who pay us to be able to see that their community is actually using the platform.

The discussion with Yuvi also revealed to me a process challenge. My desire to "help buyers see that people are using the things they bought from 2i2c" somehow coarsened that into a "build an engagement KPI". This propagated through our systems to the request in this issue that appears to have been more technically challenging and not on-point for what I initially wanted. We have a game of telephone problem.