|
7 | 7 | <hr /> |
8 | 8 |
|
9 | 9 | ## Objective |
10 | | -Crashes are already tracked on Sentry. Performance metrics are also useful health metrics to check. This can be tracked by Sentry too. |
| 10 | +Crashes are already tracked on Sentry. Performance metrics are also useful health metrics to check. They can be tracked by Sentry too. |
11 | 11 |
|
12 | | -This document specifies what metrics we track and how we measure them. It has a focus on EX but it could be extended (and renamed) to track any matrix client app performance. |
| 12 | +This document specifies what metrics we track and how we measure them. It focuses on EX but it could be extended (and renamed) to track any matrix client app performance. |
13 | 13 |
|
14 | 14 | ## UX metrics |
15 | 15 |
|
16 | | - |
17 | | -| Title | Android name / Description | What it is measuring | Initial and final conditions | Tech metrics | Notes | |
| 16 | +| Title | Android name / Description | What is it measuring? | Initial and final conditions | Tech metrics | Notes | |
18 | 17 | | :---- | :---- | :---- | :---- | :---- | :---- | |
19 | | -| Cold start time | Cold start until cached room list is displayed | How long it takes from launching the app to displaying cached data on the screen | From:<ul><li>The user is already connected</li><li>The app is not running in background User taps on the app icon</li></ul> To:<ul><li>The room list is fully loaded from the permanent cached and displayed to the end user | <ul><li>First rooms displayed after login or restoration (TBD) </li></ul> | It does not consider Sentry init time and the check that the user has consented from disk. | |
20 | | -| Catch-up | Room list says it is resumed and running | How long until the room list is up-to-date | From:<ul><li>The app was inactive in background and get debackgrounded</li><li>Or the app just finished its cold start</li></ul> To:<ul><li>The room list service turns into `Running` No more Syncing spinner | | The expected final conditions should be:<ul><li>The room list is updated</li><li>Properly sorted</li><li>Last messages are up-to-date</li></ul> But we need more work to get it | |
21 | | -| Notification to message | A notification was tapped and it opened a timeline | How long it takes from taping the notification to the message becoming visible in the app | From:<ul><li>The app is in background or active The user taps on a notification</li></ul> To:<ul><li>The message is visible in the timeline The timeline around this message is fully loaded | | | |
| 18 | +| Cold start | Cold start until the cached room list is displayed | How long it takes from launching the app to displaying cached data on the screen | From:<ul><li>The user is already connected</li><li>The app is not running in background</li><li> The user taps on the app icon</li></ul> To:<ul><li>The room list is fully loaded from the permanent cached and displayed to the end user | <ul><li>First rooms displayed after login or restoration (**TBD**) </li></ul> | The clock starts after the Sentry is initialised. | |
| 19 | +| Catch-up | The app syncs and the room list becomes up-to-date | How long until the room list is up-to-date | From:<ul><li>The app was inactive in background and get debackgrounded</li><li>Or the app just finished its cold start</li></ul> To:<ul><li>The room list service state becomes [`Running`](https://github.com/matrix-org/matrix-rust-sdk/blob/matrix-sdk-ui-0.14.0/bindings/matrix-sdk-ffi/src/sync_service.rs#L36)</li><li>No more Syncing spinner | | The expected final conditions should be:<ul><li>The room list is updated</li><li>Properly sorted</li><li>Last messages are up-to-date</li></ul> But we need more work to compute this metric | |
| 20 | +| Notification to message | A notification was tapped and it opened a timeline | How long it takes from taping the notification to the message becoming visible in the app | From:<ul><li>The app is in background or active</li><li>The user taps on a notification</li></ul> To:<ul><li>The message is visible in the timeline</li><li>The timeline around this message is fully loaded | | | |
22 | 21 | | Open a room | Open a room and see loaded items in the timeline | How long it takes from tapping a room in the room list to displaying a full page of messages | From:<ul><li>User taps a room from the room list</li></ul> To:<ul><li>The timeline is fully loaded with first items loaded | <ul><li>Get and display first timeline items</li></ul> | | |
23 | | -| Send a message | ? | How long it takes from tapping send to the message appearing in the timeline as sent | From:<ul><li>User hits the send button</li></ul> To:<ul><li>The timeline shows it as sent | | Needs to be tested for validation | |
24 | | - |
| 22 | +| Send a message | Send to sent state in timeline | How long it takes from tapping send to the message appearing in the timeline as sent | From:<ul><li>User hits the send button</li></ul> To:<ul><li>The timeline shows it as sent | | **TODO**: We need to experiment the feasibility | |
25 | 23 |
|
26 | 24 | ## Additional data |
27 | 25 |
|
28 | 26 | We need to add some data to the metrics so that we can better characterise them: |
29 | 27 |
|
30 | | -| Data | Why it is useful | Notes | |
| 28 | +| Data | Why is it useful? | Notes | |
31 | 29 | | :---- | :---- | :---- | |
32 | | -| Device information | <ul><li>It helps to detect specific device problem, especially on the fragmented Android ecosystem</li></ul> | It is provided by Sentry. No additional work | |
33 | | -| Homeserver URL hash | <ul><li>To compare matrix.org and element.io homeservers speeds</li><li>Measure the impact of a slow server</li></ul> | We use SHA-512 for the hash | |
34 | | -| DBs file size | <ul><li>Check the impact of growing DB on speed performance</li><li>Check disk space used by the app</li></ul> | We have 4 DBs: state, event, media, crypto | |
35 | | -| Account size | <ul><li>Check the impact on speed performance</li><li>Check the impact on \`/sync\` response size</li></ul> | It should be part of a new stats FFI API | |
36 | | -| First /sync response size | <ul><li>Check it stays as small as possible</li></ul> | It should be part of a new stats FFI API | |
| 30 | +| Device information | <ul><li>To detect specific device problem, especially on the fragmented Android ecosystem</li></ul> | It is provided by Sentry | |
| 31 | +| Homeserver URL hash | <ul><li>To compare matrix.org and element.io homeservers speed</li><li>To measure the impact of a slow homeserver</li></ul> | We use SHA-512 to compute the hash. </br>**TBD:** Define the input string (`https://matrix.org`, `matrix.org`() or `https://matrix-client.matrix.org` (`base_url`) or something else) | |
| 32 | +| DBs file size | <ul><li>To check the impact of growing DB on speed performance</li><li>To check disk space used by the app</li></ul> | We have 4 DBs: state, event, media, crypto.</br>**TODO**: Add here names from the new Rust SDK API | |
| 33 | +| Account size | <ul><li>To check the impact on speed performance</li><li>To check the impact on \`/sync\` response size</li></ul> | It is the number of rooms joined by the user.</br>**TODO**: Add the Rust SDK API for reference | |
| 34 | +| First /sync response size | <ul><li>To check network usage stays as low as possible</li></ul> | **TODO**: Add the Rust SDK API for reference | |
0 commit comments