Debugging user problems is hard

[rawnsley]

Statement of Problem

Debugging user problems on Hubs is difficult for a number of reasons:

1. Hubs runs on a wide variety of platforms and network conditions
2. Physical access to end user equipment is uncommon
3. Obtaining quality diagnostic data from users is difficult
4. Obtaining useful diagnostic data from the server is difficult
5. Complex faults may involve one or more users and the server

Here are some example problems that we struggle to diagnose and fix:

• "I can't connect to this room"
• "I can't hear anybody"
• "Jim could hear me and Mike, but I couldn't hear them"
• "My webcam isn't showing up on my avatar"
• "Hubs is very slow when Peter joins"
• "Large video files are not working in my room"

Ideal Solution

The ideal solution to this problem would look something like this:

1. Record browser console logs for all active users on the server
2. Record server logs for all the various services (reticulum, proxy, webrtc, youtubedl, etc...)
3. Bundle the logs together by room

These bundles would contain a wealth of information about the state of each room. They could be analyzed in a number of ways, including interleaving the log messages by time to make it easy to follow sequences of operations.

Practical First Steps

Step 1: Browser Logs

One possible first step is to make it easier for users to share their console logs when reporting problems. This can be achieved by recording the console log in memory and dumping it to a file at the request of the user. Initially this can be hidden behind a URL parameter along with any additional UI, which would probably be some sort of "Save Logs" button.

Intercepting the console log is unfortunately limited in that it doesn't include events such as CORS errors, which are a common source of failure in Hubs. These can only be seen in the console log window itself.

A pull request is that illustrates how the console log might be recorded and analyzed is here #4535

Step 2: Server Logs

It would be very useful for Hubs Cloud administrators to have a more comprehensive guide to gathering server logs, preferably a single script to execute that can pull everything together that might be relevant.

Considered Alternatives

• "Ask users to send the console logs" This is actually quite complicated on desktop devices, as these typical instructions show and the option isn't available on mobile or in VR. Also the format and quality of these logs varies by browser, for example Firefox doesn't include timestamps.
• Dedicated diagnostic pages such as the one in this proposal are very useful for specific problems, but a page is required for each class of problems and the user must be directed to the correct one. They are also limited to known failure modes such as port blocking, but they do offer the chance for self-service user diagnostics, which a console log based approach does not.

Privacy & Security

The privacy implications of users sharing their console logs needs to be considered. It is also possible that sensitive information, such as session tokens, might be written to the logs now or in the future.

If the logs are gathered and analyzed "server side" then there are no obvious additional risks. In the case where users can download and share their console logs the risk is significant, but they are already able to do this now using the "Save As" function in the developer console, so it does not represent a new class of vulnerability.

[pattersonbl2]

I have opened a new issues for this issues here (#4563)

I will mark this as answered.

[keianhzo]

Related: #4133