"Try Connecting via TCP" Message

[Rodney-hubs]

Currently running Hubs Cloud on Digital Ocean using a Droplet with 1vCPU, 2GB memory, 50GB SSD. This instance had been running for a few days without any issues but, after restarting the Droplet, I'm now getting the "Try Connecting via TCP" message. After browsing the Discord, I have tried several of the proposed fixes:

• Allowing inbound ICMP through firewall
  -- Also allowed outbound
• Using ?force_tcp=true, ?force_turn=true, as well as ?force_turn=true&force_tcp=true
• Completely disabled firewall by running sudo ufw disable
• Tried Chrome/Firefox/Opera browsers

Here is the output from bio sup status

mozillareality/imgproxy/v2.7.0/20191206065624            standalone  up       up     237          2039  imgproxy.default@polycosm
mozillareality/janus-gateway/2.0.0/20200630063359        standalone  up       up     293          1091  janus-gateway.default@polycosm
mozillareality/hubs-discord-bot/0.0.1/20200215023448     standalone  up       up     2            3827  hubs-discord-bot.default@polycosm
mozillareality/reticulum/1.0.1/20200911183406            standalone  up       up     225          2272  reticulum.default@polycosm
mozillareality/hubs/1.0.0/20200908210500                 standalone  up       up     236          2045  hubs.default@polycosm
mozillareality/spoke/1.0.0/20200803233608                standalone  up       up     236          2051  spoke.default@polycosm
mozillareality/postgrest/5.2.0/20190130035107            standalone  up       up     236          2059  postgrest.default@polycosm
mozillareality/youtube-dl-api-server/0.4/20200916110006  standalone  up       up     238          1974  youtube-dl-api-server.default@polycosm
mozillareality/coturn/4.5.3.0/20200422002445             standalone  up       up     236          2066  coturn.default@polycosm
mozillareality/ita/0.0.1/20200526203229                  standalone  up       up     260          1703  ita.default@polycosm
mozillareality/certbot/1.0.0/20191224043510              standalone  up       up     236          2071  certbot.default@polycosm

Through the browser console, this is the error I'm getting:
index.js:181 WebSocket connection to 'wss://<domain>:8443/' failed: Error in connection establishment: net::ERR_CONNECTION_REFUSED

Running journalctl -f shows a bunch of GET requests that have 200/304 responses as well as this:

15:28:48.303 [info] CONNECTED TO RetWeb.SessionSocket in 367µs
  Transport: :websocket
  Serializer: Phoenix.Socket.V2.JSONSerializer
  Parameters: %{"vsn" => "2.0.0"}
15:28:48.444 [info] JOINED ret in 254µs
  Parameters: %{"hub_id" => "5SdAWYL"}
15:28:48.596 [info] JOINED hub:5SdAWYL in 58ms
  Parameters: %{"auth_token" => nil, "context" => %{"embed" => false, "mobile" => false}, "hub_invite_id" => nil, "perms_token" => nil, "profile" => %{"avatarId" => "8LdYson", "displayName" => "Mallard-19966"}, "push_subscription_endpoint" => nil}

I'm fairly certain that destroying the Droplet and recreating the instance would get me operational but it seems like if the Droplet restarts, I won't be able to connect again.

Thanks in advance for looking at this & I'm happy to provide any additional information

[robinkwilson]

Thanks for getting the journalctl -f logs!

Quick questions (disclaimer: I'm going to ask you for repro steps and open an issue in hubs-cloud repo): https://github.com/mozilla/hubs-cloud/issues

• Anything customized on your instance? Like a custom client?
• In the Dev Console in the browser, my droplet instance is on version: App version: 1.0.0.20200908210500 same as the current version on HC Changelog
• Though, I don't believe this is the issue, the request is connecting to reticulum's websockets. You could restart reticulum via:
  1. sudo hab svc status - list services running on the box to check you're in the correct one
  2. sudo hab svc stop mozillareality/reticulum
  3. sudo hab svc start mozillareality/reticulum
• Likewise you could try restarting mozillareality/hubs

After those, can you try restarting the droplet?
If that doesn't solve the issue, mind copying and pasting this in a github issue in the hubs-cloud repo? : https://github.com/mozilla/hubs-cloud/issues

[Rodney-hubs]

Thanks for your reply, Robin!

• No custom client, only thing customized would be visual attributes like logos and button colors
• App Version: Reticulum @ <domain>: v1.0.20200911183406 on default which seems to match the latest release
• The commands you posted come back with hab: command not found but I can run sudo bio svc status

1. Status command states everything is up (reticulum, hubs, janus-gateway, among others)
2. Restarting reticulum succeeded but didn't fix the issue
3. Restarting hubs succeeded but didn't fix the issue

Restarting the Droplet seemed to be the cause of the issue but I tried restarting it again but the issue was still present

Reproduction steps seem as simple as creating a Droplet with Hubs Cloud, configuring it, and then restarting the Droplet. If I have to destroy this instance, I will provide more accurate reproduction steps.

[netpro2k]

Hmm, this is pretty surprising. The journalctl log you have seems to imply the client is connecting to reticulum (and even sending some data), but the browser side log seems to indicate its failing at the connection level... Does the name in the log ("Mallard-19966"), line up with the client you are failing to connect with? Also if you look at the "network" tab in the browser console, do you see the websocket connection get established?

Also it sounds like you already checked this, but here are the firewall settings for my working instance:

[Rodney-hubs]

Thanks for your reply!

The "Mallard-19966" name is the randomly assigned name when I was trying to connect in an private/incognito session. If I were to attempt to connect using my logged in session, I get the same results but DisplayName is tied to my Identity set up in the Admin panel.

Looking at the network tab in Dev Tools, there's several fetch requests for css, js, images, glbs, etc that finish with a 200 status and then that's followed by multiple calls to that shows a 200 status but if I filter by WebSocket, those requests have a "Finished" status with size 0 B and time "Pending". I exported a HAR file as well as an export of the console log if you want me to send those to you privately.

Side note, under the Admin panel -> Content CDN, verifying the workers gives me a "Safety warning" through Chrome. I can choose "Ignore" and that takes me to the Mozilla homepage/What's New page:

(I believe this is fine, I just don't remember seeing it before)

My firewall rules look identical to yours except that I also allowed ICMP inbound and outbound. Proof!:

[Rodney-hubs]

Another note, I'm not able to create and enter a new room even with the "Wide Open Space" set as the default scene.

[Rodney-hubs]

I created Issue-115 with steps to reproduce.

[robinkwilson]

Thank you for the issue and steps to repro!

[calvincs]

This is happening again.
After reboot, my custom client code is gone and I can no longer access rooms.

In the console, I get a wss://example.com:8443 error.

[calvincs]

[elmau]

I'm having the same issue with a Hubs-Cloud Enterprise. No modifications, just internal settings like names and theme colors. I restarted the stack, but no luck at all. I've tried connecting from different devices, but I can't access or create rooms.

[robinkwilson]

@elmau Can you paste console logs + network errors from the Developer console?

[elmau]

I don't have them since I started the stack from a cellphone and then got the error. However I think it could be a time issue? After 30 minutes I could enter a room. Maybe it takes a longer time to synchronize all the internal services?