Fixes #39083 - Use Action Cable for UI notifications

[jeremylenz]

This change (mostly) eliminates web UI polling to GET "/notification_recipients" in favor of a WebSocket connection via Rails Action Cable.

To test:

Load the web UI in the browser
Get a new notification (I used host details > Refresh applicability, but you can do anything that causes a notification in the notification bell) OR clear a notification / mark as read

Before: in the browser dev tools Network tab, you will see polling to /notification_recipients every 10 seconds.
After: No polling. Instead, you will see a "Socket" request to GET /cable and notifications should appear immediately

[jeremylenz]

this is from the rails example and needs to be fixed

[jeremylenz]

@ehelms Do you know what the correct value is here, for the production redis server?

[ekohl]

It's configurable and I'd expect that the installer manages the file to configure this. I think this file should likely be one of those things where we ship a .example, similar to settings.yaml.

[jeremylenz]

I started out with async adapter, but turns out it's really unreliable because the message sender (broadcaster) and client have to be on the same process, or else the message is not received. Redis seemed to be available already, and works well in my development environment. I assume it would also work in production, but I'm not quite sure of the correct configuration and will probably need some help there.

[ehelms]

I would be interested in comparing the redis vs. postgresql adapters. We should also investigate if the redis adapter works cleanly with valkey.

[ekohl]

As I said in https://community.theforeman.org/t/revisiting-action-cable-for-notifications-6-years-later/45738/2

First reaction when I saw the title: yes please.

I'd love to see how it scales. What if you connect 100 or 500 clients, what's the impact?

Security wise: is a channel disconnected when a user is deleted?

[jeremylenz]

Security wise: is a channel disconnected when a user is deleted?

I tested this by deleting a non-admin user in Rails console while a REX job was in progress. In the chrome dev tools Network tab (filtered to "Socket" type) the GET /cable request pings were still active. I guess this is because the user authorization is done during def connect only.

However, the user never actually got the notification about the job being complete.

^ I guess this explains that -- the job stalled because the user disappeared.

However, when a user closes or reloads the browser tab, the cable consumer is disconnected, and any subsequent attempts would require authorization again.

I'd love to see how it scales. What if you connect 100 or 500 clients, what's the impact?

Not sure, but since we're eliminating those API requests every 10 seconds I'd guess it's a net positive.

In reality I'd think the Foreman web UI isn't going to have more than a dozen or so simultaneous users.

[ehelms]

If this is the default do we need to define it?

[Thorben-D]

Awesome! Glad to see websockets getting some love.
After this year's configmgmtcamp I actually started writing a websocket interface for Foreman myself.
I have kept it pretty universal, as I see a lot of value in this for plugins as well (notably Plugins using foreman-tasks).
I was going to open an RFC soon^TM, but I'll hurry up a bit if that interests you.

I have been putting it through it's paces lately and can maybe share some insights.
I'll leave the details for the RFC, but generally:

• Speed wise, the ranking is: Redis > PostgreSQL > Solid
  Postgres has an 8Kb message size limit which, to be fair, is probably bigger than anything you'd send over a socket, but still exists. We already have Redis, which doesn't have this limit, so that's what I chose too.
  Adapter choice hardly matters though, because:
• ActionCable is Ruby. With each connection creating a seperate thread, global interpreter lock will tank performance long before either postgres or redis do.
  This happens in the range of hundreds to thousands of connections, greatly depending on Foreman server specs.
  Though, since Foreman isn't a chat app, having something like 10 concurrent connections (i.e. logins) is proabably very rare already. A production spec'd Foreman server can easily handle hundreds of connections before the performance gets bad (relatively!), so this is also not really a concern either.
• While performance, when compared to other websocket servers, might not be top-notch, the absolute speedup is still huge. Whereas an API request is in the order of 100s of milliseconds, socket broadcasts are in the order of microseconds to low milliseconds (depending on payload, concurrent connections, server specs,...), which is pretty crazy.
• Connections are not automatically disconnected when a user is deleted/unauthorized, but this can be implemented using ActionCable::RemoteConnections