Notification system for clogged server WU uploads

[Hou5e]

When this state happens (all uploads and no folding) it would be really nice to have a notification system for clogged server WU uploads to help notify the people who could decide to restart the software or servers causing the problems. This can happen for a user's local network issue, but if many PCs are reporting this issue, then it's probably a problem with the FAH servers.

NOTE: This happens a lot more often when the WUs are too small and complete in less than 20 minutes.

Most of these uploads stall out and keep retrying over and over, and don't complete the upload. The only solution currently is to turn off FAH on most or all of these PCs until the FAH servers are fixed or are rebooted. Or enable FAH on only one machine at a time until the uploads complete.

Issue is related to #98 for better upload management (and everyone not having 10+ fiber optic connections).

[arisu3]

A better solution is to limit the number of concurrent uploads, with #381 or something similar.

The fundamental issue is not that FAH servers are overloaded (though they can be slow), it is that uploads are buffered, but the client will kill the connection if it hasn't seen any transfers in 60 seconds. When you have a lot of uploads or the connection is slow, you can end up wasting gigabytes of data and never successfully completing an upload.

[Hou5e]

This request is mostly to have automated messaging for collection server admins (for many PCs in this state) versus indicating that the user's system is pretty much the only PC in this state (user's side or their ISP).

My update: Downloads were working fine, but uploads were about 1/100 of the download speed (instead of 1/10th) and had frequent errors. After having all my folding PCs and solar panels turned off for the past 2 weeks, I replaced my network switch, router, cables, and cable modem (new one has much better error correcting, which helped but didn't fix the problem). As far as I can tell this was the cable Internet ISP issue, and they most likely rebooted something on their end and my uploads are working again today.

Also note: When my ISP was bad, the completed 330MB WU uploads for Project 12129 would take 2 hours to compute the WU (on a 5090) and most likely 2-4 hours to upload (slower than computing it, so not going to work, where normally it takes 6 minutes to upload), where the upload (not done in file chunks or with resuming) seems to get cut off at 1 hour. Then, the infinite upload retries would cause a mess like the images above. Trying to dump the completed WU would not do any better, since FAH tries to upload the completed results again instead of deleting them. Pausing uploads in that odd state doesn't pause the upload, like the GUI would lead you to believe, either.