fix: Adjust timing on WorkerLocks to allow faster resolution of the lock

[jason-famedly]

Basing the time interval around a 5 seconds leaves a big window of waiting especially as this window is doubled each retry, when another worker could be making progress but can not.

Right now, the retry interval in seconds looks like [0.1, 5, 10, 20, 40, 80, 160, 320, (continues to double)] after which logging should start about excessive times

With this change, retry intervals in seconds should look more like [0.1, 0.2, 0.4, 0.8, 1.6, 3.2, 5.0, 5.0, 5.0, (stays at 5.0)]

I do not fully understand how much of the delicate balance this will change, but tests pass fine. I do believe that increasing the cap to 10 or even 20 seconds would be ok, as some places that take locks can have quite some delay. Increasing the jitter range may also be a good idea

[codecov]

Codecov Report

❌ Patch coverage is 50.00000% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 79.03%. Comparing base (1994509) to head (05e50a6).

Files with missing lines	Patch %	Lines
synapse/handlers/worker_lock.py	50.00%	1 Missing ⚠️

Additional details and impacted files

@@                    Coverage Diff                     @@
##           famedly-release/v1.143     #221      +/-   ##
==========================================================
+ Coverage                   79.02%   79.03%   +0.01%     
==========================================================
  Files                         498      498              
  Lines                       71666    71666              
  Branches                    11087    11087              
==========================================================
+ Hits                        56632    56640       +8     
+ Misses                      11326    11322       -4     
+ Partials                     3708     3704       -4     

Files with missing lines	Coverage Δ
synapse/handlers/worker_lock.py	86.00% <50.00%> (ø)

... and 7 files with indirect coverage changes

---

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1994509...05e50a6. Read the comment docs.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.