fix(queue): replace semaphore with sync.Cond to eliminate worker serialization

[jwnx]

The current implementation uses a size-1 semaphore to ensure a single
worker reads from q.items.Front() at a given time. The problem is that
after the worker realizes it has to wait until plannedToStartWorkAt,
it continues to hold the semaphore while sleeping. Other workers block
until the sleeping worker wakes up, processes its task, and releases.

The q.items list acts as a FIFO queue and the keeps the invariant that
elements are ordered by how soon they are to be processed, given the
plannedToStartWorkAt value. While a worker sleeps holding the semaphore,
elements that should be scheduled immediately cannot be processed even
though they might have nothing else to do. When we consider that the
default max backoff is 1000s, we can have a worker holding the semaphore
for as much as 16 minutes.

This patch removes the semaphore and replaces it with a sync.Cond. We'll
c.Wait in two situations: there's nothing to do (no elements to process),
so we wait until we get a Signal from insert, or we got a element but
have to wait for the plannedToStartWorkAt timer. To avoid a timer mutex
race, we prefer Signals when we know an item is ready (insert) or the
timer fired. Broadcast is used for cancellations and shutdowns.

Signed-off-by: Juliana Oliveira juliana@fly.io

[soulware]

Nice find!