Why this should be merged

Quoting #1694:

The current solution uses a combination of a modified semaphore and a channel to apply back pressure, with the goal of bounding the staleness of the most recent persisted revision. In general, mixing semaphores and channels (or any other synchronization primitive) can result in tricky and difficult to reason about solutions. In this case, there is an implicit dependence between the values of the counter inside the semaphore, and a separate counter managed by the event loop to determine when to "release permits" in the modified semaphore. This can lead to a deadlock if the semaphore counter reaches zero before the event loop counter falls below the threshold to reset the semaphore. The current solution may work, but it is fragile since a change to just one of the counter thresholds (max permits or count value before reset) could lead to a deadlock that might be difficult to find during standard tests.

How this works

Replaces the PersistSemaphore + crossbeam::channel with PersistChannel, a channel-like abstraction around the locks/condvars mentioned in #1694. By using PersistChannel, we no longer have the issue of the PersistWorker and the PersistLoop having their own notions of progress as all progress is managed via PersistChannelState.

This PR also adds a drop guard to the PersistLoop, such that if the background thread exits for whatever reason, the system is marked as shutdown which prevents the PersistWorker from hanging.

How this was tested

CI + existing deferred persistence UTs