refactor: replace PersistSemaphore with PersistChannel

[RodrigoVillar]

Why this should be merged

Quoting #1694:

The current solution uses a combination of a modified semaphore and a channel to apply back pressure, with the goal of bounding the staleness of the most recent persisted revision. In general, mixing semaphores and channels (or any other synchronization primitive) can result in tricky and difficult to reason about solutions. In this case, there is an implicit dependence between the values of the counter inside the semaphore, and a separate counter managed by the event loop to determine when to "release permits" in the modified semaphore. This can lead to a deadlock if the semaphore counter reaches zero before the event loop counter falls below the threshold to reset the semaphore. The current solution may work, but it is fragile since a change to just one of the counter thresholds (max permits or count value before reset) could lead to a deadlock that might be difficult to find during standard tests.

How this works

Replaces the PersistSemaphore + crossbeam::channel with PersistChannel, a channel-like abstraction around the locks/condvars mentioned in #1694. By using PersistChannel, we no longer have the issue of the PersistWorker and the PersistLoop having their own notions of progress as all progress is managed via PersistChannelState.

This PR also adds a drop guard to the PersistLoop, such that if the background thread exits for whatever reason, the system is marked as shutdown which prevents the PersistWorker from hanging.

How this was tested

CI + existing deferred persistence UTs

[bernard-avalabs]

A few minor suggestions.

[rkuris]

Mostly nits and some questions. Please re-request review if you make substantial changes.

[demosdemon]

just nits on overflowing arithmetic

[demosdemon]

nits:

• the uwnrap should not lint anymore inside the const { ... } block and div_ceil is defined for NonZero<_> that returns a NonZero<_>.
• wrapping_sub is fine here because we logically know that persist_interval is strictly less than or equal to commit_count so the subtraction will never wrap.

Suggested change

	let persist_interval = NonZeroU64::new(commit_count.get().div_ceil(2))
	.expect("a nonzero div_ceil(2) is always positive");
	let persist_threshold = commit_count.get().saturating_sub(persist_interval.get());
	let persist_interval = commit_count.div_ceil(const { NonZeroU64::new(2).unwrap() });
	let persist_threshold = commit_count.get().wrapping_sub(persist_interval.get());

It's important to be aware of the mathematical operations you're doing and not blanket chose saturating_, wrapping_, or checked_. saturating and checked operations have overhead that we can and should skip if we know our logical operations won't trigger overflow.

[demosdemon]

another example here. The compiler might actually skip the extra work from saturation because we have an earlier check that permits_available is not zero. However, that condition is not exclusive to letting control flow to here, so the compiler might conservatively check again. However, because we know that because state.shutdown is false at this point, we also know that state.permits_available is non-zero and saturation is extra work.