Fix volmeter teardown deadlock between audio thread and IPC (#1718)

OBSCallback runs while libobs holds the volmeter callback_mutex, then takes
the osn global mtx (and the manager's internal_mutex via find). The teardown
paths acquire those osn locks first and then take callback_mutex:
  - Destroy holds mtx, then obs_volmeter_remove_callback -> callback_mutex
  - ClearVolmeters' for_each holds internal_mutex, then
    obs_volmeter_remove_callback -> callback_mutex
This is two AB-BA cycles (callback_mutex<->mtx, callback_mutex<->internal_mutex)
that can hang on shutdown while audio is still flowing.

- OBSCallback now try_locks mtx and drops the update if busy, so the audio
  thread never blocks on mtx while holding callback_mutex.
- ClearVolmeters collects the volmeters under the manager lock, then removes
  callbacks and clears outside for_each, so callback_mutex is never taken while
  the manager lock is held.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Author: summeroff

obs-studio-server/source/osn-volmeter.cpp

@@ -56,8 +56,12 @@ void osn::Volmeter::Register(ipc::server &srv)
 
 void osn::Volmeter::ClearVolmeters()
 {
-	Manager::GetInstance().for_each(
-		[](const std::shared_ptr<osn::Volmeter> &volmeter) { obs_volmeter_remove_callback(volmeter->self, OBSCallback, &volmeter->id); });
+	// Remove callbacks outside for_each: holding the manager lock across callback_mutex deadlocks OBSCallback.
+	std::vector<std::shared_ptr<osn::Volmeter>> meters;
+	Manager::GetInstance().for_each([&](const std::shared_ptr<osn::Volmeter> &volmeter) { meters.push_back(volmeter); });
+
+	for (const auto &meter : meters)
+		obs_volmeter_remove_callback(meter->self, OBSCallback, &meter->id);
 
 	Manager::GetInstance().clear();
 }
@@ -151,7 +155,12 @@ void osn::Volmeter::Detach(void *data, const int64_t id, const std::vector<ipc::
 void osn::Volmeter::OBSCallback(void *param, const float magnitude[MAX_AUDIO_CHANNELS], const float peak[MAX_AUDIO_CHANNELS],
 				const float input_peak[MAX_AUDIO_CHANNELS])
 {
-	std::unique_lock<std::mutex> ulockMutex(mtx);
+	// Runs under callback_mutex; block-waiting on mtx would deadlock teardown, so drop the frame if mtx is busy.
+	std::unique_lock<std::mutex> ulockMutex(mtx, std::try_to_lock);
+	if (!ulockMutex.owns_lock()) {
+		return;
+	}
+
 	auto meter = Manager::GetInstance().find(*reinterpret_cast<uint64_t *>(param));
 	if (!meter) {
 		return;