rgw: RGWSI_Notify drains the finisher before deleting RGWWatchers

the `C_ReinitWatch` context stores a pointer to `RGWWatcher *watcher`

`RGWSI_Notify` schedules `C_ReinitWatch` callbacks to run on this
finisher thread, which can race with the destruction of those
`RGWWatchers` in `RGWSI_Notify::finalize_watch()`

to avoid this race, delay the `RGWWatcher` deletions until after we've
drained/stopped the Finisher thread in `RGWSI_Notify::shutdown()`

Fixes: https://tracker.ceph.com/issues/73361

Signed-off-by: Casey Bodley <[email protected]>

Author: cbodley

src/rgw/services/svc_notify.cc

@@ -237,8 +237,6 @@ void RGWSI_Notify::finalize_watch(boost::asio::yield_context yield)
         });
   }
   throttle.wait();
-
-  watchers.clear();
 }
 
 int RGWSI_Notify::do_start(optional_yield y, const DoutPrefixProvider *dpp)
@@ -303,9 +301,13 @@ void RGWSI_Notify::shutdown()
       });
   context.run();
 
+  // wait for any racing C_ReinitWatch calls on the finisher thread
+  // before destroying the RGWWatchers
   finisher.wait_for_empty();
   finisher.stop();
 
+  watchers.clear();
+
   finalized = true;
 }