common: drop stack singleton object of temp messenger for foreground ceph daemons

During the initialization, OSD needs to create a temporary messenger to read config db from the ceph-mon.
This temporary messenger will need to create async messenger threads according to the local/default value of "ms_async_op_threads" .
If this option is not specified in ceph.conf, it will by default create 3 threads, then use these threads to read config db from ceph-mon.
Those threads are associated to a stack singletion object.
Now here is the difference between OSD running in foreground and background:
a. In background mode, this singleton object will be dropped before forking the child process in function notify_pre_fork,
then the new ms_async_op_threads fetched from mon config db will be used to create later messenger threads, this is what we want.
b. In foreground mode, this singleton object will not be dropped and will be reused by all later messengers,
thus the number of threads doesn't change.

Fixes: https://tracker.ceph.com/issues/71401
Signed-off-by: dongdong tao <[email protected]>

Author: taodd

src/common/ceph_context.cc

@@ -1026,6 +1026,19 @@ CryptoHandler *CephContext::get_crypto_handler(int type)
   }
 }
 
+void CephContext::drop_temp_messenger_obj()
+{
+  auto i = associated_objs.begin();
+  while (i != associated_objs.end()) {
+    if (i->first.first.find("AsyncMessenger::NetworkStack") != std::string::npos) {
+      i = associated_objs.erase(i);
+      break;
+    } else {
+      ++i;
+    }
+  }
+}
+
 void CephContext::notify_pre_fork()
 {
   {

src/common/ceph_context.h

@@ -272,6 +272,7 @@ class CephContext {
     _fork_watchers.push_back(w);
   }
 
+  void drop_temp_messenger_obj();
   void notify_pre_fork();
   void notify_post_fork();
 

src/global/global_init.cc

@@ -469,7 +469,7 @@ int global_init_prefork(CephContext *cct)
       chown_path(conf->pid_file, cct->get_set_uid(), cct->get_set_gid(),
 		 cct->get_set_uid_string(), cct->get_set_gid_string());
     }
-
+    cct->drop_temp_messenger_obj();
     return -1;
   }