Serialize destruction of the delegate (#6111)

Serialize destruction of the delegate (#5996)

Summary:
Pull Request resolved: #5996

When workspace sharing is enabled, if we call `destroy()` from multiple threads it is possible that we enter in `xnn_delete_runtime` from multiple threads. It is not safe to do so, and can cause double free. This serializes the destroy calls for the singleton backend object if/when called multiple times.

Not locking in the failure case because we may not have a real memory to free i.e. empty workspace?

Reviewed By: digantdesai

Differential Revision: D63917052

fbshipit-source-id: 61e6b1c54f887a54501a4fd3433f8470e74dbca3
(cherry picked from commit 566902b)

Co-authored-by: Gregory Comer <[email protected]>

Author: pytorchbot

backends/xnnpack/runtime/XNNPACKBackend.cpp

@@ -132,6 +132,12 @@ class XnnpackBackend final : public ::executorch::runtime::BackendInterface {
 
   void destroy(DelegateHandle* handle) const override {
     if (handle != nullptr) {
+#ifdef ENABLE_XNNPACK_SHARED_WORKSPACE
+      // This is needed to serialize access to xnn_delete_runtime which is not
+      // thread safe. This can heppen when multiple threads call destroy() on
+      // the same backend instance.
+      const std::lock_guard<std::mutex> lock(workspace_mutex_);
+#endif
       auto executor = static_cast<xnnpack::delegate::XNNExecutor*>(handle);
 #ifdef ENABLE_XNNPACK_PROFILING
       executor->print_avg_op_timings();