[vm] Early-out in NativeCallable trampoline if isolate creation disabled

I haven't been able to get a solid repro the tsan flake locally, and I'm not sure how to write a test for it, but this change is pretty uncontroversial anyway, so I'm inclined to just land it and see what happens.

I have some weak evidence that this fixes the issue though. Before this change I ran the flaky command about 500 times before I saw a failure (though I didn't get any useful data from the failure, and it didn't look quite the same as the bug). With the change, the command ran 3500 times before flaking. I'm not sure if it's the same flake though (rr isn't working for me atm).

Bug: #56900
Fixes: #56900
Change-Id: I939e8ceca4051ec91e942ca38a2b01477b67ddd8
TEST=CI
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/392162
Reviewed-by: Ryan Macnak <[email protected]>
Commit-Queue: Liam Appelbe <[email protected]>

Author: liamappelbe

runtime/vm/isolate.cc

@@ -863,7 +863,7 @@ Isolate* IsolateGroup::EnterTemporaryIsolate() {
   Dart_IsolateFlags flags;
   Isolate::FlagsInitialize(&flags);
   Isolate* const isolate = Isolate::InitIsolate("temp", this, flags);
-  ASSERT(isolate != nullptr);
+  RELEASE_ASSERT(isolate != nullptr);
   ASSERT(Isolate::Current() == isolate);
   return isolate;
 }

runtime/vm/runtime_entry.cc

@@ -4378,6 +4378,12 @@ extern "C" Thread* DLRT_GetFfiCallbackMetadata(
   ASSERT(out_entry_point != nullptr);
   ASSERT(out_trampoline_type != nullptr);
 
+  if (!Isolate::IsolateCreationEnabled()) {
+    TRACE_RUNTIME_CALL("GetFfiCallbackMetadata called after shutdown %p",
+                       reinterpret_cast<void*>(trampoline));
+    return nullptr;
+  }
+
   Thread* const current_thread = Thread::Current();
   auto* fcm = FfiCallbackMetadata::Instance();
   auto metadata = fcm->LookupMetadataForTrampoline(trampoline);