Memory leak on ConnectionOpen then ConnectionStart on v2.5.1

[StoicSirius]

Describe the bug

Mentioned in #4362.

Continuous calls to ConnectionOpen, followed by ConnectionStart, and then ConnectionClose in the connection callback causes continuous memory growth. It's not necessary for a server to be listening on the other end. A previous test on 2.4.8 installed via vcpkg caused a similar leak, but with ConnectionStart omitted. The 2.5.1 version compiled with QuicTLS, however, does not seem to have this problem.

A trace run on the following loop on Windows 10 build 19041.450:
trace.zip

for (int i = 0; i < 200; ++i) {
  HQUIC * Connection = new HQUIC(nullptr);

  QUIC_STATUS status = api_table->ConnectionOpen(Registration, ConnectionCallback, Connection, Connection);
  api_table->ConnectionStart(*Connection, Configuration, QUIC_ADDRESS_FAMILY_INET, "127.0.0.1", 8000);
  std::this_thread::sleep_for(std::chrono::milliseconds(100));
}

With the callback:

QUIC_STATUS ConnectionCallback(HQUIC Connection, void * Context, QUIC_CONNECTION_EVENT * Event) {
  switch (Event->Type) {
    case QUIC_CONNECTION_EVENT_SHUTDOWN_COMPLETE:
      api_table->ConnectionClose(Connection);
      delete (HQUIC *)Context;
      break;

    default:
      break;
  }

  return QUIC_STATUS_SUCCESS;
}

[guhetier]

Thanks for opening this.

To clarify:

• v2.4.8 + QuicTLS: "Connection Open -> Connection Close" was causing a leak
• v2.5.1 + QuicTLS: "Connection Open -> Connection Start -> Connection Close" was causing a leak

Is that correct?

[StoicSirius]

Not exactly. If I don't call ConnectionStart, then the connection never shuts down by itself, and manually calling ConnectionClose makes this assertion on line 145 of api.c fail: QUIC_CONN_VERIFY(Connection, !Connection->State.HandleClosed);.
I instead call ConnectionShutdown manually in the for loop, and once SHUTDOWN_COMPLETE is hit, ConnectionClose is called as usual. Looking at above code, I simply replace ConnectionStart with ConnectionShutdown.

Aside from that, if the 2.4.8 version installed via vcpkg with the 0-rtt feature uses the QuicTLS fork (vcpkg isn't particularly clear to me on that matter), then yes, you are correct. I'd like to clarify that "Connection Open -> Connection Start -> Connection Close" was also causing a leak on v2.4.8.

[nibanks]

I haven't looked deeply at this, but I suspect it's a known issue with OpenSSL that holds on to some global memory forever. Can you try testing for leaks after a second allocation/free? In otherwords:

1. Alloc conn1
2. Free conn1
3. Alloc conn2
4. Free conn2

Are there any leaks between steps 3 and 4? Also, keep in mind that MsQuic uses pool allocators where we purposefully don't free things back to the heap immediately.

[StoicSirius]

I did a few tests. Took a snapshot after calling ConnectionOpen and just before calling ConnectionClose, on a second iteration. We can see OpenSSL functions down in InitializeTls:

I was unaware that MsQuic uses pools, so I did another test where I added the api, reg and config allocs/frees into the loop and took snapshots at the beginning of each iteration. I can confirm that memory did get freed every iteration, but I still noticed memory growth:

Looking at each, QuicLibraryGetBinding is present in both tests.