Add LFBP Instrumentation, Server Shutdown Ordering, and Test TearDown Hardening

[vazois]

Summary

Adds instrumentation for tracking owners and type of buffers acquired from LimitedFixedBufferPool (LFBP) to diagnose dispose issues and buffer leaks. Improves server shutdown ordering so listening sockets are closed immediately (freeing ports), and hardens cluster test teardown so cleanup always runs even when tests fail.

Changes

Buffer Pool Tracking Instrumentation

• New enums (PoolEntryTypes.cs): PoolEntryBufferType (identifies buffer role, e.g. NetworkReceiveBuffer, SaeaSendBuffer) and PoolOwnerType (identifies pool creator, e.g. ServerNetwork, Replication, ClientSession).
• PoolEntry.source field: packed int where the low byte stores PoolEntryBufferType and byte 1 stores PoolOwnerType, set when the entry is acquired via LimitedFixedBufferPool.Get().
• LimitedFixedBufferPool now accepts PoolOwnerType at construction and PoolEntryBufferType in Get(). Under #if DEBUG, a ConcurrentDictionary tracks all outstanding (checked-out) entries.
• Dispose diagnostics (DEBUG only): after a 5-second timeout, LimitedFixedBufferPool.Dispose() logs all unreturned buffer details (owner type, buffer type, size) to aid leak diagnosis.
• All call sites updated to pass appropriate PoolOwnerType and PoolEntryBufferType values (GarnetClientSession, ReplicationManager, GarnetServerTcp, NetworkHandler, TcpNetworkHandlerBase, GarnetSaeaBuffer, LightClient, MigrationManager, GarnetClient).

Server Shutdown Ordering & Port Reuse

• IGarnetServer.Close(): new interface method that stops listening without waiting for active connections to drain.
• GarnetServerTcp.Close(): closes the listening socket to free the port immediately.
• GarnetServer.InternalDispose: three-phase shutdown — (1) Close() all servers to free ports, (2) dispose the provider (storage engine), (3) Dispose() servers to drain active handlers.
• GarnetServerTcp constructor: sets SO_REUSEADDR on TCP sockets before Bind to handle TIME_WAIT states. UDS initialization refactored for clarity.
• GarnetServerTcp.Dispose: reordered to close the listening socket before calling base.Dispose() (which drains active handlers).

ExceptionInjectionHelper Improvements

• Replaced busy Task.Yield() loops with a TaskCompletionSource<bool> notification pattern, eliminating spin-waiting when tests wait for exception injection points.
• Renamed WaitOnSet → ResetAndWaitAsync for clarity.
• Updated WaitOnClearAsync to use the same TCS-based notification.
• All callers updated: MigrateSessionSlots, ReplicaSyncSession, ReplicaDisklessSync, ReplicaReceiveCheckpoint.

LocalStorageDevice Logging

• Added ILogger parameter to LocalStorageDevice constructors.
• ReadAsync / WriteAsync now log caught exceptions at Critical level instead of silently swallowing them.
• Devices.cs passes the logger through on Windows.

Cluster Test TearDown Hardening

• ClusterTestContext.TearDown(): captures TestContext.CurrentContext.Result.Outcome before cleanup begins. All cleanup phases (DisposeCluster, logger factory, delete directory, OnTearDown) now run unconditionally; Assert.Fail is deferred to the end so resources are always released.
• Increased teardown timeout from 5 → 30 seconds.
• SimplePrimaryReplicaSetup(): new helper that configures a one-primary-one-replica cluster with slot assignment and MEET.
• AttachAndWaitForSync: removed redundant Meet and BumpEpoch calls (now handled by SimplePrimaryReplicaSetup).
• ReplicaSyncSession: added cts.Token.ThrowIfCancellationRequested() at the top of the AcquireCheckpointEntry loop.

Misc

• Upgraded DisposeActiveHandlers() diagnostics from HANGDETECT to DEBUG with Stopwatch-based timeout logging of stuck handlers.
• Fixed typo: "Dipose" → "Dispose" in LimitedFixedBufferPool doc comment.

[Copilot]

Pull request overview

Fixes a teardown hang caused by incomplete network-handler disposal and leaked pooled buffers, and adds DEBUG-only diagnostics to identify unreturned pool entries and stuck handlers.

Changes:

• Ensure TcpNetworkHandlerBase.Dispose() performs full cleanup by invoking DisposeImpl().
• Fix pooled buffer leak by disposing GarnetTcpNetworkSender’s held responseObject during sender disposal.
• Add pool ownership/buffer-role tagging and DEBUG diagnostics for pool/handler disposal hangs.

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
libs/server/Servers/GarnetServerTcp.cs	Tags the server network pool with an owner type for diagnostics.
libs/server/Servers/GarnetServerBase.cs	Adds DEBUG hang diagnostics for stuck active handler disposal (but introduces a Release build issue via unused using).
libs/common/Networking/TcpNetworkHandlerBase.cs	Calls DisposeImpl() from public Dispose() and tags receive-buffer allocations.
libs/common/Networking/NetworkHandler.cs	Tags transport/network buffer allocations to aid leak diagnostics.
libs/common/Networking/GarnetTcpNetworkSender.cs	Disposes held responseObject during sender disposal to avoid pool reference leaks.
libs/common/Networking/GarnetSaeaBuffer.cs	Tags SAEA send-buffer allocations.
libs/common/NetworkBufferSettings.cs	Extends buffer-pool creation to accept an owner type.
libs/common/Memory/PoolEntryTypes.cs	Introduces enums for pool owner and buffer role tagging.
libs/common/Memory/PoolEntry.cs	Adds packed source field used for diagnostics.
libs/common/Memory/LimitedFixedBufferPool.cs	Adds owner tagging, DEBUG outstanding-entry tracking, and disposal-time diagnostics (but introduces a Release build issue via unused using).
libs/cluster/Server/Replication/ReplicationManager.cs	Tags replication-created pools with owner type.
libs/client/ClientSession/GarnetClientSession.cs	Tags client-session-created pools with owner type.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.