Add IMembershipManager abstraction for external membership systems (#9960)

* Add IMembershipManager and IClusterHealthMonitor abstractions for external membership systems

- Add IMembershipManager interface for native membership integration (e.g., RapidCluster)
- Add IClusterHealthMonitor interface to abstract failure detection
- Add NoOpClusterHealthMonitor for systems with external failure detection
- Add NoOpMembershipTable as fallback when IMembershipManager is used
- Update SiloClusteringValidator to accept IMembershipManager as alternative to IMembershipTable
- Update ClusterMembershipService, MembershipAgent, and related services to use IMembershipManager when available
- Update DefaultSiloServices to register fallback NoOpMembershipTable via TryAddSingleton

These changes allow external consensus-based membership systems to integrate natively
with Orleans without requiring a full IMembershipTable implementation.

* Add IProbeHealthMonitor for pluggable probe health checks

- Extract probe health checks from LocalSiloHealthMonitor into IProbeHealthMonitor interface
- Add DefaultProbeHealthMonitor using ProbeRequestMonitor and ClusterHealthMonitor
- Add NoOpProbeHealthMonitor for external failure detection systems
- Register DefaultProbeHealthMonitor in DefaultSiloServices
- Add InternalsVisibleTo for RapidCluster.Orleans assemblies

This allows external membership systems like RapidCluster to disable
Orleans' probe-based health checks that would otherwise report false
degradation when using a different failure detection protocol.

* Refactor membership service to use IMembershipManager interface

Replace direct MembershipTableManager dependencies with the
IMembershipManager abstraction across the runtime:

- ManagementGrain
- MembershipSystemTarget
- LocalSiloHealthMonitor
- SiloHealthMonitor
- SiloMetadataCache
- SiloStatusListenerManager
- SiloRoleBasedPlacementDirector

Also add optional targetVersion parameter to Refresh() for
waiting until membership reaches a specific version.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address PR review comments on IMembershipManager abstraction

- Remove unused 'using System.Threading.Tasks' in IClusterHealthMonitor.cs
- Add CancellationToken to IMembershipManager.Refresh to prevent infinite loops
- Fix nullable annotation: TrySuspectSilo uses SiloAddress? instead of null!
- Remove InternalsVisibleTo for RapidCluster.* from Runtime and Core csproj
- Use monitoredNodeCount parameter in DefaultProbeHealthMonitor
- Rename constructor parameter in SiloStatusListenerManager to match type
- Fix SiloClusteringValidator to detect NoOpMembershipTable
- Change SiloRoleBasedPlacementDirector to inject IMembershipManager

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Remove NoOpMembershipTable; register MembershipTableManager through IMembershipManager

When a custom IMembershipManager is registered, MembershipTableManager is no
longer instantiated. The NoOpMembershipTable fallback is no longer needed.

- Delete NoOpMembershipTable.cs
- Make IMembershipManager extend ILifecycleParticipant and IHealthCheckParticipant
  (same pattern as IClusterHealthMonitor)
- Register MembershipTableManager via TryAddSingleton<IMembershipManager> so
  custom implementations take precedence
- Resolve lifecycle/health participants from IMembershipManager interface
- Update SiloClusteringValidator to detect missing clustering config without
  depending on NoOpMembershipTable

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix nullable annotation in SiloClusteringValidator

Change IMembershipManager to IMembershipManager? to fix CS8600 error
when assigning null to a non-nullable reference type.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Remove all RapidCluster references from doc comments

Replace product-specific references with generic 'external system'
phrasing in IClusterHealthMonitor, IProbeHealthMonitor,
NoOpClusterHealthMonitor, and NoOpProbeHealthMonitor.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix nullable annotation on TrySuspectSilo explicit implementation

Match the IMembershipManager interface's SiloAddress? annotation on the
indirectProbingSilo parameter.  The file uses #nullable disable, so add
a localized #nullable enable/disable around the line.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Change IClusterHealthMonitor.SiloMonitors to IReadOnlyDictionary

Use the more general IReadOnlyDictionary<SiloAddress, SiloHealthMonitor>
instead of ImmutableDictionary so implementations are not forced to use
a specific collection type.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Add CancellationToken to all IMembershipManager Task methods

Add a required (no default value) CancellationToken parameter to every
Task-returning method on IMembershipManager:

- UpdateLocalStatus
- TryKillSilo
- TrySuspectSilo
- Refresh (remove existing defaults)
- ProcessGossipSnapshot
- UpdateIAmAlive

Update the MembershipTableManager explicit implementations and all call
sites to pass an appropriate token (shutdown token, caller's token, or
CancellationToken.None where none is available).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix ClusterHealthMonitor not participating in silo lifecycle

TryAddFromExisting skips when ANY registration for the service type
already exists.  Since ILifecycleParticipant<ISiloLifecycle> and
IHealthCheckParticipant have many prior registrations, the forwarding
entries for IClusterHealthMonitor were silently dropped.  This meant
ClusterHealthMonitor never started its probe loop, breaking failure
detection in longer-running tests (Azure Storage provider tests).

Change TryAddFromExisting to AddFromExisting so the forwarding
registrations are always added.  The TryAddSingleton on the
IClusterHealthMonitor registration itself already handles the
replacement scenario for external membership systems.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Rename DefaultProbeHealthMonitor to ProbingSiloHealthMonitor; fix threshold

- Rename DefaultProbeHealthMonitor to ProbingSiloHealthMonitor to preserve
  the log category closer to the original LocalSiloHealthMonitor.
- Fix CheckReceivedProbeResponses to use siloMonitors.Count for the
  threshold instead of the caller-provided monitoredNodeCount parameter,
  matching the original behavior that used the actual expander-graph
  monitor count.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Author: ReubenBond

src/Orleans.Core/Orleans.Core.csproj

@@ -8,6 +8,7 @@
     <OrleansBuildTimeCodeGen>true</OrleansBuildTimeCodeGen>
     <PackageReadmeFile>README.md</PackageReadmeFile>
   </PropertyGroup>
+
   <ItemGroup>
     <Content Include="buildMultiTargeting\Microsoft.Orleans.Core.targets">
       <PackagePath>%(Identity)</PackagePath>

src/Orleans.Runtime/Configuration/Validators/SiloClusteringValidator.cs

@@ -4,6 +4,7 @@
 using Microsoft.Extensions.Options;
 using Orleans.Configuration;
 using Orleans.Configuration.Validators;
+using Orleans.Runtime.MembershipService;
 
 namespace Orleans.Runtime.Configuration
 {
@@ -23,9 +24,25 @@ public SiloClusteringValidator(IServiceProvider serviceProvider)
         public void ValidateConfiguration()
         {
             var clusteringTableProvider = this.serviceProvider.GetService<IMembershipTable>();
-            if (clusteringTableProvider == null)
+
+            if (clusteringTableProvider is null)
             {
-                throw new OrleansConfigurationException(ClientClusteringValidator.ClusteringNotConfigured);
+                // No IMembershipTable configured. A custom IMembershipManager must be present
+                // (MembershipTableManager requires IMembershipTable, so it cannot be used).
+                IMembershipManager? membershipManager = null;
+                try
+                {
+                    membershipManager = this.serviceProvider.GetService<IMembershipManager>();
+                }
+                catch
+                {
+                    // Resolution failed — MembershipTableManager requires IMembershipTable.
+                }
+
+                if (membershipManager is null or MembershipTableManager)
+                {
+                    throw new OrleansConfigurationException(ClientClusteringValidator.ClusteringNotConfigured);
+                }
             }
 
             var clusterMembershipOptions = this.serviceProvider.GetRequiredService<IOptions<ClusterMembershipOptions>>().Value;
@@ -43,4 +60,4 @@ public void ValidateConfiguration()
             }
         }
     }
-}
+}

src/Orleans.Runtime/Core/ManagementGrain.cs

@@ -1,6 +1,7 @@
 using System;
 using System.Collections.Generic;
 using System.Linq;
+using System.Threading;
 using System.Threading.Tasks;
 using Microsoft.Extensions.Logging;
 using Orleans.Concurrency;
@@ -25,7 +26,7 @@ internal partial class ManagementGrain : Grain, IManagementGrain
         private readonly IInternalGrainFactory internalGrainFactory;
         private readonly ISiloStatusOracle siloStatusOracle;
         private readonly IVersionStore versionStore;
-        private readonly MembershipTableManager membershipTableManager;
+        private readonly IMembershipManager membershipManager;
         private readonly GrainManifest siloManifest;
         private readonly ClusterManifest clusterManifest;
         private readonly ILogger logger;
@@ -37,12 +38,12 @@ public ManagementGrain(
             ISiloStatusOracle siloStatusOracle,
             IVersionStore versionStore,
             ILogger<ManagementGrain> logger,
-            MembershipTableManager membershipTableManager,
+            IMembershipManager membershipManager,
             IClusterManifestProvider clusterManifestProvider,
             Catalog catalog,
             GrainLocator grainLocator)
         {
-            this.membershipTableManager = membershipTableManager;
+            this.membershipManager = membershipManager;
             this.siloManifest = clusterManifestProvider.LocalGrainManifest;
             this.clusterManifest = clusterManifestProvider.Current;
             this.internalGrainFactory = internalGrainFactory;
@@ -55,15 +56,15 @@ public ManagementGrain(
 
         public async Task<Dictionary<SiloAddress, SiloStatus>> GetHosts(bool onlyActive = false)
         {
-            await this.membershipTableManager.Refresh();
+            await this.membershipManager.Refresh(null, CancellationToken.None);
             return this.siloStatusOracle.GetApproximateSiloStatuses(onlyActive);
         }
 
         public async Task<MembershipEntry[]> GetDetailedHosts(bool onlyActive = false)
         {
-            await this.membershipTableManager.Refresh();
+            await this.membershipManager.Refresh(null, CancellationToken.None);
 
-            var table = this.membershipTableManager.MembershipTableSnapshot;
+            var table = this.membershipManager.CurrentSnapshot;
 
             MembershipEntry[] result;
             if (onlyActive)

src/Orleans.Runtime/Hosting/DefaultSiloServices.cs

@@ -162,20 +162,22 @@ internal static void AddDefaultServices(ISiloBuilder builder)
             services.AddFromExisting<ILifecycleParticipant<ISiloLifecycle>, DeploymentLoadPublisher>();
 
             services.AddSingleton<IAsyncTimerFactory, AsyncTimerFactory>();
-            services.AddSingleton<MembershipTableManager>();
-            services.AddFromExisting<IHealthCheckParticipant, MembershipTableManager>();
-            services.AddFromExisting<ILifecycleParticipant<ISiloLifecycle>, MembershipTableManager>();
+
+            services.TryAddSingleton<IMembershipManager, MembershipTableManager>();
+            services.AddFromExisting<IHealthCheckParticipant, IMembershipManager>();
+            services.AddFromExisting<ILifecycleParticipant<ISiloLifecycle>, IMembershipManager>();
             services.AddSingleton<MembershipSystemTarget>();
             services.AddFromExisting<IMembershipService, MembershipSystemTarget>();
             services.AddFromExisting<ILifecycleParticipant<ISiloLifecycle>, MembershipSystemTarget>();
             services.AddSingleton<IMembershipGossiper, MembershipGossiper>();
             services.AddSingleton<IRemoteSiloProber, RemoteSiloProber>();
             services.AddSingleton<SiloStatusOracle>();
             services.TryAddFromExisting<ISiloStatusOracle, SiloStatusOracle>();
-            services.AddSingleton<ClusterHealthMonitor>();
-            services.AddFromExisting<ILifecycleParticipant<ISiloLifecycle>, ClusterHealthMonitor>();
-            services.AddFromExisting<IHealthCheckParticipant, ClusterHealthMonitor>();
+            services.TryAddSingleton<IClusterHealthMonitor, ClusterHealthMonitor>();
+            services.AddFromExisting<ILifecycleParticipant<ISiloLifecycle>, IClusterHealthMonitor>();
+            services.AddFromExisting<IHealthCheckParticipant, IClusterHealthMonitor>();
             services.AddSingleton<ProbeRequestMonitor>();
+            services.TryAddSingleton<IProbeHealthMonitor, ProbingSiloHealthMonitor>();
             services.AddSingleton<LocalSiloHealthMonitor>();
             services.AddFromExisting<ILocalSiloHealthMonitor, LocalSiloHealthMonitor>();
             services.AddFromExisting<ILifecycleParticipant<ISiloLifecycle>, LocalSiloHealthMonitor>();
@@ -535,7 +537,7 @@ private class AllowOrleansTypes : ITypeNameFilter
         {
             public bool? IsTypeNameAllowed(string typeName, string assemblyName)
             {
-                if (assemblyName is { Length: > 0} && assemblyName.Contains("Orleans"))
+                if (assemblyName is { Length: > 0 } && assemblyName.Contains("Orleans"))
                 {
                     return true;
                 }

src/Orleans.Runtime/MembershipService/ClusterHealthMonitor.cs

@@ -18,13 +18,14 @@ namespace Orleans.Runtime.MembershipService
 {
     /// <summary>
     /// Responsible for ensuring that this silo monitors other silos in the cluster.
+    /// This is the default, probe-based implementation of <see cref="IClusterHealthMonitor"/>.
     /// </summary>
-    internal partial class ClusterHealthMonitor : IHealthCheckParticipant, ILifecycleParticipant<ISiloLifecycle>, ClusterHealthMonitor.ITestAccessor, IDisposable, IAsyncDisposable
+    internal partial class ClusterHealthMonitor : IClusterHealthMonitor, ClusterHealthMonitor.ITestAccessor, IDisposable, IAsyncDisposable
     {
         private readonly CancellationTokenSource shutdownCancellation = new CancellationTokenSource();
         private readonly ILocalSiloDetails localSiloDetails;
         private readonly IServiceProvider serviceProvider;
-        private readonly MembershipTableManager membershipService;
+        private readonly IMembershipManager membershipManager;
         private readonly ILogger<ClusterHealthMonitor> log;
         private readonly IFatalErrorHandler fatalErrorHandler;
         private readonly IOptionsMonitor<ClusterMembershipOptions> clusterMembershipOptions;
@@ -46,15 +47,15 @@ internal interface ITestAccessor
 
         public ClusterHealthMonitor(
             ILocalSiloDetails localSiloDetails,
-            MembershipTableManager membershipService,
+            IMembershipManager membershipManager,
             ILogger<ClusterHealthMonitor> log,
             IOptionsMonitor<ClusterMembershipOptions> clusterMembershipOptions,
             IFatalErrorHandler fatalErrorHandler,
             IServiceProvider serviceProvider)
         {
             this.localSiloDetails = localSiloDetails;
             this.serviceProvider = serviceProvider;
-            this.membershipService = membershipService;
+            this.membershipManager = membershipManager;
             this.log = log;
             this.fatalErrorHandler = fatalErrorHandler;
             this.clusterMembershipOptions = clusterMembershipOptions;
@@ -71,14 +72,14 @@ public ClusterHealthMonitor(
         /// <summary>
         /// Gets the collection of monitored silos.
         /// </summary>
-        public ImmutableDictionary<SiloAddress, SiloHealthMonitor> SiloMonitors => this.monitoredSilos;
+        public IReadOnlyDictionary<SiloAddress, SiloHealthMonitor> SiloMonitors => this.monitoredSilos;
 
         private async Task ProcessMembershipUpdates()
         {
             try
             {
                 LogDebugStartingToProcessMembershipUpdates(log);
-                await foreach (var tableSnapshot in this.membershipService.MembershipTableUpdates.WithCancellation(this.shutdownCancellation.Token))
+                await foreach (var tableSnapshot in this.membershipManager.MembershipUpdates.WithCancellation(this.shutdownCancellation.Token))
                 {
                     var utcNow = DateTime.UtcNow;
 
@@ -131,9 +132,9 @@ private async Task EvictStaleStateSilos(
                     try
                     {
                         LogDebugStaleSiloFound(log);
-                        await this.membershipService.TryToSuspectOrKill(member.Key);
+                        await this.membershipManager.TrySuspectSilo(member.Key, null, this.shutdownCancellation.Token);
                     }
-                    catch(Exception exception)
+                    catch (Exception exception)
                     {
                         LogErrorTryToSuspectOrKillFailed(log, exception, member.Value.SiloAddress, member.Value.Status);
                     }
@@ -316,12 +317,12 @@ private async Task OnProbeResultInternal(SiloHealthMonitor monitor, ProbeResult
             {
                 if (probeResult.Status == ProbeResultStatus.Failed && probeResult.FailedProbeCount >= this.clusterMembershipOptions.CurrentValue.NumMissedProbesLimit)
                 {
-                    await this.membershipService.TryToSuspectOrKill(monitor.TargetSiloAddress).ConfigureAwait(false);
+                    await this.membershipManager.TrySuspectSilo(monitor.TargetSiloAddress, null, this.shutdownCancellation.Token).ConfigureAwait(false);
                 }
             }
             else if (probeResult.Status == ProbeResultStatus.Failed)
             {
-                await this.membershipService.TryToSuspectOrKill(monitor.TargetSiloAddress, probeResult.Intermediary).ConfigureAwait(false);
+                await this.membershipManager.TrySuspectSilo(monitor.TargetSiloAddress, probeResult.Intermediary, this.shutdownCancellation.Token).ConfigureAwait(false);
             }
         }
 

src/Orleans.Runtime/MembershipService/ClusterMembershipService.cs

@@ -13,22 +13,22 @@ namespace Orleans.Runtime
     internal partial class ClusterMembershipService : IClusterMembershipService, ILifecycleParticipant<ISiloLifecycle>, IDisposable
     {
         private readonly AsyncEnumerable<ClusterMembershipSnapshot> updates;
-        private readonly MembershipTableManager membershipTableManager;
+        private readonly IMembershipManager membershipManager;
         private readonly ILogger<ClusterMembershipService> log;
         private readonly IFatalErrorHandler fatalErrorHandler;
         private ClusterMembershipSnapshot snapshot;
 
         public ClusterMembershipService(
-            MembershipTableManager membershipTableManager,
+            IMembershipManager membershipManager,
             ILogger<ClusterMembershipService> log,
             IFatalErrorHandler fatalErrorHandler)
         {
-            this.snapshot = membershipTableManager.MembershipTableSnapshot.CreateClusterMembershipSnapshot();
+            this.snapshot = membershipManager.CurrentSnapshot.CreateClusterMembershipSnapshot();
             this.updates = new AsyncEnumerable<ClusterMembershipSnapshot>(
                 initialValue: this.snapshot,
                 updateValidator: (previous, proposed) => proposed.Version > previous.Version,
                 onPublished: update => Interlocked.Exchange(ref this.snapshot, update));
-            this.membershipTableManager = membershipTableManager;
+            this.membershipManager = membershipManager;
             this.log = log;
             this.fatalErrorHandler = fatalErrorHandler;
         }
@@ -37,7 +37,7 @@ public ClusterMembershipSnapshot CurrentSnapshot
         {
             get
             {
-                var tableSnapshot = this.membershipTableManager.MembershipTableSnapshot;
+                var tableSnapshot = this.membershipManager.CurrentSnapshot;
                 if (this.snapshot.Version == tableSnapshot.Version)
                 {
                     return this.snapshot;
@@ -64,25 +64,25 @@ async ValueTask RefreshAsync(MembershipVersion v, CancellationToken cancellation
                 do
                 {
                     cancellationToken.ThrowIfCancellationRequested();
-                    if (!didRefresh || this.membershipTableManager.MembershipTableSnapshot.Version < v)
+                    if (!didRefresh || this.membershipManager.CurrentSnapshot.Version < v)
                     {
-                        await this.membershipTableManager.Refresh();
+                        await this.membershipManager.Refresh(null, cancellationToken);
                         didRefresh = true;
                     }
 
                     await Task.Delay(TimeSpan.FromMilliseconds(10), cancellationToken);
-                } while (this.snapshot.Version < v || this.snapshot.Version < this.membershipTableManager.MembershipTableSnapshot.Version);
+                } while (this.snapshot.Version < v || this.snapshot.Version < this.membershipManager.CurrentSnapshot.Version);
             }
         }
 
-        public async Task<bool> TryKill(SiloAddress siloAddress) => await this.membershipTableManager.TryKill(siloAddress);
+        public async Task<bool> TryKill(SiloAddress siloAddress) => await this.membershipManager.TryKillSilo(siloAddress, CancellationToken.None);
 
         private async Task ProcessMembershipUpdates(CancellationToken ct)
         {
             try
             {
                 LogDebugStartingToProcessMembershipUpdates(log);
-                await foreach (var tableSnapshot in this.membershipTableManager.MembershipTableUpdates.WithCancellation(ct))
+                await foreach (var tableSnapshot in this.membershipManager.MembershipUpdates.WithCancellation(ct))
                 {
                     this.updates.TryPublish(tableSnapshot.CreateClusterMembershipSnapshot());
                 }

src/Orleans.Runtime/MembershipService/IClusterHealthMonitor.cs

@@ -0,0 +1,48 @@
+#nullable enable
+
+using System.Collections.Generic;
+
+namespace Orleans.Runtime.MembershipService;
+
+/// <summary>
+/// Responsible for monitoring the health of silos in the cluster and detecting failures.
+/// </summary>
+/// <remarks>
+/// <para>
+/// Orleans supports two failure detection models:
+/// </para>
+/// <list type="number">
+///   <item>
+///     <description>
+///       <b>Built-in probe-based detection</b> (default): Orleans silos actively probe each other
+///       using an expander graph topology. When a silo misses too many probes, it is suspected
+///       and eventually declared dead. This is implemented by <see cref="ClusterHealthMonitor"/>.
+///     </description>
+///   </item>
+///   <item>
+///     <description>
+///       <b>External failure detection</b>: An external membership system
+///       handles failure detection through its own protocol. In this case, register a no-op
+///       implementation to disable Orleans' built-in probing while allowing the external system
+///       to manage membership changes through <see cref="IMembershipManager"/>.
+///     </description>
+///   </item>
+/// </list>
+/// <para>
+/// When using external failure detection, liveness IS still enabled - it's just handled by
+/// the external system rather than Orleans' probe-based mechanism. This is different from
+/// <c>ClusterMembershipOptions.LivenessEnabled = false</c>, which was a workaround that
+/// disabled the DeclareDead step without properly disabling the probing infrastructure.
+/// </para>
+/// </remarks>
+internal interface IClusterHealthMonitor : ILifecycleParticipant<ISiloLifecycle>, IHealthCheckParticipant
+{
+    /// <summary>
+    /// Gets the collection of silos currently being monitored by this silo.
+    /// </summary>
+    /// <remarks>
+    /// For external failure detection systems, this returns an empty dictionary since
+    /// monitoring is handled externally.
+    /// </remarks>
+    IReadOnlyDictionary<SiloAddress, SiloHealthMonitor> SiloMonitors { get; }
+}