Unable to establish SSL Connection issue - After OS Upgrade

[ajnavsun1806]

Hi,

We are using Service Fabric to host our service. We recently migrated to NET6.0. After migration, we started seeing this peculiar issue.
Network calls starts failing with "Unable to establish SSL Connection" for some dependent APIs. Restarting the process or VM both mitigates this issue until the next OS Upgrade, But with NET3.1, this issue was not present. We are unable to find a solution for this problem.

We use IHttpClientFactory with Polly for calling dependent APIs. We have multiple named clients with following configuration. The named clients are created on the basis of timeout instead for each dependent API.

services.AddHttpClient(Core.Constants.HttpRetryClientKeyDefaultTimeout)
    .SetHandlerLifetime(TimeSpan.FromMinutes(5))
    .AddPolicyHandler((services, request) =>
    {
        IDiagnosticsLogger logger = services.GetRequiredService<IDiagnosticsLogger>();
        return GetRetryOnTimeoutPolicy(60, options.RetryCount, logger);
    })
    .ConfigurePrimaryHttpMessageHandler(() =>
    {
        return new HttpClientHandler()
        {
            UseCookies = false,
            AllowAutoRedirect = false
        };
    });

The exception logged in the telemetry is: An exisiting connection was forcibly closed by the remote host.

The SSL connection could not be established, see inner exception. Exception Details : System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception.
 ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host..
 ---> System.Net.Sockets.SocketException (10054): An existing connection was forcibly closed by the remote host.
   --- End of inner exception stack trace ---
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource<System.Int32>.GetResult(Int16 token)
   at System.Net.Security.SslStream.<FillHandshakeBufferAsync>g__InternalFillHandshakeBufferAsync|189_0[TIOAdapter](TIOAdapter adap, ValueTask`1 task, Int32 minSize)
   at System.Net.Security.SslStream.ReceiveBlobAsync[TIOAdapter](TIOAdapter adapter)
   at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm)
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsync(SslClientAuthenticationOptions sslOptions, HttpRequestMessage request, Boolean async, Stream stream, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsync(SslClientAuthenticationOptions sslOptions, HttpRequestMessage request, Boolean async, Stream stream, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.AddHttp11ConnectionAsync(HttpRequestMessage request)
   at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync(CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.GetHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at Microsoft.Kiota.Http.HttpClientLibrary.Middleware.HeadersInspectionHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at Microsoft.Kiota.Http.HttpClientLibrary.Middleware.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at Microsoft.Kiota.Http.HttpClientLibrary.Middleware.RetryHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at Microsoft.Kiota.Http.HttpClientLibrary.Middleware.CompressionHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at Microsoft.Kiota.Http.HttpClientLibrary.HttpClientRequestAdapter.GetHttpResponseMessage(RequestInformation requestInfo, CancellationToken cancellationToken, Activity activityForAttributes, String claims, Boolean isStreamResponse)
   at Microsoft.Kiota.Http.HttpClientLibrary.HttpClientRequestAdapter.SendNoContentAsync(RequestInformation requestInfo, Dictionary`2 errorMapping, CancellationToken cancellationToken)
   at Microsoft.Graph.Core.Requests.BatchRequestBuilder.PostAsync(BatchRequestContent batchRequestContent, CancellationToken cancellationToken, Dictionary`2 errorMappings)
   at Microsoft.Graph.Core.Requests.BatchRequestBuilder.PostAsync(BatchRequestContentCollection batchRequestContentCollection, CancellationToken cancellationToken, Dictionary`2 errorMappings)
   at Microsoft.Admin.Coherence.Services.Core.MsGraphSdkClient.GetBatchResponseAsync[TResponse,TResponseCollection](GraphServiceClient graphClient, List`1 requestIds, BatchRequestContentCollection batchRequest, Func`2 convertToResponseList) in D:\dbs\el\9e7b\sources\dev\Coherence\Core\Common\Clients\MsGraphSdkClient.cs:line 916, Exception Message: The SSL connection could not be established, see inner exception., Exception Type: System.Net.Http.HttpRequestException, Exception HResult : -2146232800

We require all the help in resolving this issue. Thanks in advance.