A strange exception occurs after some hours of working in an Azure Function environment

[ynnova]

Dears all,

I have an Azure Function connected to an Azure Iothub endpoint.
Each time new data arrives to the Iothub endpoint an Azure function (Consumption plan on Windows) is raised where some AsyncSession are created pointing to different database on same RavenDb cloud cluster.

The DocumentStore is Singleton.

After some hours of working the following exception starts to appear in the log:

2024-08-19T15:09:49.582 [Error] ________________________________>> Cannot complete the REALTIME ingestion. Exception: System.AggregateException: Failed to retrieve database topology from all known nodes.https://*****.ravendb.cloud -> The SSL connection could not be established, see inner exception. -> Authentication failed, see inner exception. -> The credentials supplied to the package were not recognizedhttps://*****.ravendb.cloud -> The SSL connection could not be established, see inner exception. -> Authentication failed, see inner exception. -> The credentials supplied to the package were not recognizedhttps://*****ravendb.cloud -> The SSL connection could not be established, see inner exception. -> Authentication failed, see inner exception. -> The credentials supplied to the package were not recognized (The SSL connection could not be established, see inner exception.) (The SSL connection could not be established, see inner exception.) (The SSL connection could not be established, see inner exception.)---> System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception.---> System.Security.Authentication.AuthenticationException: Authentication failed, see inner exception.---> System.ComponentModel.Win32Exception (0x8009030D): The credentials supplied to the package were not recognizedat System.Net.SSPIWrapper.AcquireCredentialsHandle(ISSPIInterface secModule, String package, CredentialUse intent, SCHANNEL_CRED* scc)at System.Net.Security.SslStreamPal.AcquireCredentialsHandle(CredentialUse credUsage, SCHANNEL_CRED* secureCredential)at System.Net.Security.SslStreamPal.AcquireCredentialsHandleSchannelCred(SslStreamCertificateContext certificateContext, SslProtocols protocols, EncryptionPolicy policy, Boolean isServer)at System.Net.Security.SslStreamPal.AcquireCredentialsHandle(SslStreamCertificateContext certificateContext, SslProtocols protocols, EncryptionPolicy policy, Boolean isServer, Boolean newCredentialsRequested)--- End of inner exception stack trace ---at System.Net.Security.SslStreamPal.AcquireCredentialsHandle(SslStreamCertificateContext certificateContext, SslProtocols protocols, EncryptionPolicy policy, Boolean isServer, Boolean newCredentialsRequested)at System.Net.Security.SecureChannel.AcquireClientCredentials(Byte[]& thumbPrint, Boolean newCredentialsRequested)at System.Net.Security.SecureChannel.GenerateToken(ReadOnlySpan1 inputBuffer, Byte[]& output)at System.Net.Security.SecureChannel.NextMessage(ReadOnlySpan1 incomingBuffer)at System.Net.Security.SslStream.ProcessBlob(Int32 frameSize)at System.Net.Security.SslStream.ReceiveBlobAsync[TIOAdapter](TIOAdapter adapter)at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm)at System.Net.Http.ConnectHelper.EstablishSslConnectionAsync(SslClientAuthenticationOptions sslOptions, HttpRequestMessage request, Boolean async, Stream stream, CancellationToken cancellationToken)--- End of inner exception stack trace ---at System.Net.Http.ConnectHelper.EstablishSslConnectionAsync(SslClientAuthenticationOptions sslOptions, HttpRequestMessage request, Boolean async, Stream stream, CancellationToken cancellationToken)at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)at System.Net.Http.HttpConnectionPool.AddHttp2ConnectionAsync(HttpRequestMessage request)at System.Threading.Tasks.TaskCompletionSourceWithCancellation1.WaitWithCancellationAsync(CancellationToken cancellationToken)at System.Net.Http.HttpConnectionPool.GetHttp2ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)at System.Net.Http.DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)at System.Net.Http.DecompressionHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)at System.Net.Http.HttpClient.g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)at Raven.Client.Http.RequestExecutor.SendAsync[TResult](ServerNode chosenNode, RavenCommand1 command, SessionInfo sessionInfo, HttpRequestMessage request, CancellationToken token)

Any suggestion?

[ynnova]

I can share details in private discussion eventually.

[ppekrol]

I think this can be related to missing WEBSITE_LOAD_USER_PROFILE
https://github.com/projectkudu/kudu/wiki/Configurable-settings#add-user-profile-support-for-a-site
and maybe even WEBSITE_LOAD_CERTIFICATES https://github.com/projectkudu/kudu/wiki/Configurable-settings#load-certificates-in-the-web-app
Some context: https://stackoverflow.com/questions/54983993/aure-function-app-returns-the-credentials-supplied-to-the-package-were-not-reco
Can you check if you have them set?

[ynnova]

Tks. I will check your suggensions.

[ynnova]

Dear @ppekrol. It seems like the WEBSITE_LOAD_USER_PROFILE=1 setting has solved my problem.
Thank you for your contribution.