[Service Bus] TSG Lock Lost Updates (Azure#48349)

* [Service Bus] TSG Updates

The focus of these changes is to update the Troubleshooting Guide (TSG) for
Service Bus to expand on potential causes for locks being lost.

Author: jsquire

sdk/servicebus/Azure.Messaging.ServiceBus/TROUBLESHOOTING.md

@@ -226,7 +226,27 @@ When attempting to do a batch receive, i.e. passing a `maxMessages` value of 2 o
 
 ### Message or session lock is lost before lock expiration time
 
-The Service Bus service leverages the AMQP protocol, which is stateful. Due to the nature of the protocol, if the link that connects the client and the service is detached after a message is received, but before the message is settled, the message is not able to be settled on reconnecting the link. Links can be detached due to a short-term transient network failure, a network outage, or due to the service enforced 10-minute idle timeout. The reconnection of the link happens automatically as a part of any operation that requires the link, i.e. settling or receiving messages. Because of this, you may encounter `ServiceBusException` with `Reason` of `MessageLockLost` or `SessionLockLost` even if the lock expiration time has not yet passed. 
+A Service Bus queue or topic subscription has a lock duration set at the resource level. When the receiver client pulls a message from the resource, the Service Bus broker applies an initial lock to the message. The initial lock lasts for the lock duration set at the resource level. If the message lock isn't renewed before it expires, then the Service Bus broker releases the message to make it available for other receivers. If the application tries to complete or abandon a message after the lock expiration, the call fails.
+
+Service Bus leverages the AMQP protocol, which is stateful. Due to the nature of the protocol, if the link that connects the client and the service is detached after a message is received, but before the message is settled, the message is not able to be settled on reconnecting the link. Links can be detached due to a short-term transient network failure, a network outage, or due to the service enforced 10-minute idle timeout. The reconnection of the link happens automatically as a part of any operation that requires the link, i.e. settling or receiving messages. Because of this, you may encounter `ServiceBusException` with `Reason` of `MessageLockLost` or `SessionLockLost` even if the lock expiration time has not yet passed. 
+
+In addition, the following usage patterns and scenarios may cause locks to be lost:
+
+- When using a receiver type, the application's message processing time exceeds the lock duration set at the resource level and the lock has not been renewed.  Locks are renewed by calling [RenewMessageLockAsync](https://learn.microsoft.com/dotnet/api/azure.messaging.servicebus.servicebusreceiver.renewmessagelockasync?view=azure-dotnet) or [RenewSessionLockAsync](https://learn.microsoft.com/dotnet/api/azure.messaging.servicebus.servicebussessionreceiver.renewsessionlockasync?view=azure-dotnet) on the receiver.
+
+- The application's message processing time exceeds both the lock duration set at the resource level and the [MaxAutoLockRenewalDuration](https://learn.microsoft.com/dotnet/api/azure.messaging.servicebus.servicebusprocessoroptions.maxautolockrenewalduration?view=azure-dotnet) configured for your `ServiceBusProcessor` or [MaxAutoLockRenewalDuration](https://learn.microsoft.com/dotnet/api/azure.messaging.servicebus.servicebussessionprocessoroptions.maxautolockrenewalduration?view=azure-dotnet) for your `ServiceBusSessionProcessor`.  Note that if the lock renew duration is not set explicitly, it defaults to 5 minutes.
+
+  This can be mitigated by manually renewing locks by calling [RenewMessageLockAsync](https://learn.microsoft.com/dotnet/api/azure.messaging.servicebus.processmessageeventargs.renewmessagelockasync?view=azure-dotnet#azure-messaging-servicebus-processmessageeventargs-renewmessagelockasync(azure-messaging-servicebus-servicebusreceivedmessage-system-threading-cancellationtoken)) or [RenewSessionLockAsync](https://learn.microsoft.com/dotnet/api/azure.messaging.servicebus.processsessionmessageeventargs.renewsessionlockasync?view=azure-dotnet) on the event arguments passed to your handler.  
+
+- The application has turned on the Prefetch feature by setting the `PrefetchCount` property in the options to a non-zero value. When Prefetch is enabled, the client will retrieve the number of messages equal to the count from Service Bus and store them in a memory buffer. The messages stay in the prefetch buffer until they're read by the application. The client cannot access messages in the prefetch buffer and is unable to extend their locks. If the application processing takes long enough that message locks expire while held in prefetch, the application will read the messages with an already expired lock. For more information, see [Why is Prefetch not the default option?](https://learn.microsoft.com/azure/service-bus-messaging/service-bus-prefetch?tabs=net#why-is-prefetch-not-the-default-option)
+
+- The host environment lacks enough CPUs or has shortages of CPU cycles intermittently that delays the lock renew task from running on time.
+
+- The host system time isn't accurate - for example, the clock is skewed - delaying the lock renew task and keeping it from running on time.
+
+- The application has configured a degree of concurrency too aggressive for the host's resources using the [MaxConcurrentCals](https://learn.microsoft.com/dotnet/api/azure.messaging.servicebus.servicebusprocessoroptions.maxconcurrentcalls?view=azure-dotnet) option for `ServiceBusProcessor` or a combination of [MaxConcurrentSessions](https://learn.microsoft.com/dotnet/api/azure.messaging.servicebus.servicebussessionprocessoroptions.maxconcurrentsessions?view=azure-dotnet) and [MaxConcurrentCallsPerSession](https://learn.microsoft.com/dotnet/api/azure.messaging.servicebus.servicebussessionprocessoroptions.maxconcurrentcallspersession?view=azure-dotnet) for `ServiceBusSessionProcessor`.  This will cause a high number of concurrent background network operations to run, competing for resources and potentially either being queued on the client or triggering Service Bus throttles.
+
+  **NOTE:** If the application host environment is not sufficiently resourced, locks can still be lost even if there are only a few lock renew tasks running.  When using processor types, we recommend at least two full CPU cores.  For more context and discussion, see [Processor appears to hang or have latency issues when using high concurrency](#processor-appears-to-hang-or-have-latency-issues-when-using-high-concurrency).
 
 ### How to browse scheduled or deferred messages