azure: configure a default TryTimeout of 60 seconds

We have seen some evidence of stuck or slow requests to azure blob
storage. Setting a try timeout allows us to retry these slow operations
and may allow us to make forward progress if the underlying error is
transient. Operations that run into the TryTimeout will be internally
retried by the cloud package.

The TryTimeout can be controlled via the cloudstorage.azure.try.timeout
setting. Setting it to zero disables the per-attempt timeout.

60 seconds is a relatively long timeout. The main reason it is set that
long is the timeout is applied to read operations and CRDB sometimes
performs long lived stream reads when it is merging many SSTs during a
restore.

Release note: Add a default TryTimeout of 60 seconds for Azure Blob
Storage to mitigate occasional stuck operations.
Fixes: #154085

Author: jeffswenson

pkg/cloud/azure/azure_storage.go

@@ -12,6 +12,7 @@ import (
 	"net/url"
 	"path"
 	"strings"
+	"time"
 
 	"github.com/Azure/azure-sdk-for-go/sdk/azcore"
 	"github.com/Azure/azure-sdk-for-go/sdk/azidentity"
@@ -48,6 +49,12 @@ var maxRetries = settings.RegisterIntSetting(
 	"the maximum number of retries per Azure operation",
 	10)
 
+var tryTimeout = settings.RegisterDurationSetting(
+	settings.ApplicationLevel,
+	"cloudstorage.azure.try.timeout",
+	"the timeout for individual retry attempts in Azure operations",
+	60*time.Second)
+
 // A note on Azure authentication:
 //
 // The standardized way to authenticate a third-party identity to the Azure
@@ -242,6 +249,11 @@ func makeAzureStorage(
 	// Azure SDK defaults to 3 retries, which is too low to survive the 30 second
 	// brownout in TestAzureFaultInjection.
 	opts.Retry.MaxRetries = int32(maxRetries.Get(&args.Settings.SV))
+	// We occasionally see individual requests get stuck for 10+ minutes. If the
+	// source of the stuckness is transient or applies to individual
+	// connections/requests, then starting a new request after a timeout may
+	// succeed and allow the client to make forward progress.
+	opts.Retry.TryTimeout = tryTimeout.Get(&args.Settings.SV)
 
 	var azClient *service.Client
 	switch conf.Auth {