contention: Remove unlimited retries in the contention resolver

Unlimited retries in the resolver for in progress transactions results in
contention events for aborted transactions getting stuck in the resolver
until the aborted transaction is removed from the txn id cache on the
respective gateway node.

If the gateway node is getting 100 txns per second, that roughly translates
to 7 hours.

If the transaction got aborted, we'd prefer to have the partially resolved
contention event available in a timely manner.

This commit changes unlimited retries for in progress transactions to 60
retries (30 minutes).

Fixes: #139258
Release note: None

Author: alyshanjahani-crl

pkg/sql/contention/resolver.go

@@ -7,7 +7,6 @@ package contention
 
 import (
 	"context"
-	"math"
 	"sort"
 	"strconv"
 
@@ -80,10 +79,11 @@ const (
 	// the case where the node is permanently removed from the cluster.
 	retryBudgetForRPCFailure = uint32(3)
 
-	// retryBudgetForTxnInProgress is a special value indicating that the resolver should
-	// indefinitely retry the resolution. This is because the retry is due to the
-	// transaction is still in progress.
-	retryBudgetForTxnInProgress = uint32(math.MaxUint32)
+	// retryBudgetForTxnInProgress is the number of times the resolverQueue will
+	// retry resolving until giving up. This is because the retry is due to the
+	// transaction is still in progress. 60 retries at the default value for the
+	// cluster setting txn_id_resolution_interval translates to 30 minutes.
+	retryBudgetForTxnInProgress = uint32(60)
 )
 
 // ResolverEndpoint is an alias for the TxnIDResolution RPC endpoint in the