Skip to content

Commit a694f6a

Browse files
committed
admission: call LowerBound(0) instead of Reset on token bucket for elastic CPU
The goal of this periodic operation is to ensure the token bucket is not extremely negative, which can happen if elastic work consumes more CPU than initially granted. However, instead of merely ensuring a lower bound of 0 tokens, it was resetting the token bucket to be full, which would admit another burst. Observed in https://cockroachlabs.slack.com/archives/C01SRKWGHG8/p1745891638545989?thread_ts=1745881523.172559&cid=C01SRKWGHG8. Epic: none Release note: None
1 parent fd30157 commit a694f6a

File tree

1 file changed

+6
-6
lines changed

1 file changed

+6
-6
lines changed

pkg/util/admission/elastic_cpu_granter.go

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -208,12 +208,12 @@ func (e *elasticCPUGranter) setUtilizationLimit(utilizationLimit float64) {
208208
e.mu.utilizationLimit = utilizationLimit
209209
e.mu.tb.UpdateConfig(tokenbucket.TokensPerSecond(rate), tokenbucket.Tokens(rate))
210210
if now := timeutil.Now(); now.Sub(e.mu.tbLastReset) > 15*time.Second { // TODO(irfansharif): make this is a cluster setting?
211-
// Periodically reset the token bucket. This is just defense-in-depth
212-
// and at worst, over-admits. We've seen production clusters where the
213-
// token bucket was severely in debt and caused wait queue times of
214-
// minutes, which can be long enough to fail backups completely
215-
// (#102817).
216-
e.mu.tb.Reset()
211+
// Periodically ensure the tokens are at least 0. This is just
212+
// defense-in-depth and at worst, over-admits. We've seen production
213+
// clusters where the token bucket was severely in debt and caused wait
214+
// queue times of minutes, which can be long enough to fail backups
215+
// completely (#102817).
216+
e.mu.tb.EnsureLowerBound(0)
217217
e.mu.tbLastReset = now
218218
}
219219
e.metrics.NanosExhaustedDuration.Update(e.mu.tb.Exhausted().Microseconds())

0 commit comments

Comments
 (0)