Skip to content

ref(ratelimiter): Reduce redis socket timeout and emit metric on timeout #7662

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
phacops merged 8 commits into from
Jan 23, 2026
Merged

ref(ratelimiter): Reduce redis socket timeout and emit metric on timeout #7662

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
8d3c5b7
ref(ratelimiter): Reduce redis socket timeout and emit metric on timeout
phacops Jan 22, 2026
0d9ea2e
ref(ratelimiter): Skip logging for redis timeout exceptions
phacops Jan 22, 2026
99c073f
ref(ratelimiter): Remove redundant metric from generic exception handler
phacops Jan 22, 2026
14618be
ref(ratelimiter): Move RateLimitStats return into finally block
phacops Jan 22, 2026
c7f29b1
Clean the PR
phacops Jan 22, 2026
bab9792
ref(ratelimiter): Move redis timeout handling to allocation policy
phacops Jan 22, 2026
97c1631
Re-use the same metric name and add a tag instead
phacops Jan 22, 2026
82dcf43
Robot is wrong, there is a default value to 1 but we can be explicit
phacops Jan 22, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion snuba/redis.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -128,7 +128,7 @@ class RedisClientKey(Enum):
socket_timeout=1,
),
RedisClientKey.RATE_LIMITER: _initialize_specialized_redis_cluster(
settings.REDIS_CLUSTERS["rate_limiter"], socket_timeout=0.5
settings.REDIS_CLUSTERS["rate_limiter"], socket_timeout=0.1
),
RedisClientKey.SUBSCRIPTION_STORE: _initialize_specialized_redis_cluster(
settings.REDIS_CLUSTERS["subscription_store"],
Expand Down
6 changes: 6 additions & 0 deletions snuba/state/rate_limit.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,8 @@
from typing import Any, Iterator, MutableMapping, Optional, Sequence, Type
from typing import ChainMap as TypingChainMap

from redis.exceptions import TimeoutError as RedisTimeoutError

from snuba import environment, state
from snuba.redis import RedisClientKey, get_redis_client
from snuba.state import get_configs, set_config
Expand Down Expand Up @@ -279,6 +281,8 @@ def rate_limit_start_request(
# if something goes wrong, we don't want to block the request,
# set the values such that they pass under any limit
logger.exception(ex)
if isinstance(ex, RedisTimeoutError):
metrics.increment("ratelimiter_redis_timeout", tags={"function": "start_request"})
return RateLimitStats(rate=-1, concurrent=-1)

per_second = historical / float(state.rate_lookback_s)
Expand Down Expand Up @@ -313,6 +317,8 @@ def rate_limit_finish_request(
pipe.execute()
except Exception as ex:
logger.exception(ex)
Copy link
Member

@volokluev volokluev Jan 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you don't want to log this exeption. it just becomes a sentry error and we want to avoid that.

if isinstance(ex, RedisTimeoutError):
metrics.increment("ratelimiter_redis_timeout", tags={"function": "finish_request"})


@contextmanager
Expand Down
Loading