What can I do about `Retry header did not contain a valid timeout`?

[meichstedt]

Hey there,

I've been running into 3 occurrences of this error: Retry header did not contain a valid timeout (from here)

The client was using the outdated 5.15.0 WebApi, so it didn't tell me the url.

My main questions are

• is there a way to prevent getting rate limit responses w/o retrySec information?
• are there any ideas about what causes this?
• Is there a best practice on how to handle these?

Thanks in advance

Packages:

Select all that apply:

• @slack/web-api
• @slack/rtm-api
• @slack/webhooks
• @slack/oauth
• @slack/socket-mode
• I don't know

Reproducible in:

npm ls | grep -o "\S\+@\S\+$" | tr @ ' ' | awk -v q='"' '{print q$1q": "q"^"$2q","}' | grep slack
node --version
sw_vers && uname -v # or `ver`

The Slack SDK version

@slack/web-api: 5.15.0

Python runtime version

v15.14.0

OS info

Chrome/98.0.4758.80
Macintosh; Intel Mac OS X 10_15_7

Steps to reproduce:

(Share the commands to run, source code, and project settings)

1. Send a slack message
2. Receive a 429
3. Miss the timeout

Expected result:

I don't know, assume a default timeoutSec?

Actual result:

(Tell what actually happened with logs, screenshots)

Requirements

For general questions/issues about Slack API platform or its server-side, could you submit questions at https://my.slack.com/help/requests/new instead. 🙇

Please read the Contributing guidelines and Code of Conduct before creating this issue or pull request. By submitting, you are agreeing to those rules.

[filmaj]

Hey @meichstedt, sorry about the trouble you're experiencing. If at all possible, if you have some logs to share around when this happened and which URL/API you saw was returning no retry delay information in its header, that would be helpful to dig into this further. I can then dig into the relevant backend API code to see how this could come about.

At this point, I don't have any good answer for your first two questions, but with more information like logs and which APIs, we could take a deeper look.

As for workarounds, what about wrapping the API invocation with a try/catch, inspecting the error message for "Retry header did not contain a valid timeout", and if it contains that, defaulting to some reasonable default delay before trying again? Certainly not ideal, and the intention behind the retry and queueing features of web-api is so that you don't have to think about that! Ideally we could get to the bottom of the underlying issue and address that.

[meichstedt]

Hey @filmaj thanks for getting back to me so quickly.
I don't have logs unfortunately since the requests were made on a client machine, however I do have timestamps:

timestamp: 1644428130.537
timestamp: 1644514610.975
timestamp: 1644516295.745

I also have IP addresses of the machine that did the requests – I can share those if you have a non-public way to exchange.

The code in question is only using the conversations.list endpoint with these parameters:

return this.ctx.client.conversations.list({
  cursor,
  types: "public_channel,private_channel,im,mpim",
});

where cursor is string | undefined and set via results.response_metadata?.next_cursor || ""

I'm happy to catch those and retry later; we have similar workaround in other places; the main thing that worries me in this case is that it seems that the client didn't do many requests at all – if you could verify using logs/IP addresses, that would be great.

Let me know if the timestamps help, and whether I can share the IP somehow.

[filmaj]

Hey @meichstedt, thanks a lot for the extra information, that's really helpful. I am asking around internally to see if if/how I can get access to the production logs for these APIs based on the timestamps and potentially the IPs you could provide. I am relatively new to the company so I'm still getting my bearings and it might take me a little bit to figure out where to look for this stuff - but as soon as I figure this out I'll report back. FYI it probably will have to wait until after the weekend.

[filmaj]

Hey @meichstedt, I can indeed peruse our logs to try to identify the requests and see what happened. Beyond timestamps, can you also provide me with the timezone of your machines (unless the timestamps are UTC). I can also use the IP addresses to help filter through the information. Any other info you can provide to me to help narrow down the requests, like channel IDs, bot IDs, app IDs, would be helpful too. All this information you could ideally provide to me over a secure channel. I use keybase.io (keybase.io/filmaj) - would that work for you?

[meichstedt]

Hi @filmaj perfect thank you so much. Yes the timestamps are all UTC; I'll send you all that information via keybase.

[filmaj]

Got the info, thanks, I will dig around!

[filmaj]

Alright, I've dug around, and I believe I can find the relevant logs from the backend for these calls. I see many calls to the conversations.list API with a status=ok log, followed by a single error=rate_limited log. Note that the conversations.list API is a Tier 2 rate limited API, which officially only supports about 20 requests per minute. By my rough calculations, your app is issuing about 10 requests per second. So most definitely your app is bound to hit the rate limit for this API quite quickly 😄

Unfortunately, on our side, while we do log the request parameters and some other helpful tidbits, we do not log the full response returned. As such, I am limited to trying to work backwards through the logic of the backend code to see how this situation could arise. That said, we do log a specific SHA of the backend code as well as a kind of stack trace through the backend code - so I can see the separate paths of logic executed between the rate-limited call and the successful calls.

I will list out my assumptions based on what I've seen from the backend code and how the node-slack-sdk parses the headers:

1. I am 99.9% sure that some kind of value is returned by the backend in the Retry-After HTTP response header - what kind of value, I am not sure and it is hard for me to infer as it comes from a separate "quota" system. There is some float math applied to this value before writing it to the response header as it seems the quota system stores rate limits in nanoseconds whereas the Retry-After header returns it to clients as seconds. So perhaps there is an issue here in the float math followed by casting to an integer.
2. Looking at the node-slack-sdk's parseRetryHeaders method, if the header is defined and can be parsed as an integer, then the exception you noted in your original post should never arise. However, if the header is not present or it cannot be parsed as an integer / ends up as a NaN, then the exception you noted is raised.

I think the key to solving this problem lies in identifying what was returned in these rate-limited response headers.

Perhaps as a baby step to helping with this issue, I can improve the exception being raised such that it records the values returned in the HTTP response's Retry-After header? This way, whenever this situation arises again, at least we can get a clue that would be helpful in determining how this could arise, and could inform our next step after that.

Sorry this is so indirect / taking so long! However, if you have the time and patience to work with me on this issue, I hope we can get to the bottom of it 😃