429 (too many requests) after running 3 unit tests?

[axolotl-logic]

UPDATE: Can not make it to the end of 7 files, executed with maximum concurrency 1 individually, without being rate limited. This is a major blocker. Is there any way of increasing rate limits?

Hello,

A coworker and I are both getting 429s trying to run unit tests from the Evals/ directory. They only invoked the command three times (1 file each invocation) before being blocked and I made the mistake of running all of Evals/ in one invocation (still blocked 10 hours later). In both our cases, the first 3 succeeded and subsequent were 429s.

Is it possible to support more than a couple invocations in general?

Ideally we'd have this as part of our CI/CD, but that sounds like a recipe for instant-throttling.

Thank you,
Autumn.

TLDR: we get blocked after a few requests and remain blocked seemingly indefinitely.

[ruyrocha]

@axolotl-logic did you manage to solve it? I got two timeouts and a successful submission within 1 hour lifespan, and now I'm blocked - 429 {"status":429,"traceId":"00-ebdff2f5b45b396de0e27a7900ab88f2-b8b227f8677dc0ce-00"}

[kayyar-roblox]

We're looking into how to better document the limit + increase it. Thank you for the early feedback here.

[ruyrocha]

@kayyar-roblox the HTTP headers on #12 add a little bit of info, but they're not honored; I mean that I can make 100 requests in 24 hours, but I was blocked after some timeouts, hence I had 80+ remaining requests at that time.

[axolotl-logic]

Currently it polls every 10 seconds for changes, so you eat through the 100 requests quickly. That's at least my hypothesis. I've edited locally poll_interval to be 60 seconds and we'll see if that helps.

[ruyrocha]

Currently it polls every 10 seconds for changes, so you eat through the 100 requests quickly. That's at least my hypothesis. I've edited locally poll_interval to be 60 seconds and we'll see if that helps.

There's a bandaid for this: create a new account and generate new API key whenever you're blocked. It works 💯

[axolotl-logic]

We're looking into how to better document the limit + increase it. Thank you for the early feedback here.

Any update?