Reading & Writing too quickly - {"message":"Too Many Requests"}

[requiem240sx]

The Problem

Terraform can't run, as it is calling too many routing keys at the same time. Sometimes it works, other times it will fail. I've noticed this effects the plan as well as the apply. Sometimes it can't refresh the state, other times it skips creation of new routing keys. Sometimes the plan is not accurate, as it missed reading the correct routing keys etc.. This was not a problem when we had fewer routing keys and escalation policies (less than 25) however this has slowly became a bigger and bigger issue as we add more policies and routing keys. (We are well over 100 now)

This is the exact error:

module.victorops_REDACTED_team.victorops_routing_key.s1_routing_key: Refreshing state... [id=REDACTED_s1]
module.victorops_REDACTED_team.victorops_routing_key.s2_routing_key: Refreshing state... [id=REDACTED_s2]
module.victorops_REDACTED_team.victorops_routing_key.s3_routing_key: Refreshing state... [id=REDACTED_s3]
module.victorops_REDACTED_team.victorops_routing_key.s4_routing_key: Refreshing state... [id=REDACTED_s4]
module.victorops_REDACTED_team.victorops_routing_key.s1_routing_key: Refreshing state... [id=REDACTED_s1]
module.victorops_REDACTED_team.victorops_routing_key.s2_routing_key: Refreshing state... [id=REDACTED_s2]
module.victorops_REDACTED_team.victorops_routing_key.s3_routing_key: Refreshing state... [id=REDACTED_s3]
module.victorops_REDACTED_team.victorops_routing_key.s4_routing_key: Refreshing state... [id=REDACTED_s4]

│ Error: failed to get escaltion policy (429): {"message":"Too Many Requests"}

My suspect reason to why

This provider looks to be calling the read for each routing key with no sort of throttle/limit, however the API docs for routing keys says it has a limit of 2 call per second.

Retrieves a list of routing keys and associated teams. This API may be called a maximum of 2 times per second

It fails when trying to refresh many routing key(s) at roughly the same time.