RetryBackoffFunc assumes the server always returns the Retry-After header during 429 responses

[mboruta1]

As discussed in detail here, the default RetryBackoffFunc function assumes that all 429 error responses will contain a Retry-After header, which is not the case. According to RFC6585, 429 responses:

   The response representations SHOULD include details explaining the
   condition, and MAY include a Retry-After header indicating how long
   to wait before making a new request.

Not all users own their Openstack installations and can modify their HTTP load balancer to specify this header. Those that cannot can indeed overwrite said function, but when using higher level tools such as Terraform the process for doing so is either not obvious or does not exist.

Modifying the default retry implementation to use a user-specified Retry-After value if one was not provided by the server would be a relatively simple change and would be simple to integrate into tools like terraform-provider-openstack (much like this PR).

Lastly, 429 errors can be returned by load balancers charged with QoS/rate limiting excessive users and do not necessarily (or at all, at least in my experience) originate from the Openstack API endpoints themselves.

[jtopjian]

@kayrus @ozerovandrei I wanted to get your thoughts since Terraform would see changes made to the RetryBackoffFunc function. What do you think about adding a default backoff time for cases when the Retry-After header doesn't exist?

[kayrus]

Shall we use a default timeout or add a timeout option in terraform config?

[jtopjian]

I'm thinking that because it's not guaranteed that OpenStack services will respond with a Retry-Backoff header, then the RetryBackoffFunc in this repo should be modified so that a default backoff timer is used when there is no Retry-After header.