Retry functionality

[sreeise]

Implement retry functionality for the client to retry requests. Cases that this implementation should solve and objectives:

• Client is throttled and needs to check retry header to see when retry is possible. Same as 429 rate limits.
• 500 error codes have different possibilties and the Microsoft Graph Api may have their own way of returning certain 500 error codes. Some 500 errors are used in concert with a Retry-After header so that should be looked at. The 500 Internal server error will depend on the situation and error occuring.
• Whenever there is a retry after header then always use that to figure out when to retry.

Functionality:

• Retry after throttle using header value of time to wait. If possible, this should just be a flip of the switch to turn on. Meaning no other inputs should be required to be given except that this should be enabled (such passing true or false for enabled).
• Backoff or retry policy that provides
  • min time to retry
  • max time to retry,
  • max retries tried - we want to stop trying to send requests when we reach this
• backoff strategy - exponential backoff, and the retry after throttle mentioned above (name to be decided by who works on this). These strategies use the above policy inputs (min time/max time, etc)

Any other important use cases and implementation notes? Feedback welcome.

Update:

Implementation should include the following 2 retry types at the minimum and more can be added or updated later on as well:

• Throttle Retry: Handle retrying for when Microsoft throttles requests and sends back the 429 http error code and Retry-After header. The impl should check if the http code is a 429 and if it is then get the seconds from the Retry-After header, and wait the amount of seconds provided in the Retry-After header before retrying the request again. See https://learn.microsoft.com/en-us/graph/throttling#best-practices-to-handle-throttling
• Exponential backoff: retries requests exponentially, increasing the waiting time between retries up to a maximum backoff time. Each retry will have a wait time of 2x the previous wait time.

There may be cases where these conflict with each other and we will need to come up with a way to deal with this such as not using any other retry except the throttle retry when a 429 response code is returned.

[sreeise]

@DevLazio Created this for the retry feature we talked about. I listed a few things that I thought would be good objectives for this functionality and some of the things that typically good retry implementations have.

I found a few interesting crates. The first is the backoff crate which looks promising and provides mechanisms for doing retries and has reqwest examples.

The next is reqwest-retry which is a wrapper around reqwest that provides retry functionality. I don't want to use this crate mainly because it wraps the reqwest Client and provides its own interface to do requests which means you lose any choice of setting reqwest version, or using any features that this crate hasnt adopted yet. But its something to look at with regards to how a retry implementation could be done or some functionality can be done.

[sreeise]

This is the design document for retry handlers in Microsofts SDKs and I thinking following this specification will probably work out best. https://github.com/microsoftgraph/msgraph-sdk-design/blob/master/middleware/RetryHandler.md

Kiota is Microsofts code generation for different SDK's. They have good examples of Retry objects implemented with different scenarios and properties.

https://github.com/microsoft/kiota-http-dotnet/blob/main/src/Middleware/RetryHandler.cs
https://github.com/microsoft/kiota-http-dotnet/blob/main/src/Middleware/Options/RetryHandlerOption.cs

[DevLazio]

Thanks, will take a look at this !

[DevLazio]

I finally had a bit of time to look into this.
The backoff crate looks like the right choice for the task. The only pain point is using an async FnMut function so I had to do a bit of gymnastics around it.

Nothing is final, just looking for comment at this time.

Usage :

    let config = GraphClientConfiguration::new()
        .retries(true)
        .retries_config(RetriesConfig {
            initial_interval: Duration::from_millis(500),
            randomization_factor: 0.0,
            multiplier: 1.5,
            max_interval: Duration::from_secs(450),
            max_elapsed_time: Some(Duration::from_millis(450)),
        })
        .access_token(ACCESS_TOKEN);

    let client = Graph::from(config);

    for _i in 0..100 {
        let response = client
            .reports()
            .get_email_activity_counts_by_period("D180")
            .send()
            .await.unwrap();

        println!("Got response {}", response.status());
    }

It's not necessary to provide a RetriesConfig, there is already a default one with values that seems okay enough for GraphAPI imo.

[sreeise]

I finally had a bit of time to look into this. The backoff crate looks like the right choice for the task. The only pain point is using an async FnMut function so I had to do a bit of gymnastics around it.

Nothing is final, just looking for comment at this time.

Usage :

    let config = GraphClientConfiguration::new()
        .retries(true)
        .retries_config(RetriesConfig {
            initial_interval: Duration::from_millis(500),
            randomization_factor: 0.0,
            multiplier: 1.5,
            max_interval: Duration::from_secs(450),
            max_elapsed_time: Some(Duration::from_millis(450)),
        })
        .access_token(ACCESS_TOKEN);

    let client = Graph::from(config);

    for _i in 0..100 {
        let response = client
            .reports()
            .get_email_activity_counts_by_period("D180")
            .send()
            .await.unwrap();

        println!("Got response {}", response.status());
    }

It's not necessary to provide a RetriesConfig, there is already a default one with values that seems okay enough for GraphAPI imo.

I guess I would say can you give me more information on how it would be used? Is the ranomization factor based off of the backoff crate? I didn't go too far into looking at the backoff crate about its use so I apologize. Whats confusing about that is typically exponential backoff increases the rate at which the backoff happens. And this is typically done at a 2x factor. For instance first retry happens at 2 then 4 then 8 then 16. This is how its done in other graph sdks as well.

For retry functionality we should allow providing policies such as ExponentialBackoffPolicy where the user can set certain fields that change how the retry is done whereas fields like max retry time or max retries allowed will be set internally. The user can still set those but only if they don't go over the given limit. And that limit would be based off of other graph sdks to be consistent.

The unoffical Rust Azure sdk is similar but with the possibility of adding a random delay: https://github.com/Azure/azure-sdk-for-rust/blob/main/sdk/core/src/policies/retry_policies/exponential_retry.rs - I wonder the randomization factor is for the same thing?

[DevLazio]

I guess I would say can you give me more information on how it would be used? Is the ranomization factor based off of the backoff crate? I didn't go too far into looking at the backoff crate about its use so I apologize. Whats confusing about that is typically exponential backoff increases the rate at which the backoff happens. And this is typically done at a 2x factor. For instance first retry happens at 2 then 4 then 8 then 16. This is how its done in other graph sdks as well.

The five parameters are from the backoff crate. At first I was thinking that we could provide a simpler configuration then I realised that at some point we would get a ticket to add such and such parameter and "re-exported" them all.

For retry functionality we should allow providing policies such as ExponentialBackoffPolicy where the user can set certain fields that change how the retry is done whereas fields like max retry time or max retries allowed will be set internally. The user can still set those but only if they don't go over the given limit. And that limit would be based off of other graph sdks to be consistent.

There is an open issue about adding a max retry count, but currently the backoff crate does not provide that functionnality. I guess we can compute it by modifying the time parameters, but it's not the same. If we do this we have to implement it ourselves.

The unoffical Rust Azure sdk is similar but with the possibility of adding a random delay: https://github.com/Azure/azure-sdk-for-rust/blob/main/sdk/core/src/policies/retry_policies/exponential_retry.rs - I wonder the randomization factor is for the same thing?
As I understand it, pretty much.

[sreeise]

@DevLazio Yeah probably the best thing to do here is to provide a configuration model, a wrapper around the backoff config, that has those fields:

• initial delay: cannot be greater than max delay and cannot be greater than the max delay that the crate sets by default
• max retries: can be set by the user as long as it doesnt go over the max retries that the crate sets by default
• max delay
• max time elapsed: can be set by the user as long as it doesnt go over the max time ther crate sets by default

For the max retries, that is something that will have to be done on our side. A counter of some sort.

The configuration model will need to be done in such as a way as to allow other types of retry configuration models such as a throttle retry configuration. This can be done by providing a mechamism to allow multiple types or configurations such as an enum where each enum field is one of the configurations or by using a trait object and the GraphConfiguration storing a Box<dyn trait> for those configurations.

[DevLazio]

struct ThrottleConfig;

#[derive(Clone, Debug)]
pub enum Retries {
    NoRetry,
    ExponentialBackoff(ExponentialBackoffConfig),
    Throttle(ThrottleConfig),
}

#[derive(Clone)]
struct ClientConfiguration {
    access_token: Option<String>,
    headers: HeaderMap,
    referer: bool,
    timeout: Option<Duration>,
    retries: Retries,
    connect_timeout: Option<Duration>,
    connection_verbose: bool,
    https_only: bool,
    min_tls_version: Version,
}

Something like that ?

For the max retries I can't find a way to implement an attempt counter, neither inside the closure nor outside. I think the only way is to implement it inside the ExponentialBackoff struct (see ihrwein/backoff#60).

I'm curious, what would the Throttle retry do ?

[sreeise]

@DevLazio I havent forgotten about this. Sorry for the delay. Just been super busy between work and other things. I have a design that I think will work for this and I can give a high level overview and we can discuss. Just give me a few more days and i'll reply back with that design information.

[sreeise]

I'm curious, what would the Throttle retry do ?

The graph api throttles the amount of requests that can be sent to its servers based on a few factors but mainly too many requests being sent at one time. Here is a small excerpt from their docs:

When a throttling threshold is exceeded, Microsoft Graph limits any further requests from that client for a period of time. When throttling occurs, Microsoft Graph returns HTTP status code 429 (Too many requests), and the requests fail. A suggested wait time is returned in the response header of the failed request. Throttling behavior can depend on the type and number of requests. For example, if you have a high volume of requests, all requests types are throttled. Threshold limits vary based on the request type. Therefore, you could encounter a scenario where writes are throttled but reads are still permitted.

When requests are throttled, Microsoft sends the response with the 429 error code, which can be used to detect throttling, as well as a Retry-After header which provides the number of seconds to wait before retrying the request again. The throttle retry will wait those number of seconds and then retry the request. We can possibly add an option for doing these waits on another thread so its non-blocking as well but default would block.

You can read more about it here: https://learn.microsoft.com/en-us/graph/throttling#best-practices-to-handle-throttling

I also added a general summary in the issue description above of the two retry types that I think is a good starting place for getting the code in the library. Meaning as long as those two are met we should merge it in.

[sreeise]

@DevLazio

I will have some more information on this soon but in general the implementation should follow that of a http pipeline using policies. That can be a little confusing given its such a broad term but a simple description is that a pipeline stores a list of policies, which are things like retry policies or telementry policies or specific use case policies, and these policies are called in a specific order on each request.

So Policies are not specific to retrying requests but the basics are:

1. Policies accept a request as input
2. Policies can modify the request. Some policies can bail out of the pipeline such as when the maximum retries have been met.
3. When the last policy is reached, usually called a transport policy, this policy calls the request and returns the response.

A pipeline:

1. Stores a list of policies, both user defined and sdk defined but for now user defined will come at a later point.
2. These policies can be updated and/or removed except where sdk defined policies are required (transport policy)
3. Ensures proper order of policies being executed on each request.

Retry policies will be included in this list of policies. So this means the retry policy will modify the request in such a way that the transport policy knows how to handle any retries (there are different implementations and ways of doing things but I am providing a simple example). The request that I am talking about may not be just the http Request object but will have other fields, and will have a field for the current actual http request that is required.

I will provide a general impl of this and the retry policies soon. It won't be too in depth but it will provide enough of it to understand what I have mentioned above or at least make it more clear.

[DevLazio]

I like your pipeline approach. Might be especially usefull when trying to follow the Microsoft RFC for retries and the user's needs.

The graph api throttles the amount of requests that can be sent to its servers based on a few factors but mainly too many requests being sent at one time.

I thought you meant another thing. The way I started experimenting with it, the "429 Too Many requests" and their Retry-After header has priority over the user-defined exponential backoff settings, which resemble your pipeline model. The exponential backoff was used when their was no Retry After header or another error append.
Having the Enum be {None, ExponentialBackoff, Throttle} feels weird to me because we kinda have to abide by GraphAPI limitations, (I discovered that for endpoints you can continue to try before the Retry-After period is up and will a bit of luck you can get a successfull response if your request reach another instance or something) so ExponentialBackoff should include Throttle.
That's why I think your pipeline model is quite elegant.

We must be carrefull while implementing this pipeline though, because each endpoint has his own throttle limits, so the client must not delay a request because a request to another endpoint got a 429 response.

[sreeise]

Ive been quite busy so its taking me longer to get back to you. So I wanted to provide a quick example that follows what other msgraph sdks do as far as a really simple example can.

This is a small example of what typical sdks do but not a thorough or exactly correct one by any means. Note how the policies are taking in a list of policies and calling the next in the list of policies.

Not sure how much youve looked at http pipelines when it comes to azure or graph sdks but they typically take multiple types of policies and its basicaly a modification of the request and then calls the next policy to put it simply.

use http::Request;
use serde_json::Value;
use std::error::Error;
use std::sync::Arc;

pub struct RequestContext {
    // ... request context fields
}
// request context impl somewhere

// Just here as an example. The actual struct/impl would be different.
pub struct SomePolicyResult;

pub trait HttpPipelinePolicy {
    // Modify the request.
    fn process_async(
        &self,
        context: &RequestContext,
        request: &mut http::Request<Value>,
        pipeline: &[Arc<dyn HttpPipelinePolicy>],
    ) -> Result<SomePolicyResult, Box<dyn std::error::Error>>;

    fn process_next_async(
        &self,
        context: &RequestContext,
        request: &mut http::Request<Value>,
        pipeline: &[Arc<dyn HttpPipelinePolicy>],
    ) -> Result<SomePolicyResult, Box<dyn std::error::Error>> {
        pipeline[0].process_async(context, request, &pipeline[1..])
    }
}

// Example only. Not exact at all.
pub struct ExponentialBackoffRetryPolicy {
    // ... retry fields
    pub min_retries: u32,
}

impl HttpPipelinePolicy for ExponentialBackoffPolicy {
    fn process_async(
        &self,
        context: &RequestContext,
        request: &mut Request<Value>,
        pipeline: &[Arc<dyn HttpPipelinePolicy>],
    ) -> Result<SomePolicyResult, Box<dyn Error>> {
        // modify request...

        Ok(SomePolicyResult)
    }
}

pub struct ThrottleRetryPolicy {
    // ... impl
}

impl HttpPipelinePolicy for ThrottlePolicy {
    fn process_async(
        &self,
        context: &RequestContext,
        request: &mut Request<Value>,
        pipeline: &[Arc<dyn HttpPipelinePolicy>],
    ) -> Result<SomePolicyResult, Box<dyn Error>> {
        // modify request...

        Ok(SomePolicyResult)
    }
}