Rate Limit handling

[marcogruhl]

Hi,
this wrapper is the best i found for my twitter project. I played around and hit constantly rate limits. Sadly, this information is not provided.
I look at the code and have two ideas to implement this based on Twitter API Documentation

the twitter has to use GetAsync instead of GetStringAsync

1. Extend the Response objects

public class StreamInfo : XRate, IEquatable<StreamInfo>

the XRate Object would look like:

public class XRate
{
    public int Limit { get; set; }
    public int Remaining { get; set; }
    public int Reset { get; set; }
}

and than use the ParseArrayData Method like this:

! VERY UNOPTIMIZED !

private T[] ParseArrayData<T>(string json, HttpResponseHeaders headers = null) where T : XRate
{
    var answer = JsonSerializer.Deserialize<Answer<T[]>>(json, _jsonOptions);
    if (answer.Detail != null)
    {
        throw new TwitterException(answer.Detail);
    }
    if (answer.Data == null)
    {
        return Array.Empty<T>();
    }
    InternalIncludesParse(answer);
    
    if(headers != null)
    {
        if (headers.TryGetValues("x-rate-limit-limit", out var limit))
        {
            foreach (var data in answer.Data)
            {
                data.Limit = Convert.ToInt32(limit.FirstOrDefault());
            }
        }
        if (headers.TryGetValues("x-rate-limit-remaining", out var remaining))
        {
            foreach (var data in answer.Data)
            {
                data.Remaining = Convert.ToInt32(remaining.FirstOrDefault());
            }
        }
        if (headers.TryGetValues("x-rate-limit-reset", out var reset))
        {
            foreach (var data in answer.Data)
            {
                data.Reset = Convert.ToInt32(reset.FirstOrDefault());
            }
        }
    }

    return answer.Data;
}

2. Provide via EventHandler

TweetClient.cs:

public event EventHandler<XRate> XRateChanged;

public async Task<StreamInfo[]> GetInfoTweetStreamAsync()
{
    // var str = await _httpClient.GetStringAsync(_baseUrl + "tweets/search/stream/rules");
    var str = await _httpClient.GetAsync(_baseUrl + "tweets/search/stream/rules");

    XRateChanged.Invoke(this, new XRate(nameof(StreamInfo), str.Headers));

    return ParseArrayData<StreamInfo>(await str.Content.ReadAsStringAsync(), str.Headers);
}

XRate Class:

public class XRate
{
    public string Endpoint { get; set; }
    public int Limit { get; set; }
    public int Remaining { get; set; }
    public int Reset { get; set; }

    public XRate()
    {
        
    }

    public XRate(string endpoint, HttpResponseHeaders headers)
    {
        Endpoint = endpoint;

        if (headers != null)
        {
            if (headers.TryGetValues("x-rate-limit-limit", out var limit))
                Limit = Convert.ToInt32(limit.FirstOrDefault());
            
            if (headers.TryGetValues("x-rate-limit-remaining", out var remaining))
                Remaining = Convert.ToInt32(remaining.FirstOrDefault());

            if (headers.TryGetValues("x-rate-limit-reset", out var reset))
                Reset = Convert.ToInt32(reset.FirstOrDefault());
        }
    }
}

I will implement this if its fitting to your idea of this project

[Xwilarg]

I like the idea of extending the Response object but maybe headers shouldn't be parsed by default on every request?

What do you think of having a BaseObject class like that:

public class BaseObject
{
    internal BaseObject(HttpResponseHeaders headers)
        => _headers = headers;

    public XRate GetRateLimits()
    {
        // Implement header parsing as shown above
    }

    private HttpResponseHeaders _headers;
}

And every result object would inherit it so GetRateLimits() would be accessible from every response?

[Xwilarg]

Well it's in the headers anyway so I don't think JsonSerializer will help much

[marcogruhl]

Sorry, I dont understand. At least idea 1 and 2 have problems with DeleteTweetStreamAsync and its ParseData<object> return and NextTweetStreamAsync which returns not until the first tweet. Maybe idea 3 is the least breaking version.

[Xwilarg]

Uh yeah I forgot about that
I feel like event would not be the best solution but at the same time I don't have any other idea

[marcogruhl]

I implemented the EventHandler logic wich has no breaking changes: EventHandler branch
Can you look at it? Should i create a pull request? Do i have to extend the tests?
Sorry for those questions, it would be my first public pull request.

[Xwilarg]

Yup you can create a PR, that will also allow the current tests to run and see if everything still works well
It would indeed also be nice if you could add some tests to them, maybe like, do a request, get current rate limit and see if it changes when you do another one?

[marcogruhl]

Created the pull request #14
I also extended the test TestStreamProcess() because there are already a method which is called multiple times, so i can check the remaining rate limit decreasing like you suggested.