RFC: Upstream a Tier Based Rate Limiter.

[michaelfeil]

I would like to have a per model, frontend set-able.

I require to expose this function to python.
We would prefer this metric would be a global. For now, setting this in every frontend.
We would like to use the routers information on requests in flight, so exposing it in python gives the user a lot of freedom. We are running the code from the frontend (our frontend is in python).

/// Set the global rate limit level used by the HTTP rate limiter.
///
/// # Arguments
/// * `limit` - Usage fraction limit (non-negative float).
#[pyfunction]
#[pyo3(signature = (limit))]
pub fn set_rate_limit_level(limit: f64) -> PyResult<()> {
    b10_rate_limiter::set_current_rate_limit_level(limit);
    Ok(())
}

// returns a axum response with 429 status code if the request should be rate limited, otherwise returns None
pub fn check_rate_limit(headers: &HeaderMap) -> Option<(String, f64)> {
    // upstream sets a header "X-Baseten-Usage-Fraction" with a float
    // expected range is [0, 1] but can exceed 1 (e.g., 5.0 or 100s/1000s)
    let request_limit: f64 = headers
        .get(HEADER_NAME_RATE_LIMIT_REQUEST)
        .and_then(|h| h.to_str().ok())
        .and_then(|s| s.parse().ok())
        .unwrap_or(0.0);

    // get the current global rate limit level
    let current_level = get_current_rate_limit_level();
    if request_level > current_level {
        // return a 429 response with a message about the rate limit
        Some((
            format!(
                "Rate limit exceeded: usage fraction {} percent exceeds current level {} percent",
                request_level * 100.0,
                current_level * 100.0
            ),
            request_level,
        ))
    } else {
        None
    }
}

I want to be able to make soft-rate limits this way, e.g. allowing 100% traffic under normal hours, or 200% during off-peak hours. Idea is e.g. that you can serve two user groups (free and paid) and only run free tier in some hours.

[itay]

@michaelfeil can you share a bit more on the conceptual requirements, for example:

1. What information are the rate limits using (tokens in flight, user identity, requests in flight, etc)?
2. What are example tiers?

[michaelfeil]

Yes, can but privatley. Its important to me that this is extensible in python.