How do I retry mid-body response read network failures?

[dolzenko]

It seems as if none of the libs from https://github.com/earthboundkid/requests/wiki#retries will help if the request fails somewhere in the middle of response read. Meanwhile the API itself suggests it might be possible. So I'm wondering if

1. my reading is correct
2. if there are better solutions than slapping generic retries on top of requests to get reliable retries of all kinds of errors at any stage of request send/response read.

❤️ for the great lib btw!

[earthboundkid]

Thanks. Yes, for a body read failure, you would need to go outside the transport and redo the whole request with a generic retry mechanism. Transports can only retry status errors like 429 Too Many Requests.

[bkane-msft]

This is how we're handling it "from the outside" to catch connection errors with github.com/cenkalti/backoff/v5 (code shortened for demonstration purposes)

// Validator that returns errors the backoff library uses
func ToRetryOrNotToRetry(resp *http.Response) error {

	switch {

	case resp.StatusCode == http.StatusOK:
		return nil

	case resp.StatusCode == http.StatusTooManyRequests:
		seconds, err := strconv.ParseInt(resp.Header.Get("Retry-After"), 10, 64)
		if err == nil {
			return backoff.RetryAfter(int(seconds))
		}
		return fmt.Errorf("429: will retry")

	case resp.StatusCode >= 500 && resp.StatusCode <= 599:
		return fmt.Errorf("5xx: will retry")

	// default to permanent errors
	default:
		return backoff.Permanent(fmt.Errorf("unknown status: won't retry"))
	}
}

// The retry function
func Retry(ctx context.Context, f func() error) error {
	operation := func() (struct{}, error) {
		err := f()
		return struct{}{}, err
	}
	_, err := backoff.Retry(
		ctx,
		operation,
		backoff.WithBackOff(backoff.NewExponentialBackoff()),
		backoff.WithMaxTries(3),
	)
	return err
}

// making the request
err := Retry(ctx, func() error {
	err := req.AddValidator(ToRetryOrNotToRetry).Fetch(ctx)
	if err != nil {
		slog.WarnContext(ctx, "error fetching data", "err", err.Error())
	}
	return err
})