HTTPRoute Retries

[sjberman]

As an NGF user,
I want to specify retries per request for my HTTPRoutes
So that my client applications can retry failed requests without returning immediately.

The Gateway API defines Retries here as an extended feature.

The closest thing NGINX has to retries is by using the proxy_next_upstream directive. However, by default this doesn't quite accomplish what is desired by the API. NGINX will not retry the same server, it will move on to the next server in the upstream, and try that one. So if a user has scaled their backend application to 3 pods, we can do a maximum of 3 retries. If only 1 pod exists, there won't be any retries.

There may be a way to make this work by duplicating servers within the upstream, to ensure we can retry as many times as the user has defined in their HTTPRoute.

For example, if only one pod exists, but the user defines 3 retries, the following may work:

server 10.0.0.1:3001;
server 10.0.0.1:3001;
server 10.0.0.1:3001;

The caveat to this is that nginx doesn't support backoff timers when retrying, which is part of the API spec. Backoff is an extended feature, so in theory we could support some of the spec, but not that field. Not an ideal UX, but we simply may not be able to support it.

There is an nginx-retry module written in Rust. However, it uses subrequests (as would an NJS module) which are not recommended for our use. Described here; additionally they don't play well with NAP modules.

Acceptance

• Investigate and understand what it takes to configure NGINX to support retries as defined by the Gateway API
• Implement the ability for users to configure HTTPRoute retries
• Enable any conformance tests that may exist that we can support for retries
• Create a user guide for configuring retries, and document any limitations