MeshRetry do not allow retry GRPC connection errors

[tony-bondarenko]

Description

All MeshRetry retryOn options available for grpc protocol (Canceled, DeadlineExceeded, Internal, ResourceExhausted, Unavailable) only trigger retry in case http headers are returned from the grpc service.
If request failure is related to connection issues (reset, timeout), no retry is triggered.

Such options are available for http protocol: Reset, ConnectFailure. Also 5xx is a combination of these.
But http MeshRetry policy is filtered out if service port has appProtocol = grpc and is not being applied.

[Icarus9913]

I did a test for that and got your point, will talk to the team about the design.
Thanks for raising it.

[Icarus9913]

In 2.10.* version, the MeshRetry policy has a strong isolation about the retryOn conditions between http and grpc protocols on each 'Service'.

For GRPC, we can only use retryOn: Canceled, DeadlineExceeded, Internal, ResourceExhausted, Unavailable

For example, here's my MeshRetry policy which is set the retryOn property explicitly and k8s service configuration:

apiVersion: kuma.io/v1alpha1
kind: MeshRetry
metadata:
  annotations:
    kuma.io/display-name: mesh-retry-all-default
  name: mesh-retry-all-default
  namespace: kuma-system
spec:
  targetRef:
    kind: Mesh
  to:
  - default:
      grpc:
        backOff:
          baseInterval: 25ms
          maxInterval: 250ms
        numRetries: 5
        perTryTimeout: 16s
        retryOn:                                                       // here!
        - Canceled
        - DeadlineExceeded
        - Internal
      http:
        backOff:
          baseInterval: 25ms
          maxInterval: 250ms
        numRetries: 5
        perTryTimeout: 16s
        retryOn:                                                     // here!
        - Reset
        - ConnectFailure
      tcp:
        maxConnectAttempt: 5
    targetRef:
      kind: Mesh
---------------------------
apiVersion: v1
kind: Service
spec:
...
  ports:
  - appProtocol: grpc                                      // here!
    name: name-of-service-port
    port: 80
    protocol: TCP
    targetPort: http-web-svc

Then I'll get my xds configurations:

 "route": {
    "cluster": "nginx-service_kuma-demo_svc_80",
    "timeout": "15s",
    "retry_policy": {
      "retry_on": "canceled,deadline-exceeded,internal",                               // here!
      "num_retries": 5,
      "per_try_timeout": "16s",
      "retry_back_off": {
        "base_interval": "0.025s",
        "max_interval": "0.250s"
      }
    },
    "idle_timeout": "1800s"
  },

You can see the above xds configuration only has grpc retryOn without basic HTTP reset and connection_failure capabilities. But refer to the envoy documentation config.route.v3.RetryPolicy you can see there's one sentence:

retry_on: (string) Specifies the conditions under which retry takes place. These are the same conditions documented for x-envoy-retry-on and x-envoy-retry-grpc-on.

Actually, the retry_on property is shared by HTTP and GRPC, no?
Another one is from our codes, we set the default GPRC retryOn property with cancelled,connect-failure,gateway-error,refused-stream,reset,resource-exhausted,unavailable, which proves we use the HTTP retryOn confs for GRPC by default.

So, my opinion is to set it free for users to include BASIC/ADDITIONAL HTTP retryOn confs into the GRPC's.

[lobkovilya]

Triage: when configuring grpc in MeshRetry policy we should allow setting retryOn values for http as well. Should be possible to create a policy like

apiVersion: kuma.io/v1alpha1
kind: MeshRetry
metadata:
  annotations:
    kuma.io/display-name: mesh-retry-all-default
  name: mesh-retry-all-default
  namespace: kuma-system
spec:
  targetRef:
    kind: Mesh
  to:
  - default:
      grpc:
        backOff:
          baseInterval: 25ms
          maxInterval: 250ms
        numRetries: 5
        perTryTimeout: 16s
        retryOn:                                                       
        - Canceled
        - DeadlineExceeded
        - Internal
        - Reset
        - ConnectFailure