Infinite retry to establish connection to FlagSyncService in Flagd golang provider

[khan-rasul]

Golang Flagd provider tries to establish connection to FlagSyncService in an infinite loop. The retry at the this point happens on any error. This is unwanted as the connection can fail due to permission issues and shouldn't be retired infinitely.

[khan-rasul]

Findings

What happens?

1. Stream connection is established successfully with a 4 second delay (code)
2. Stream fails to receive first message due to permission issues (code). That's how streaming RPC work by design, permissions are validated on first message and not (at least not all permissions) during establishing connection.
3. As stream closes, the process is retried again (code). Thus retrying indefinitely.

In this case, one instance running Flagd can generate 15 requests per minute.

Logs:

{"level":"warn","ts":"2025-10-21T15:22:36Z","msg":"connection re-establishment attempt in-progress for grpc target: saasconfig.googleapis.com:443"}
{"level":"info","ts":"2025-10-21T15:22:36Z","msg":"connection re-established with grpc target: saasconfig.googleapis.com:443"}
{"level":"warn","ts":"2025-10-21T15:22:36Z","msg":"error with stream listener: error receiving payload from stream: rpc error: code = PermissionDenied desc = Permission 'saasconfig.featureFlagsConfigs.get' denied on resource <redacted> (or it may not exist)."}
{"level":"warn","ts":"2025-10-21T15:22:40Z","msg":"connection re-establishment attempt in-progress for grpc target: saasconfig.googleapis.com:443"}
{"level":"info","ts":"2025-10-21T15:22:40Z","msg":"connection re-established with grpc target: saasconfig.googleapis.com:443"}
{"level":"warn","ts":"2025-10-21T15:22:40Z","msg":"error with stream listener: error receiving payload from stream: rpc error: code = PermissionDenied desc = Permission 'saasconfig.featureFlagsConfigs.get' denied on resource <redacted> (or it may not exist)."}
<redacted>
{"level":"warn","ts":"2025-10-21T15:22:44Z","msg":"connection re-establishment attempt in-progress for grpc target: saasconfig.googleapis.com:443"}
{"level":"info","ts":"2025-10-21T15:22:44Z","msg":"connection re-established with grpc target: saasconfig.googleapis.com:443"}
{"level":"warn","ts":"2025-10-21T15:22:44Z","msg":"error with stream listener: error receiving payload from stream: rpc error: code = PermissionDenied desc = Permission 'saasconfig.featureFlagsConfigs.get' denied on resource <redacted> (or it may not exist)."}
<redacted>
{"level":"warn","ts":"2025-10-21T15:22:48Z","msg":"connection re-establishment attempt in-progress for grpc target: saasconfig.googleapis.com:443"}
{"level":"info","ts":"2025-10-21T15:22:48Z","msg":"connection re-established with grpc target: saasconfig.googleapis.com:443"}
{"level":"warn","ts":"2025-10-21T15:22:48Z","msg":"error with stream listener: error receiving payload from stream: rpc error: code = PermissionDenied desc = Permission 'saasconfig.featureFlagsConfigs.get' denied on resource <redacted> (or it may not exist)."}
{"level":"warn","ts":"2025-10-21T15:22:52Z","msg":"connection re-establishment attempt in-progress for grpc target: saasconfig.googleapis.com:443"}
{"level":"info","ts":"2025-10-21T15:22:52Z","msg":"connection re-established with grpc target: saasconfig.googleapis.com:443"}
{"level":"warn","ts":"2025-10-21T15:22:52Z","msg":"error with stream listener: error receiving payload from stream: rpc error: code = PermissionDenied desc = Permission 'saasconfig.featureFlagsConfigs.get' denied on resource <redacted> (or it may not exist)."}
<redacted>
{"level":"warn","ts":"2025-10-21T15:22:56Z","msg":"connection re-establishment attempt in-progress for grpc target: saasconfig.googleapis.com:443"}
{"level":"info","ts":"2025-10-21T15:22:56Z","msg":"connection re-established with grpc target: saasconfig.googleapis.com:443"}
{"level":"warn","ts":"2025-10-21T15:22:56Z","msg":"error with stream listener: error receiving payload from stream: rpc error: code = PermissionDenied desc = Permission 'saasconfig.featureFlagsConfigs.get' denied on resource <redacted> (or it may not exist)."}
<redacted>
{"level":"warn","ts":"2025-10-21T15:23:00Z","msg":"connection re-establishment attempt in-progress for grpc target: saasconfig.googleapis.com:443"}
{"level":"info","ts":"2025-10-21T15:23:00Z","msg":"connection re-established with grpc target: saasconfig.googleapis.com:443"}
{"level":"warn","ts":"2025-10-21T15:23:00Z","msg":"error with stream listener: error receiving payload from stream: rpc error: code = PermissionDenied desc = Permission 'saasconfig.featureFlagsConfigs.get' denied on resource <redacted> (or it may not exist)."}
{"level":"warn","ts":"2025-10-21T15:23:04Z","msg":"connection re-establishment attempt in-progress for grpc target: saasconfig.googleapis.com:443"}
{"level":"info","ts":"2025-10-21T15:23:04Z","msg":"connection re-established with grpc target: saasconfig.googleapis.com:443"}
{"level":"warn","ts":"2025-10-21T15:23:04Z","msg":"error with stream listener: error receiving payload from stream: rpc error: code = PermissionDenied desc = Permission 'saasconfig.featureFlagsConfigs.get' denied on resource <redacted> (or it may not exist)."}
<redacted>
{"level":"warn","ts":"2025-10-21T15:23:08Z","msg":"connection re-establishment attempt in-progress for grpc target: saasconfig.googleapis.com:443"}
{"level":"info","ts":"2025-10-21T15:23:08Z","msg":"connection re-established with grpc target: saasconfig.googleapis.com:443"}
{"level":"warn","ts":"2025-10-21T15:23:08Z","msg":"error with stream listener: error receiving payload from stream: rpc error: code = PermissionDenied desc = Permission 'saasconfig.featureFlagsConfigs.get' denied on resource <redacted> (or it may not exist)."}

[beeme1mr]

Yeah, the provider shouldn't reconnect indefinitely if we get a permission denied. It should also return the error code PROVIDER_FATAL.

[alexandraoberaigner]

I'd like to work on this :)

[aepfli]

do you think that #744 might resolve this. i removed our custom backoff logic, and also added grace period etc. this might solve your issue - reviews highly appreciated

[tangenti]

do you think that #744 might resolve this. i removed our custom backoff logic, and also added grace period etc. this might solve your issue - reviews highly appreciated

Is the DialOptions for gRPC connections also applied to stream establishment?

Also, I agree with what @beeme1mr brought here. PERMISSION_DENIED is an irrecoverable error state, and retries are not super helpful.

[alexandraoberaigner]

do you think that #744 might resolve this. i removed our custom backoff logic, and also added grace period etc. this might solve your issue - reviews highly appreciated

I tried this with a local flagd & envoy filter to return the permission denied error. With just the changes from #744 it will not be sufficient to exclude the PERMISSION_DENIED from the retryPolicy.

Instead I opened a PR (#783) with the logic that would fix the problem. Let me know if we should go with this, then I guess it would make sense to add proper tests, its only tested manually so far.

Not a go expert here, so please also let me know in the PR what to improve.