fix(flagd): do not retry for certain status codes (#756) #799

alexandraoberaigner · 2025-11-13T09:15:47Z

This PR

This pull request introduces configurable retry and fatal error handling for the in-process gRPC sync provider in the flagd project. The main changes include adding new configuration options for retry backoff timing and fatal status codes, refactoring environment variable parsing, and updating service initialization and error handling logic.

Related Issues

Reopens the changes from fix(flagd): do not retry for certain status codes (#756) #783
Fixes Infinite retry to establish connection to FlagSyncService in Flagd golang provider #756

Changes

Retry and fatal error configuration (most important):

Added new configuration options (RetryBackoffMs, RetryBackoffMaxMs, FatalStatusCodes) to ProviderConfiguration, with environment variable support and helper functions for parsing integer values from environment variables. [1] [2] [3] [4] [5] [6]
Updated provider and service initialization to pass the new retry and fatal error configuration fields, enabling customization of retry timing and fatal error handling for sync streams. [1] [2] [3]

Refactoring and code quality:

Refactored environment variable parsing for integer values using a new helper function, simplifying and unifying logic for multiple configuration fields. [1] [2] [3]
Moved gRPC retry policy construction and fatal status code normalization to a new file grpc_config.go, making the code more modular and testable. [1] [2] [3] [4] [5]

Error handling improvements:

Enhanced sync error handling to detect fatal gRPC status codes and transition the provider to a fatal state, preventing endless retries on unrecoverable errors.
Updated test coverage for retry policy construction, fatal status code normalization, and camel-case conversion logic in grpc_config_test.go.

aepfli · 2025-11-14T08:52:04Z

As we are adding new config options, we should wait for open-feature/flagd-testbed#311 to be merged to ensure property names are in consistent for all the providers based on the docs.

aepfli · 2025-11-17T12:48:30Z

tests/flagd/testframework/utils.go

 		}
 		return reflect.ValueOf(longVal).Convert(fieldType)
+	case "StringList":
+		arrayVal := strings.Split(value, ",")


do we also need to trim here?

yes, good catch! will update this

guidobrei · 2025-11-21T09:32:43Z

providers/flagd/pkg/service/in_process/grpc_config.go

+func (g *Sync) initNonRetryableStatusCodesSet()  {
+	nonRetryableCodes = make(map[string]struct{})
+	for _, code := range g.FatalStatusCodes {
+		normalized := toCamelCase(code)


Why are the codes camelCase? In the retry policy up in L35 they are Upper case.

because grpc doesnt export a function to get the UPPER case representation it uses internally. However, I changed the implementation to use codes.Code for lookup instead of the string, making it cleaner & remove the need for the case conversion

guidobrei · 2025-11-21T09:37:10Z

providers/flagd/pkg/service/in_process/grpc_sync.go

 func (g *Sync) Sync(ctx context.Context, dataSync chan<- sync.DataSync) error {
 	g.Logger.Info("starting continuous flag synchronization")

+	time.Sleep(500 * time.Millisecond)


Is this a debug leftover?

guidobrei · 2025-11-21T09:46:12Z

providers/flagd/pkg/service/in_process/grpc_sync.go

+			}
+
+			// Backoff before retrying
+			time.Sleep(time.Duration(g.RetryBackOffMs) * time.Millisecond)


Suggested change

time.Sleep(time.Duration(g.RetryBackOffMs) * time.Millisecond)

time.Sleep(time.Duration(g.RetryBackOffMaxMs) * time.Millisecond)

This is a fix of a different problem. Can be moved to a separate PR. Not part of the non-retryable status codes.

is this part of the retry discussion? or are you suggesting to open a new one to discuss this?

As we are always mixing features and functionalities, i would also love to see this as two separate prs. it allows to focus on one implementation with its tests, and makes the changes clearer.

I'm reluctant to put this change in a separate PR since Todd's idea was to unify implementations (Java and Go) as feedback on the initial PR -> see comment here.
Based on this I applied @guidobrei 's suggestion. I noticed just now, it was actually a mistake I made due to different variable naming in Java.

I don't feel very strong about splitting this PR up, but @aepfli @guidobrei seem to disagree - I think this change might somehow also be causing the e2e test failure now, but not 100% sure about that - might be an option needs updating, so splitting this up might actually make things easier for you.

guidobrei · 2025-11-21T09:49:21Z

providers/flagd/pkg/provider.go

 			CustomSyncProviderUri:   provider.providerConfiguration.CustomSyncProviderUri,
 			GrpcDialOptionsOverride: provider.providerConfiguration.GrpcDialOptionsOverride,
 			RetryGracePeriod:        provider.providerConfiguration.RetryGracePeriod,
+			RetryBackOffMs:          provider.providerConfiguration.RetryBackoffMs,


Separate issue

alexandraoberaigner · 2025-11-26T08:45:25Z

providers/flagd/pkg/service/in_process/grpc_sync.go

-	// Mark as ready on first successful stream
-	g.initializer.Do(func() {
-		g.ready = true
-		g.Logger.Info("sync service is now ready")
-	})


Note: This moved to lines 275-279 to set ready = true only after the first stream cycle was successful not during the cycle.

Yes I think this was actually a small bug. 👍

aepfli · 2025-11-26T12:44:44Z

[Q] does someone have an idea whats wrong with the DCO?

Commit sha: 2eeebc3, Author: Alexandra Oberaigner, Committer: alexandraoberaigner; Expected "Alexandra Oberaigner [email protected]", but got "Alexandra Oberaigner [email protected]".

i am not sure, but worst case the how-to-fix section in https://github.com/open-feature/go-sdk-contrib/pull/799/checks?check_run_id=56445883813 can be helpful and should fix this ;)

aepfli

I think this pullrequest merges two features, fatalErrorCodes and backoff - to keep the changes distinct i suggest separating them into two different pull requests (does not mean they will not be released within one changeset) as they can be also delivered separately.

Furthermore we should rethink our sleeps as I think this is not good practice and there are alternatives, I also created an improvement for the java provider for this.

aepfli · 2025-11-26T13:16:30Z

providers/flagd/pkg/service/in_process/grpc_sync.go

+			}
+
+			// Backoff before retrying
+			time.Sleep(time.Duration(g.RetryBackOffMaxMs) * time.Millisecond)


I am not a big fan of our blocking sleeps, as they clearly have some disadvantages, should we maybe stick to a timer for this kind of logic? like

select { case <-time.After(time.Duration(g.RetryBackOffMaxMs) * time.Millisecond): // ... code here ... case <-ctx.Done(): return // Allows cancellation }

There's disadvantages of sleep, but I think right now this is better than nothing, because nothing == a tight loop in some cases that is a serious bug in some situations.

alexandraoberaigner · 2025-11-26T15:59:35Z

I think this pullrequest merges two features, fatalErrorCodes and backoff - to keep the changes distinct i suggest separating them into two different pull requests (does not mean they will not be released within one changeset) as they can be also delivered separately.

Pls consider my comment above :)

Furthermore we should rethink our sleeps as I think this is not good practice and there are alternatives, I also created an improvement for the java provider for this.

We can do an improvement issue for golang too -> this is just a bug fix / consistency PR

toddbaert

This implementation looks good to me, nice work overall. One thing I think we need in addition is that we should use the same FATAL codes in RPC mode - so I think you will have to add something similar to what you have done in pkg/service/rpc/service.go... do you agree? Since both modes have streams, I think it makes sense for both streams to use this rule.

With respect to @aepfli and @guidobrei 's comment about separating things... I can go either way, but you will need 1 more approval besides mine and I think it might make it easier for you to debug the e2e CI failure.

Signed-off-by: Alexandra Oberaigner <[email protected]>

aepfli

thank you, looks good to me, one little nit, but nothing blocking this pr from getting merged

providers/flagd/pkg/service/in_process/service.go

Signed-off-by: Alexandra Oberaigner <[email protected]>

alexandraoberaigner · 2025-11-28T11:56:38Z

[Q] does someone have an idea why the gherkin tests for SYNC_PORT still fail even though I excluded them with ~@sync-port

Signed-off-by: Alexandra Oberaigner <[email protected]>

Signed-off-by: Todd Baert <[email protected]>

toddbaert · 2025-11-28T17:20:02Z

[Q] does someone have an idea why the gherkin tests for SYNC_PORT still fail even though I excluded them with ~@sync-port

@alexandraoberaigner I pushed this.

github-actions bot assigned bacherfl, Kavindu-Dodan and toddbaert Nov 13, 2025

github-actions bot requested review from Kavindu-Dodan, bacherfl and toddbaert November 13, 2025 09:16

alexandraoberaigner force-pushed the fix/inifinite-loop-error branch 4 times, most recently from a2312a3 to a25f7be Compare November 14, 2025 07:39

alexandraoberaigner marked this pull request as ready for review November 14, 2025 07:45

alexandraoberaigner requested review from a team as code owners November 14, 2025 07:45

alexandraoberaigner mentioned this pull request Nov 14, 2025

Infinite retry to establish connection to FlagSyncService in Flagd golang provider #756

Open

aepfli reviewed Nov 17, 2025

View reviewed changes

guidobrei suggested changes Nov 21, 2025

View reviewed changes

alexandraoberaigner commented Nov 26, 2025

View reviewed changes

alexandraoberaigner force-pushed the fix/inifinite-loop-error branch from 3035374 to c8e8e10 Compare November 26, 2025 12:27

This comment was marked as resolved.

Sign in to view

alexandraoberaigner requested review from aepfli and guidobrei November 26, 2025 12:42

alexandraoberaigner force-pushed the fix/inifinite-loop-error branch from bf2e6f2 to 56800b2 Compare November 26, 2025 12:55

aepfli reviewed Nov 26, 2025

View reviewed changes

toddbaert reviewed Nov 26, 2025

View reviewed changes

fix: return fatal on certain error codes during first stream cycle

8ff44da

Signed-off-by: Alexandra Oberaigner <[email protected]>

alexandraoberaigner force-pushed the fix/inifinite-loop-error branch from 7ab3ce0 to 8ff44da Compare November 27, 2025 14:59

aepfli approved these changes Nov 28, 2025

View reviewed changes

providers/flagd/pkg/service/in_process/service.go Outdated Show resolved Hide resolved

update testbed, remove PR-split leftovers

f531ab0

Signed-off-by: Alexandra Oberaigner <[email protected]>

alexandraoberaigner force-pushed the fix/inifinite-loop-error branch from 12b55af to f531ab0 Compare November 28, 2025 11:41

alexandraoberaigner and others added 2 commits November 28, 2025 13:07

extend timeout for e2e tests

9730a43

Signed-off-by: Alexandra Oberaigner <[email protected]>

chore: Add sync-port exclusions for now

7d6fff6

Signed-off-by: Todd Baert <[email protected]>

toddbaert force-pushed the fix/inifinite-loop-error branch from 6e45d12 to 7d6fff6 Compare November 28, 2025 17:07

toddbaert self-requested a review November 28, 2025 17:19

Merge branch 'main' into fix/inifinite-loop-error

abd23e1

toddbaert approved these changes Nov 28, 2025

View reviewed changes

	time.Sleep(time.Duration(g.RetryBackOffMs) * time.Millisecond)
	time.Sleep(time.Duration(g.RetryBackOffMaxMs) * time.Millisecond)

fix(flagd): do not retry for certain status codes (#756) #799

Are you sure you want to change the base?

fix(flagd): do not retry for certain status codes (#756) #799

Uh oh!

Conversation

alexandraoberaigner commented Nov 13, 2025

This PR

Related Issues

Changes

Retry and fatal error configuration (most important):

Refactoring and code quality:

Error handling improvements:

Uh oh!

aepfli commented Nov 14, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alexandraoberaigner Nov 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alexandraoberaigner Nov 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alexandraoberaigner Nov 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alexandraoberaigner Nov 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This comment was marked as resolved.

aepfli commented Nov 26, 2025

Uh oh!

aepfli left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alexandraoberaigner commented Nov 26, 2025

Uh oh!

toddbaert left a comment

Choose a reason for hiding this comment

Uh oh!

aepfli left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

alexandraoberaigner commented Nov 28, 2025

Uh oh!

toddbaert commented Nov 28, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

alexandraoberaigner Nov 20, 2025 •

edited

Loading

alexandraoberaigner Nov 26, 2025 •

edited

Loading

alexandraoberaigner Nov 26, 2025 •

edited

Loading

alexandraoberaigner Nov 26, 2025 •

edited

Loading