[discourse] Multiple Flipt instances are not working properly

[markphelps]

Originally posted by weibin on Discourse

Hi team!

We are using Flipt in our project and it works great, but we recently ran into an issue.

When multiple Flipt instances are deployed in Kubernetes, only one instance can correctly evaluate the variant. The other instances will return

{
    "match": true,
    "segmentKeys": [
        "everyone"
    ],
    "reason": "MATCH_EVALUATION_REASON",
    "variantKey": "",
    "variantAttachment": "",
    "requestId": "53b2c74b-71ed-43a7-b517-98703126513f",
    "requestDurationMillis": 1.922781,
    "timestamp": "2024-11-07T02:29:32.462681869Z",
    "flagKey": "mobile_core_config"
}

The variantKey and variantAttachment are empty.

We are deploying Flipt on Kubernetes using a Helm chart, and we have enabled autoscaling.

Additionally, we have set up PostgreSQL and Redis for Flipt.

We are also configuring Git as the source for the feature flag state.

The Flipt version is v1.51.0

This is the log that can evaluate the variant, and I noticed that there is a matched distribution

Snipaste_2024-11-07_10-31-192224×258 86.4 KB

This is the log that can not evaluate the variant, with no distributions for rule

Snipaste_2024-11-07_10-33-472350×472 151 KB

When we scale it down to one instance, it can correctly evaluate the variant.

This is the rule and variants for the flag

image1576×745 51.3 KB

image1572×227 11.9 KB

I would appreciate your assistance. Thank you!

[dosubot[bot]]

This issue is likely caused by synchronization delays or cache inconsistencies between your Flipt instances. Flipt uses a versioned caching mechanism with Redis to keep state in sync across replicas. When a flag, rule, or variant changes, the namespace version is incremented, invalidating the cache for that namespace and forcing a refresh on the next read. However, if one instance updates the state and others haven't yet refreshed their cache (due to TTL or propagation delay), you can see symptoms like "no distributions for rule" and empty variant results on some pods, while others work as expected. In the worst case, this staleness can persist for up to the cache TTL, but it's usually brief and self-healing once the cache expires or is invalidated by a write operation. PostgreSQL remains the source of truth, so eventual consistency is guaranteed as long as all instances share the same Redis and database backends [source].

To minimize these issues, make sure:

• All Flipt instances use the same Redis and PostgreSQL backends (never SQLite in multi-replica setups) [source].
• Redis cache TTL is set appropriately (default is 60s; lowering it can reduce staleness but increases load) [source].
• There are no network partitions or connectivity issues between pods and Redis/PostgreSQL.

If you continue to see persistent cache inconsistencies, check your Redis logs for errors or missed cache invalidations, and verify that all Flipt pods are using the same configuration and environment variables.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}   

[markphelps]

Reply from erka on Discourse:

The original discussion and all details could be found in Discord thread