The experimental data still exists after the experiment is closed

[Simplegood]

Hi,

We are currently using ‌GrowthBook iOS version 1.0.71‌. We noticed that ‌an experiment turned off in early May‌ is still generating data by the end of May, and the data volume hasn’t decreased gradually in iOS. Below is our relevant code snippet:

self.init( growthbook: GrowthBookBuilder( apiHost: “https://cdn.growthbook.io/”, clientKey: “xxx”, attributes: [“xxx”: “xxx”], trackingCallback: { experiment, result in let params = [ “experiment_Id”:experiment.key, “variation_Id”: result.key ] log(“log_event”, params) } ).initializer() )

The data is not decreasing, instead, it's showing an upward trend. We cannot reproduce this issue in offline environments, and the experiment isn't visible in the configurations distributed via cdn.growthbook.io.

[vazarkevych]

Hi @Simplegood . Please let us check that and we will write to you.

[Simplegood]

Hi @vazarkevych
This issue is having a significant impact on us. Have you found anything out yet?‌

[vazarkevych]

Hi, @Simplegood . We are working on reproducing that. We will write as soon as we will reproduce that issue and start working on fixing.

[vazarkevych]

Hi @Simplegood, thanks for your patience.
Based on our investigation so far, we suspect that your app might still be using cached features, and that they haven't been updated properly after the experiment was turned off.
In my local setup (GrowthBook iOS v1.0.71), I noticed that the features only update if I explicitly call refreshCache() or enable backgroundSync = true. You could try either of these as an immediate workaround.
By design, features should refresh in the background after they are accessed, but this behavior doesn’t seem to be working reliably in this version. That’s why I’ve submitted this pull request implementing a stale-while-revalidate strategy that updates the features in the background when they are stale — as described in the GrowthBook docs.
I'm trying to better understand the root cause of the issue, so could you let me know if the feature state in your app updates correctly when you toggle a feature on or off from the GrowthBook dashboard? That would really help us narrow down the possible causes.

[Simplegood]

Hi, @vazarkevych
We tested this and found that the 'gjh-xxx-p2' experiment delivered via CDN isn't returning in the trackingCallback, resulting in data inconsistency

Our code:

 trackingCallback: { experiment, result in
       let analyticsParams = [
          "experimentId": experiment.key,
          "variationId": result.key
       ]
                        
       debugLog("Growthbook test --- \(String(describing: analyticsParams))")
}

Console Result:

 ["Growthbook test --- [\"experimentId\": \"G-9\", \"variationId\": \"1\"]"]
 ["Growthbook test --- [\"variationId\": \"3\", \"experimentId\": \"G-2\"]"]
 ["Growthbook test --- [\"experimentId\": \"srp\", \"variationId\": \"2\"]"]
 ["Growthbook test --- [\"experimentId\": \"gjh-xxx-p1\", \"variationId\": \"1\"]"]

Response of https://cdn.growthbook.io/api/features/xxxx

[vazarkevych]

Hi @Simplegood,

Thanks a lot for your patience and for providing additional details.

After reviewing the behavior and your findings, we believe this is the expected behavior based on how GrowthBook currently works.

To clarify:

• The reason you’re still receiving feature data (even though the experiment has been turned off) is because the associated feature flag is still active.
• GrowthBook always fetches all active features from the CDN — this includes features linked to experiments that are turned off.
• The trackingCallback is only triggered when a user is actively placed into an experiment, as described in the docs here. Since the experiment has been turned off, users are no longer being placed into it, and that’s why the callback is not invoked.

[Simplegood]

Hi, @vazarkevych
Thank you for your explaining. I still have some follow-up questions.

1. Regarding「The reason you’re still receiving feature data (even though the experiment has been turned off) is because the associated feature flag is still active.」

• If GrowthBook fetches all feature flags while the experiment is turned off, should all the resulting keys consistently return 0? Or is it possible to get a mix of both 0 and 1 values in this scenario?
  Currently, with the experiment turned off, we're seeing both 1 and 0 results coexisting, and the distribution between these values appears uneven.
  

2. Regarding「GrowthBook always fetches all active features from the CDN — this includes features linked to experiments that are turned off.」

• Does that mean we have to delete the feature flag to prevent the trackingCallback from being triggered? Also, we'd like to track data in the production environment only when the experiment switch is turned on, and avoid tracking when it's turned off. Which callback should we use for that purpose?(We require the feature flag for testing in our test environment, so deletion isn’t an option.)

[vazarkevych]

Hi @Simplegood,

Thank you for the follow-up — happy to clarify both points:

---

1. Why are we still seeing both 0 and 1 values after turning off the experiment?

In GrowthBook, there are several types of rules that can control a feature’s behavior: Force, Percentage Rollout, and Experimentation.

If an experiment rule is disabled (i.e., turned off in the UI), then the experiment logic is no longer applied — users should receive the default value for the feature (e.g., consistently 0 if that’s the configured default).

However, if your feature flag still includes other types of rules — such as a Percentage Rollout — then users may continue to receive a mix of values, like both 0 and 1. A percentage rollout assigns values based on a random hash of a user attribute (e.g., userId) and operates independently of the experiment being on or off.

So in your case, if the experiment is turned off but you're still seeing mixed values with uneven distribution, it's very likely that a rollout rule is still active on the feature.

To confirm this, you can:

• Review the rules list for the feature in the GrowthBook UI or via the API.
• Make sure there are no active Force or Rollout rules in addition to the (disabled) experiment.

Let me know if the experiment rule is the only one you're using — or feel free to share a screenshot of your feature rule configuration, and I’d be happy to take a closer look.

---

2. Will trackingCallback still fire if the experiment is turned off? Do we need to delete the feature?

No, you don’t need to delete the feature flag.

While GrowthBook does fetch all active features, including those linked to disabled experiments, the trackingCallback is only triggered when a user is assigned to a variation through an active experiment rule.

So:

• If the experiment is off, users are not placed into the experiment, and trackingCallback will not be called.
• Other rules (like Force or Rollout) can still assign different values to users — but they do not trigger experiment tracking.

“We'd like to track data in production only when the experiment switch is on.”

You're already using the correct callback — just continue using trackingCallback. GrowthBook takes care of this internally:

• It only triggers the callback when an active experiment rule matches and assigns a variation.
• If the experiment is disabled (or skipped due to targeting or namespace logic), no tracking is sent.

This way, you can keep the feature flag active for test environments, while ensuring that no production tracking occurs unless the experiment is actually running.

---

Let me know if you’d like help checking the configuration or if you have any other questions!

[Simplegood]

Hi, @vazarkevych
However, looking at the data, experiments with the same configuration that have been turned off on Web and Android aren't showing tracked data. Only the iOS ones are. Could you help me understand why the implementation logic differs here?
Would checking if the experiment's "isActive" flag is set to true help resolve this issue?

[vazarkevych]

Hi @Simplegood, thanks again!

I’m currently unable to reproduce this issue on my side — the trackingCallback doesn’t fire for disabled experiments in my setup.

To help us investigate further, could you please share screenshots of:

• The feature flag configuration (especially the rules tab)
• The experiment configuration
• And how the SDK is set up on Android and Web, where this issue doesn’t occur

This will help us understand the difference in behavior and reproduce the issue more accurately.

Thanks a lot!

[Simplegood]

Hi @vazarkevych
Thanks for your answer, here are screenshots of configurations and set up code:

1. The feature flag configuration (especially the rules tab)

2. The experiment configuration

3. Set up on Android

[vazarkevych]

Thanks for providing that! I’ll try to reproduce the issue and investigate whether there are any differences between the trackingCallback invocation rules on Android and iOS, to see if there’s a difference in implementation.

[vazarkevych]

In your screenshots, I don’t see whether the experiment is active or the full list of feature rules. Could you please share the complete setup of the feature and experiment?

Also, I would like to clarify — you earlier said that the trackingCallback wasn’t triggered. Has that changed, or did I misunderstand your problem?