Exceeding actor limit kills your application

[mlh758]

We hit the default actor limit of 100 during a feature rollout in our application. We also had a flipper.rb registering some things and a memoize:

Rails.application.configure do
  config.flipper.memoize = ->(request) { !request.path.start_with?("/assets") }
end

Because of that memoize, and because Flipper throws when you hit that 100 feature limit, our health checks started failing since those came through on an http endpoint. This caused our load balancer to move all app instances out of rotation effectively bringing down the whole app.

We have monitoring, realized what was going on, and immediately removed one of the actors we added in the Flipper cloud UI. However, the app couldn't start to sync so it didn't recover automatically. We have another deployment that is just used to access a production console that doesn't boot the same way so I got into that and tried to invoke Flipper.sync. However, that also failed with the error about having more than 100 actors so I couldn't sync it away even after removing an actor.

We removed a row from the the feature flag table in the DB manually and then tried to sync and we still got the error. There are definitely only 99 actors now:

We then had to hotfix an increase to the limit.

There has to be a better way to enforce that limit than causing all invocations of Flipper to fail, even the ones that you might use to get back under the limit.

Flipper 1.3.4
Flipper ActiveRecord 1.3.4

[jnunemaker]

Oh no. That is quite the series of events. Sorry to hear this happened. The goal is always to help availability not to hurt it. I’ll see what I can come up with. For starters I think you should be able to set that limit on the cloud side per organization or project or environment. Then you could remove it from your local adapter. Or the local adapter could infer it from cloud. Really glad you brought this up. Well harden and make sure it doesn’t happen again. Short term can you increase your local adapter limit or remove it?

…

On Thu, Oct 30, 2025 at 10:49 AM Michael Harris ***@***.***> wrote: *mlh758* created an issue (flippercloud/flipper#951) <#951> We hit the default actor limit of 100 during a feature rollout in our application. We also had a flipper.rb registering some things and a memoize: Rails.application.configure do config.flipper.memoize = ->(request) { !request.path.start_with?("/assets") } end Because of that memoize, and because Flipper throws when you hit that 100 feature limit, our health checks started failing since those came through on an http endpoint. This caused our load balancer to move all app instances out of rotation effectively bringing down the whole app. We have monitoring, realized what was going on, and immediately removed one of the actors we added in the Flipper cloud UI. However, the app couldn't start to sync so it didn't recover automatically. We have another deployment that is just used to access a production console that doesn't boot the same way so I got into that and tried to invoke Flipper.sync. However, that also failed with the error about having more than 100 actors so I couldn't sync it away even after removing an actor. We removed a row from the the feature flag table in the DB manually and then tried to sync and we still got the error. There are definitely only 99 actors now: image.png (view on web) <https://github.com/user-attachments/assets/348ce52a-1a01-4c3f-88bc-e2299b1a24fa> We then had to hotfix an increase to the limit. There has to be a better way to enforce that limit than causing all invocations of Flipper to fail, even the ones that you might use to get back under the limit. — Reply to this email directly, view it on GitHub <#951>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAAB2YQPXT3344DAG4DTQ332IQRLAVCNFSM6AAAAACKVSPVKKVHI2DSMVQWIX3LMV43ASLTON2WKOZTGU3TCMJWGI3DKMI> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[mlh758]

We then had to hotfix an increase to the limit.

Yeah we deployed a change to increase the limit. We were just trying to use other methods first since it takes about 10-15 minutes for us to run CI, deploy, and for the new containers to come up.

I didn't know we could change it on the cloud side, although I'm a little skeptical that would have worked since we couldn't use the sync method to get updates from the cloud side once the limit was breached.

[mlh758]

Between this and the billing changes we've opted to remove Flipper entirely.

[jnunemaker]

@mlh758 sorry to hear this! What billing changes?

We changed pricing to be cheaper for the majority of customers, but didn't force anyone to change their plan (if the changes increased your pricing).

The only account I could find for you was on a free account. Is that correct? Did your pricing version change somehow and that affect you? I'm happy to move you back to an older pricing version if needed.

Regardless, I'm still going to fix this so I'm going to leave it open. I haven't yet as its relatively complicated. I need to make the local flipper respect the actor limit in cloud. This will require changes on the client (gem) and the server (cloud).

Again, sorry for the pain you felt. I'm happy to go above and beyond to repair however I can.

[zwygo]

We just ran into this issue, we had added over 100 actors to a toggle in Flipper Cloud, realized it was not working, and removed some to get under 100. Subsequent changes in Flipper Cloud were not properly syncing to our application, and attempting to get it back into a good state with Flipper.sync failed with the Actor limit of 100 exceeded error.

I temporarily upped the limit with Flipper.adapter.local.instance_variable_set(:@limit, 150), after which Flipper.sync worked.

There was no indication Flipper Cloud was not syncing to our app other than experiencing features not working properly and manually checking toggle evaluations with Flipper.enabled?.

[mlh758]

We were going to have to leave fee tier due to having a custom environment for our end to end tests. A reasonable enough requirement on your part. Another product we use has feature flagging as well so we decided to move the functionality over rather than change plans after this incident.