Add explicit eager_load! to rails preload.rb example

[danielwestendorf]

Types of Changes

Adds an Rails.application.eager_load! to the Rails preload.rb example.

• Performance improvement.

I was seeing wild memory consumption in a rails v7.2 app running on heroku with WEB_CONCURRENCY=3 running Performance-M dynos. Memory usage would steadily increase, often rapidly after boot. I added this change the the preload.rb file and am now seeing much more consistent memory usage and growth. This was the only change introduced during the time below.

I was surprised this worked! I expected require "config/environment" to be sufficient, especially since my production environment has:

  # Eager load code on boot. This eager loads most of Rails and
  # your application in memory, allowing both threaded web servers
  # and those relying on copy on write to perform better.
  # Rake tasks automatically ignore this option for performance.
  config.eager_load = true

Given config/environment calls Rails.application.initialize! I would have expected eager loading to happen then. I did find this example in the rails source which calls eager_load! after requiring config/environment.

I've only let this cook 12 hours, but my traffic and usages are pretty consistent with this app so I feel confident with my outcome thus far. I wonder if this change may be circumstantial and not needed with most apps, but thought a PR would be a good place to discuss.

Contribution

• I tested my changes locally.
• I agree to the Developer's Certificate of Origin 1.1.

[trevorturk]

I spent a little time searching to see if/how Puma handles this, but it seems a bit different, and the ActionCable example is interesting. I wouldn't think this would be necessary, but perhaps so! I'll give it a shot as well and report back...

[trevorturk]

I'm seeing a pretty clear improvement already, screenshot attached:

[trevorturk]

Reporting back after ~24hrs to say unfortunately I'm not seeing any improvement here. Note I'm using YJIT which I believe explains the gradual memory increase, but over this 7 day period, you can see the last deployment with the change results in a lower initial memory footprint, but it grows to be about the same over time.

[danielwestendorf]

Here is my last 24h. I too am running YJIT. Additionally, I'm using jemalloc and MALLOC_CONF=dirty_decay_ms:1000,narenas:2,background_thread:true, would recommend for consistent memory consumption.

I was able to increase WEB_CONCURRENCY=4 this morning. I plan to let this run for 24hrs as-is, and then I'll disable the eager_load! change in my preload, run for 24h, and see what the delta is.

[ioquatix]

Nice work, looking forward to seeing conclusive results.

[danielwestendorf]

Okay, so here are my findings, which I think are enough for me to make a decision. I'm back to running WEB_CONCURRENCY=3 for this experiment, as I didn't want to risk memory limits.

This first graph is from the regular heroku instance restart. Eager loading is enabled in my preload.rb. The starting RSS is 1,643MB and the peak before the next deploy/restart is 1,862MB.

This second shows the value after disabling eager loading in my preload.rb. The starting RSS is 1,498MB. You'll notice it is ~150MB lighter, which makes sense (fewer classes loaded). The max, however, is 2,159MB, which is significantly higher.

I think this is how the math plays out. Eager loading loads ~150MB per process in app. This is done once before fork with eager loading in my preload.rb, so that saves roughly ~300MB with a web concurrency of 3 (150MB of application code per process * 3 - 150MB paid before fork). 2,159MB - 1,862MB = 297MB.

I'm declaring this a win for my application. But is it for everyone? I've been noodling...

Imagining my application had a lot of code that was run in a web process only (like controllers) and also a lot of code that is only ever loaded in another process (like libs used in background workers), then we could make the argument that eager_load! before fork may consume more memory than is needed, as we're forcing all the code to load regardless. If that were the case, I could see why this default doesn't make sense. I do not, however, believe that most rails applications would meet this criteria.

[danielwestendorf]

Suggested change

	Rails.application.eager_load!
	# This will load *all* of your application code into memory before fork, regardless
	# of its usage (or not) in a web process.
	# Depending on the structure of code in your application, you may want to disable
	# this, however, this is probably not true for most rails applications.
	Rails.application.eager_load!

[jakeonfire]

fwiw, i believe rails is supposed to call eager_load! here when the environment is configured to do so: https://github.com/rails/rails/blob/b779a0189c0e780c7c4c49b983e949ce63a034b9/railties/lib/rails/application/finisher.rb#L76-L89

i believe this is (also) called via Rails.application.eager_load!

[jakeonfire]

also, i believe the expectation is that rails apps eager load in production, so your hypothetical case (where doing so would use more memory) is likely not only a rare one but also still typically expected to eager load. there are other ways to reduce the memory footprint for such applications. 👍

[trevorturk]

Yeah, I think this is likely safe to close, or maybe worth mention as an aside in the docs if we think it's got a wider use case?

[danielwestendorf]

Yes, this is a red herring. I am unsure why the differences in memory allocation are surfacing for my application.

I confirmed my application code is loaded by starting an irb session in a rails environment that has config.eager_loading = true

irb(main):003> require_relative "config/environment"
irb(main):004> defined? User
=> "constant"