bundler/inline: re-exec script after installation

[byroot]

Alternate: #7930
Fix: #7930

When bundler inline has to install gems, it loads more dependencies than when it goes through the fast path of all gems being installed.

One of them is securerandom so if trying to use bundler/inline with a gem that have a dependency on securerandom that don't match the default version, the script fails with Gem::LoadError.

This can be preproduced on Ruby 3.2.x, after making sure to gem uninstall securerandom so only the default gem remains`, and then running the following script:

require 'bundler/inline'

gemfile do
  source 'https://rubygems.org'
  gem 'activesupport', '7.2.0' # depends on securerandom >= 0.3
end

require 'securerandom'

[indirect]

Thanks @byroot! I'd like to hear if @hsbt or @deivid-rodriguez have any concerns before we merge, but this is my personally preferred solution. 👍🏻

[byroot]

One issue though is that Ruby doesn't really have a way to grab all the script arguments to do a re-exec without loosing anything, as evidenced by the failing tests (they invoke the code with -e).

So this approach wouldn't work in some cases.

Ref: https://bugs.ruby-lang.org/issues/6648

[byroot]

Shower thoughts: if we can't re-exec, an alternative could be to do the install in a subprocess. If fork is available it's super simple (albeit a bit unorthodox).

[deivid-rodriguez]

It seems "clean re-exec" is moving forward in Ruby (thanks @byroot!). I think we could vendor securerandom to fix the immediate problem for now, and then introduce this re-execution once usage of original flags is available in Ruby, and progressively unvendor gems only needed for bundler/inline.

[byroot]

I think we could vendor securerandom to fix the immediate problem for now, and then introduce this re-execution once usage of original flags is available in Ruby,

That would take years. I think doing the install in a subprocess is a more attractive solution short term.

[deivid-rodriguez]

That would take years. I think doing the install in a subprocess is a more attractive solution short term.

What if fork is not available? Don't we have the same "no access to original arguments" issue? I don't think time is an issue as long as we start moving things forward, but if we can find a faster approach of course I'd be happy!

[byroot]

What if fork is not available? Don't we have the same "no access to original arguments" issue?

I don't think so.

If fork isn't available, we can spawn a subprocess, that subprocess doesn't need to have the same flags. We could also just not bother, use fork only when available.

[deivid-rodriguez]

My thought was that if the ruby process using bundler/inline has been started with certain ruby flags, and bundler/inline starts running gem installation in a subprocess, it should still respect the original ruby flags in the subprocess? For example, I think our tests would probably need that.

Regarding not bothering with any fallbacks if fork is not available, I guess that would still be an improvement, but then this problem we're trying to solve here still exists on non-fork platforms.

[byroot]

it should still respect the original ruby flags in the subprocess?

Is there any Ruby flag that would have an impact on gem installation?

[deivid-rodriguez]

In the case of our tests, I think they may complain because of flags that may affect the Bundler code that is loaded, for example, -I, or -r. For example, we use -r in our tests subprocesses to load some code to stub network requests before Bundler is loaded.

[byroot]

In the case of our tests, I think they may complain because of flags that may affect the Bundler code that is loaded, for example, -I, or -r.

Right, for tests specifically we could simply pass them via RUBYOPT this way they'd be preserved in the subprocess.

That said, IMO just using fork if available should be relatively simple and would solve the problem for the overwhelming majority of users, so I think it would be valuable to ship just that. Worst case on other platform it's installed in the same process and either that cause no problem or either you need to re-run the script.

[casperisfine]

IMO just using fork if available should be relatively simple and would solve the problem for the overwhelming majority of users

#7941

[deivid-rodriguez]

Coincidentially I found the same issue yesterday when running gem exec. We can also consider shelling out as well during gem exec, but I think the best immediate fix is to vendor securerandom, regardless of considering other options, so I proposed #7960.

[indirect]

Ok, sounds good to me. Thank you for your flexibility @byroot while we discussed options, and hopefully #7960 will solve things for you, at least for now.

[deivid-rodriguez]

I think @byroot's issue may still need a bit more work. Basically we'd need both #7960 and #7930 because both RubyGems and Bundler load securerandom. Fortunately we've improved vendoring a lot recently (@segiddins work 😍), so it's quite easy to do at the moment.

I can also look into what's not working in #7941, but given that it's more complicated than expected and doesn't work in all platforms, I do prefer the re-exec solution.