Vendor `securerandom` #7930

byroot · 2024-08-12T21:26:02Z

It is loaded by Fetcher so in most case it's fine.

But if using bundler/inline and a gem need to be fetched, securerandom will be loaded and cause a conflict.

Can be reproduced with:

require 'bundler/inline'

gemfile do
  source 'https://rubygems.org'
  gem 'graphql', '~> 2.0'
  gem 'graphql-client', '~> 0.18'
end

require 'json'
require 'graphql/client'
require 'graphql/client/http'

Ref: rails/rails#52473 (comment)

It is loaded by `Fetcher` so in most case it's fine. But if using `bundler/inline` and a gem need to be fetched, `securerandom` will be loaded and cause a conflict. Can be reproduced with: ```ruby require 'bundler/inline' gemfile do source 'https://rubygems.org' gem 'graphql', '~> 2.0' gem 'graphql-client', '~> 0.18' end require 'json' require 'graphql/client' require 'graphql/client/http' ``` Ref: rails/rails#52473 (comment)

welcome · 2024-08-12T21:26:04Z

Thanks for opening a pull request and helping make RubyGems and Bundler better! Someone from the RubyGems team will take a look at your pull request shortly and leave any feedback. Please make sure that your pull request has tests for any changes or added functionality.

We use GitHub Actions to test and make sure your change works functionally and uses acceptable conventions, you can review the current progress of GitHub Actions in the PR status window below.

If you have any questions or concerns that you wish to ask, feel free to leave a comment in this PR or join our #rubygems or #bundler channel on Slack.

For more information about contributing to the RubyGems project feel free to review our CONTRIBUTING guide

indirect · 2024-08-12T21:27:50Z

is it possible to solve this more generally by having bundler inline re-exec itself once the gem install is done? we have historically tried to limit what we vendor by installing in a separate process from bundle exec.

byroot · 2024-08-12T21:29:16Z

bundler inline re-exec itself once the gem install is done?

I don't know. On paper that seems possible, but I'm not super familiar with the bundler codebase.

byroot · 2024-08-12T21:36:45Z

diff --git a/bundler/lib/bundler/inline.rb b/bundler/lib/bundler/inline.rb
index ae4ccf213..d62dd8775 100644
--- a/bundler/lib/bundler/inline.rb
+++ b/bundler/lib/bundler/inline.rb
@@ -60,6 +60,11 @@ def definition.lock(*); end
             Bundler.ui.info "Post-install message from #{name}:\n#{message}"
           end
         end
+
+        unless install
+          Bundler.ui.info "Re-executing script with installed gems"
+          Process.exec(RbConfig.ruby, $PROGRAM_NAME, *ARGV)
+        end
       end
 
       runtime = Bundler::Runtime.new(nil, definition)

Seem to work on the happy path, no idea if it could cause issues. Assuming there is no side effect in scripts before bundler/inline, it's probably fine.

indirect · 2024-08-12T21:50:45Z

awesome, thanks for investigating. I'd like to wait a little bit to get feedback from other maintainers and users, but I think in the long run it reduces the team's maintenance burden if installing can use non-vendored stdlib gems etc.

Alternate: ruby#7930 Fix: ruby#7930 When bundler inline has to install gems, it loads more dependencies than when it goes through the fast path of all gems being installed. One of them is `securerandom` so if trying to use `bundler/inline` with a gem that have a dependency on `securerandom` that don't match the default version, the script fails with `Gem::LoadError`. This can be preproduced on Ruby 3.2.x, after making sure to `gem uninstall securerandom` so only the default gem remains`, and then running the following script: ```ruby require 'bundler/inline' gemfile do source 'https://rubygems.org' gem 'activesupport', '7.2.0' # depends on securerandom >= 0.3 end require 'securerandom' ```

byroot · 2024-08-13T08:24:45Z

Opened the alternative at #7933

AliSoftware · 2024-08-13T11:55:51Z

Thanks a lot for investigating and taking the time to look into this! 🙇

Unless of course fork isn't available. Alternate: ruby#7930, ruby#7933 Fix: ruby#7930, ruby#7933 When bundler inline has to install gems, it loads more dependencies than when it goes through the fast path of all gems being installed. One of them is `securerandom` so if trying to use bundler/inline with a gem that have a dependency on securerandom that don't match the default version, the script fails with `Gem::LoadError`. This can be preproduced on Ruby 3.2.x, after making sure to `gem uninstall securerandom` so only the default gem remains, and then running the following script: ```ruby require 'bundler/inline' gemfile do source 'https://rubygems.org' gem 'activesupport', '7.2.0' # depends on securerandom >= 0.3 end require 'securerandom' ```

jeromedalbert · 2024-08-28T19:46:30Z

SecureRandom has been vendored in PR #7960 which got merged. Can we close this issue?

deivid-rodriguez · 2024-08-29T09:39:52Z

@byroot Can you still reproduce the problem? My suspicion was that to fix this issue we still need to vendor securerandom in Bundler like you did here (except for using the RubyGems securerandom if available) to make sure Fetcher loads the vendorized version.

byroot · 2024-08-29T09:41:05Z

I don't know how to test bundler/rubygems edge.

deivid-rodriguez · 2024-08-29T09:44:12Z

I normally use an alias like this dbundle='ruby -I $HOME/code/rubygems/rubygems/lib $HOME/code/rubygems/rubygems/bundler/spec/support/bundle.rb'. It's not the exact same thing as a real install, but it reproduces most of the issues.

byroot · 2024-08-29T09:51:48Z

Seems not to be working still:

$ ruby -I /Users/byroot/src/github.com/byroot/rubygems/lib/:/Users/byroot/src/github.com/byroot/rubygems/bundler/lib/ /tmp/secure.rb 
<internal:/Users/byroot/src/github.com/byroot/rubygems/lib/rubygems/core_ext/kernel_require.rb>:136:in `require'
<internal:/Users/byroot/src/github.com/byroot/rubygems/lib/rubygems/core_ext/kernel_require.rb>:136:in `require'
/Users/byroot/src/github.com/byroot/rubygems/bundler/lib/bundler/fetcher.rb:6:in `<top (required)>'
<internal:/Users/byroot/src/github.com/byroot/rubygems/lib/rubygems/core_ext/kernel_require.rb>:136:in `require'
<internal:/Users/byroot/src/github.com/byroot/rubygems/lib/rubygems/core_ext/kernel_require.rb>:136:in `require'
/Users/byroot/src/github.com/byroot/rubygems/bundler/lib/bundler/source/rubygems.rb:262:in `block in remote_fetchers'
/Users/byroot/src/github.com/byroot/rubygems/bundler/lib/bundler/source/rubygems.rb:260:in `to_h'
/Users/byroot/src/github.com/byroot/rubygems/bundler/lib/bundler/source/rubygems.rb:260:in `remote_fetchers'
/Users/byroot/src/github.com/byroot/rubygems/bundler/lib/bundler/source/rubygems.rb:267:in `fetchers'
/Users/byroot/src/github.com/byroot/rubygems/bundler/lib/bundler/source/rubygems.rb:405:in `block in remote_specs'
/Users/byroot/src/github.com/byroot/rubygems/bundler/lib/bundler/index.rb:9:in `build'
/Users/byroot/src/github.com/byroot/rubygems/bundler/lib/bundler/source/rubygems.rb:404:in `remote_specs'
/Users/byroot/src/github.com/byroot/rubygems/bundler/lib/bundler/source/rubygems.rb:138:in `specs'
/Users/byroot/src/github.com/byroot/rubygems/bundler/lib/bundler/resolver.rb:41:in `block in setup_solver'
/Users/byroot/src/github.com/byroot/rubygems/bundler/lib/bundler/resolver.rb:256:in `all_versions_for'
/Users/byroot/src/github.com/byroot/rubygems/bundler/lib/bundler/resolver.rb:54:in `block in setup_solver'
/Users/byroot/src/github.com/byroot/rubygems/bundler/lib/bundler/resolver.rb:360:in `filtered_versions_for'
/Users/byroot/src/github.com/byroot/rubygems/bundler/lib/bundler/resolver.rb:58:in `block in setup_solver'
/Users/byroot/src/github.com/byroot/rubygems/bundler/lib/bundler/resolver.rb:450:in `select_sorted_versions'
/Users/byroot/src/github.com/byroot/rubygems/bundler/lib/bundler/resolver.rb:426:in `block in prepare_dependencies'
/Users/byroot/src/github.com/byroot/rubygems/bundler/lib/bundler/resolver.rb:420:in `each'
/Users/byroot/src/github.com/byroot/rubygems/bundler/lib/bundler/resolver.rb:420:in `map'
/Users/byroot/src/github.com/byroot/rubygems/bundler/lib/bundler/resolver.rb:420:in `prepare_dependencies'
/Users/byroot/src/github.com/byroot/rubygems/bundler/lib/bundler/resolver.rb:63:in `setup_solver'
/Users/byroot/src/github.com/byroot/rubygems/bundler/lib/bundler/resolver.rb:28:in `start'
/Users/byroot/src/github.com/byroot/rubygems/bundler/lib/bundler/definition.rb:637:in `start_resolution'
/Users/byroot/src/github.com/byroot/rubygems/bundler/lib/bundler/definition.rb:306:in `resolve'
/Users/byroot/src/github.com/byroot/rubygems/bundler/lib/bundler/definition.rb:179:in `resolve_remotely!'
/Users/byroot/src/github.com/byroot/rubygems/bundler/lib/bundler/installer.rb:242:in `resolve_if_needed'
/Users/byroot/src/github.com/byroot/rubygems/bundler/lib/bundler/installer.rb:80:in `block in run'
/Users/byroot/src/github.com/byroot/rubygems/bundler/lib/bundler/process_lock.rb:12:in `block in lock'
/Users/byroot/src/github.com/byroot/rubygems/bundler/lib/bundler/process_lock.rb:9:in `open'
/Users/byroot/src/github.com/byroot/rubygems/bundler/lib/bundler/process_lock.rb:9:in `lock'
/Users/byroot/src/github.com/byroot/rubygems/bundler/lib/bundler/installer.rb:71:in `run'
/Users/byroot/src/github.com/byroot/rubygems/bundler/lib/bundler/installer.rb:23:in `install'
/Users/byroot/src/github.com/byroot/rubygems/bundler/lib/bundler/inline.rb:58:in `block (3 levels) in gemfile'
/Users/byroot/src/github.com/byroot/rubygems/bundler/lib/bundler/settings.rb:158:in `temporary'
/Users/byroot/src/github.com/byroot/rubygems/bundler/lib/bundler/inline.rb:57:in `block (2 levels) in gemfile'
/Users/byroot/src/github.com/byroot/rubygems/bundler/lib/bundler/settings.rb:158:in `temporary'
/Users/byroot/src/github.com/byroot/rubygems/bundler/lib/bundler/inline.rb:51:in `block in gemfile'
/Users/byroot/src/github.com/byroot/rubygems/bundler/lib/bundler.rb:430:in `block in with_unbundled_env'
/Users/byroot/src/github.com/byroot/rubygems/bundler/lib/bundler.rb:702:in `with_env'
/Users/byroot/src/github.com/byroot/rubygems/bundler/lib/bundler.rb:430:in `with_unbundled_env'
/Users/byroot/src/github.com/byroot/rubygems/bundler/lib/bundler/inline.rb:42:in `gemfile'
/tmp/secure.rb:3:in `<main>'
/Users/byroot/src/github.com/byroot/rubygems/bundler/lib/bundler/runtime.rb:310:in `check_for_activated_spec!': You have already activated securerandom 0.2.2, but your Gemfile requires securerandom 0.3.1. Since securerandom is a default gem, you can either remove your dependency on it or try updating to a newer version of bundler that supports securerandom as a default gem. (Gem::LoadError)
	from /Users/byroot/src/github.com/byroot/rubygems/bundler/lib/bundler/runtime.rb:25:in `block in setup'
	from /Users/byroot/src/github.com/byroot/rubygems/bundler/lib/bundler/spec_set.rb:191:in `each'
	from /Users/byroot/src/github.com/byroot/rubygems/bundler/lib/bundler/spec_set.rb:191:in `each'
	from /Users/byroot/src/github.com/byroot/rubygems/bundler/lib/bundler/runtime.rb:24:in `map'
	from /Users/byroot/src/github.com/byroot/rubygems/bundler/lib/bundler/runtime.rb:24:in `setup'
	from /Users/byroot/src/github.com/byroot/rubygems/bundler/lib/bundler/inline.rb:66:in `block (2 levels) in gemfile'
	from /Users/byroot/src/github.com/byroot/rubygems/bundler/lib/bundler/settings.rb:158:in `temporary'
	from /Users/byroot/src/github.com/byroot/rubygems/bundler/lib/bundler/inline.rb:51:in `block in gemfile'
	from /Users/byroot/src/github.com/byroot/rubygems/bundler/lib/bundler.rb:430:in `block in with_unbundled_env'
	from /Users/byroot/src/github.com/byroot/rubygems/bundler/lib/bundler.rb:702:in `with_env'
	from /Users/byroot/src/github.com/byroot/rubygems/bundler/lib/bundler.rb:430:in `with_unbundled_env'
	from /Users/byroot/src/github.com/byroot/rubygems/bundler/lib/bundler/inline.rb:42:in `gemfile'
	from /tmp/secure.rb:3:in `<main>'

jeromedalbert · 2024-08-29T22:56:47Z

I can also reproduce the problem with the latest rubygems main branch:

$ asdf local ruby 3.2.2 # You need to be on Ruby 3.2 to reproduce the issue

$ ruby --version
ruby 3.2.2 (2023-03-30 revision e51014f9c0) [arm64-darwin23]

$ gem uninstall securerandom --force; ruby -I ~/c/tmp/rubygems/lib repro.rb
/Users/jerome/.asdf/installs/ruby/3.2.2/lib/ruby/gems/3.2.0/gems/bundler-2.5.18/lib/bundler/runtime.rb:299:in `check_for_activated_spec!': You have already activated securerandom 0.2.2, but your Gemfile requires securerandom 0.3.1. Since securerandom is a default gem, you can either remove your dependency on it or try updating to a newer version of bundler that supports securerandom as a default gem. (Gem::LoadError)
        from /Users/jerome/.asdf/installs/ruby/3.2.2/lib/ruby/gems/3.2.0/gems/bundler-2.5.18/lib/bundler/runtime.rb:25:in `block in setup'
        from /Users/jerome/.asdf/installs/ruby/3.2.2/lib/ruby/gems/3.2.0/gems/bundler-2.5.18/lib/bundler/spec_set.rb:191:in `each'
        from /Users/jerome/.asdf/installs/ruby/3.2.2/lib/ruby/gems/3.2.0/gems/bundler-2.5.18/lib/bundler/spec_set.rb:191:in `each'
        from /Users/jerome/.asdf/installs/ruby/3.2.2/lib/ruby/gems/3.2.0/gems/bundler-2.5.18/lib/bundler/runtime.rb:24:in `map'
        from /Users/jerome/.asdf/installs/ruby/3.2.2/lib/ruby/gems/3.2.0/gems/bundler-2.5.18/lib/bundler/runtime.rb:24:in `setup'
        from /Users/jerome/.asdf/installs/ruby/3.2.2/lib/ruby/gems/3.2.0/gems/bundler-2.5.18/lib/bundler/inline.rb:66:in `block (2 levels) in gemfile'
        from /Users/jerome/.asdf/installs/ruby/3.2.2/lib/ruby/gems/3.2.0/gems/bundler-2.5.18/lib/bundler/settings.rb:158:in `temporary'
        from /Users/jerome/.asdf/installs/ruby/3.2.2/lib/ruby/gems/3.2.0/gems/bundler-2.5.18/lib/bundler/inline.rb:51:in `block in gemfile'
        from /Users/jerome/.asdf/installs/ruby/3.2.2/lib/ruby/gems/3.2.0/gems/bundler-2.5.18/lib/bundler.rb:429:in `block in with_unbundled_env'
        from /Users/jerome/.asdf/installs/ruby/3.2.2/lib/ruby/gems/3.2.0/gems/bundler-2.5.18/lib/bundler.rb:684:in `with_env'
        from /Users/jerome/.asdf/installs/ruby/3.2.2/lib/ruby/gems/3.2.0/gems/bundler-2.5.18/lib/bundler.rb:429:in `with_unbundled_env'
        from /Users/jerome/.asdf/installs/ruby/3.2.2/lib/ruby/gems/3.2.0/gems/bundler-2.5.18/lib/bundler/inline.rb:42:in `gemfile'
        from repro.rb:3:in `<main>'

jeromedalbert · 2024-08-29T23:09:30Z

To be fair I can also reproduce the problem when I check out this PR locally. So maybe the -I trick doesn't work as well, or something else is wrong.

deivid-rodriguez · 2024-08-30T12:46:48Z

I created #7984 based on this PR, and I verified it fixes the problem. Also fixes the issue reported by @byroot about improper namespacing of Random::Formatter.

Alternate: ruby#7930 Fix: ruby#7930 When bundler inline has to install gems, it loads more dependencies than when it goes through the fast path of all gems being installed. One of them is `securerandom` so if trying to use `bundler/inline` with a gem that have a dependency on `securerandom` that don't match the default version, the script fails with `Gem::LoadError`. This can be preproduced on Ruby 3.2.x, after making sure to `gem uninstall securerandom` so only the default gem remains`, and then running the following script: ```ruby require 'bundler/inline' gemfile do source 'https://rubygems.org' gem 'activesupport', '7.2.0' # depends on securerandom >= 0.3 end require 'securerandom' ```

byroot mentioned this pull request Aug 12, 2024

Declare Active Support dependency on securerandom >= 0.3 rails/rails#52473

Merged

byroot mentioned this pull request Aug 13, 2024

bundler/inline: re-exec script after installation #7933

Open

casperisfine mentioned this pull request Aug 19, 2024

bundler/inline: perform installation from a forked child #7941

Open

byroot closed this Aug 28, 2024

byroot deleted the vendor-secure-random branch August 28, 2024 19:47

deivid-rodriguez mentioned this pull request Aug 30, 2024

Fix bundler/inline failing in Ruby 3.2 due to conflicting securerandom versions #7984

Merged

4 tasks

Uh oh!

Vendor securerandom #7930

Vendor securerandom #7930

Uh oh!

Conversation

byroot commented Aug 12, 2024

Uh oh!

welcome bot commented Aug 12, 2024

Uh oh!

indirect commented Aug 12, 2024

Uh oh!

byroot commented Aug 12, 2024

Uh oh!

byroot commented Aug 12, 2024

Uh oh!

indirect commented Aug 12, 2024

Uh oh!

byroot commented Aug 13, 2024

Uh oh!

AliSoftware commented Aug 13, 2024

Uh oh!

jeromedalbert commented Aug 28, 2024

Uh oh!

deivid-rodriguez commented Aug 29, 2024

Uh oh!

byroot commented Aug 29, 2024

Uh oh!

deivid-rodriguez commented Aug 29, 2024

Uh oh!

byroot commented Aug 29, 2024

Uh oh!

jeromedalbert commented Aug 29, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jeromedalbert commented Aug 29, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

deivid-rodriguez commented Aug 30, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Vendor `securerandom` #7930

Vendor `securerandom` #7930

jeromedalbert commented Aug 29, 2024 •

edited

Loading

jeromedalbert commented Aug 29, 2024 •

edited

Loading