improve gem install --default

[mpapis]

Forwarding problems with the new rubinius and rubysl => rvm/rvm#2260 ... basically it is about standard library gems not found in few cases, usually with bundler.

What I would like to see from rubygems side is to make gem install --default to fully install the gems in $LOAD_PATH and build native extensions (like zlib), most of the code is in place there would be needed an extra if to recognize --default-stdlib - which would install the files in stdlib and compile extensions - compared to current --default which only adds specifications.

I'm opening this more for discussion and asking if that change would be approved, I can prepare code for it.

cc @headius, @brixen.

[yorickpeterse]

Little chat in the #rubinius channel about this:

       mpapis | txus, yorickpeterse https://github.com/rubygems/rubygems/issues/677
yorickpeterse | mpapis: Hm, looks interesting. How would this deal with GEM_PATH being unset?
yorickpeterse | (as is the case with bundler, or maybe it was GEM_HOME)
       mpapis | yorickpeterse, the gems would be already in $LOAD_PATH - nothing to deal with, it's
              | already loaded (the gem part, not require)
       mpapis | jruby uses already part of it - but the files are installed by jruby installation
              | not with gem install, this ticket is to add next option to also install the files
yorickpeterse | Aah                                                                            

Based on this I'd say this would be a rather nice solution to the problem (certainly in the long run).

[mpapis]

and here is the code that mostly needs to be changed https://github.com/rubygems/rubygems/blob/40ce68d1ffd67a6db61b7a3087748ad05cfa49b5/lib/rubygems/installer.rb#L218-L231

[headius]

👍 from me for sure. I wanted to do this when I added --default flag but had some questions...

• Where's the appropriate place to "permanently install" a gem? I think the answer is not putting it directly into the main lib dir, but that makes the question more difficult.
• Should the user specify a location? gem install --default --target= and that dir gets the lib files? or --target-lib= --target-bin=? The latter would be the most flexible and would allow permanently upgrading even in main lib dir.

A few things to note:

• I'm not sure the install logic currently supports dicing a gem into pieces like this, and I have no idea how hard it would be to do it.
• C extensions would obviously need to be built. They'd probably need to be built based on the original structure, too, and then copied into the target lib.
• Gems that use relative pathing could break when installed by default. Of course the answer to that is that they shouldn't use relative pathing.

Definitely want to see this in the short term. It completes (or helps complete) the default gem story.

[mpapis]

an idea after following discussion in #rubinius - create separate directory for the default gems like rubygems-default and make it add to $LOAD_PATH via require "rubygems/default" + add this line to rubygems.rb - so the default gems are loaded when rubygems is loaded and it's easier to separate it from the rest of ruby.

[drbrain]

Why not install to RbConfig::CONFIG["sitelibdir"]?

This directory is already on $LOAD_PATH and seems like the proper place for installing gems that should never be updated (like a default gem).

[mpapis]

I'm not sure it is properly set in rubinius /home/mpapis/.rvm/rubies/rbx-2.0.0/vm/include/capi:

capi_constants.h  capi_oop.h  gen  rbx_config.h  rbxti  rbxti.hpp  ruby  ruby.h

it does not exist for jruby and in mri it is path to rubygems

[drbrain]

My jruby and rubinius have a sitelibdir:

$ jruby -rrbconfig -e 'p RbConfig::CONFIG["sitelibdir"]'
"/Library/Frameworks/JRuby.framework/Versions/1.7.4/lib/ruby/1.9/site_ruby"
$ ~/tmp/rbx/bin/rbx -rrbconfig -e 'p RbConfig::CONFIG["sitelibdir"]'
"/Users/drbrain/tmp/rbx/site"

[mpapis]

yes they are there, it's just jruby directory did not exist and for rbx-2.0.0 sitelib pointed to capi ...

[headius]

We had some discussion on IRC about this with @evanphx @brixen and @enebo. This is the initial email I sent, and I will post some of the follow-ups:

Here's my notes on what we're looking for in JRuby wrt "stdlib" gems.

Current Status in JRuby

JRuby 1.7.4 shipped with a number of gem-based libraries in stdlib. Of these, only jruby-openssl could be activated from a gem. This was done by having the stdlib openssl.rb attempt to activate the gem before proceeding with requires.

This mechanism worked ok, but it was a manual change to the stdlib version of openssl files and it was a little fragile.

JRuby 1.7.5 shipped with several libraries moved out to gems using the RubyGems "default gems" mechanism. The files for these gems were in a flat stdlib dir exactly as they would have existed in the gems, and the "default" mechanism activated the gem version if installed. This has worked well for us and we've had no reports of issues.

Future Needs

As pointed out, the "default" mechanism requires a special check for every require for "should this file trigger gem activation". I concede this is special, but it works pretty well.

However, it feels like it might get cumbersome as more and more gems are moved out of stdlib. I think the real requirement this addressed is having "standard" libraries available without RubyGems being involved.

So, most of this was discussed on IRC, but here's the requirements as I see them:

• "Standard" libraries should be available without users having to opt-in (e.g. Gemfile containing "rubysl"). In other words, they should be loadable without RG having to do its gem activation dance and modify load paths and such.
• These "standard library gems" should be upgradable with RG. So if an updated version of the gem is available, that will be used instead of the preinstalled/default one.
• If a project claims they depend on a certain set of gems, they should not be forced to add additional gems just to get stdlib behavior. Standard library gems should be implicitly available.

If I've left something out let me know. I will also say that I currently don't care what happens beneath the shroud of "require" so long as users can get access to stdlib libraries without opting into the gems that represent them.

[headius]

Additional replies to that email:

@enebo added:

Another possible requirement:

• Require time should not get slower as a result of gemification of stdlib

@brixen added (I'm paraphrasing) that he was not interested in any special treatment for "stdlib", gems should just be gems, adding to Gemfile and so on does not concern him at all. The goals of this issue and the JRuby and MRI teams do not align with what he wants, I believe.

I still feel like there's a large value in having a standard library that's "standard" rather than dicing it up into a hundred dependencies you have to track. To that end, my primary requirement is that requiring a standard library like "optparse" or "json" should never fail and should never require you to opt into those dependencies in e.g. a Gemfile or other dependency mechanism.

[headius]

I spent some time reading over the way Rubinius handles bootstrapping RubyGems when it needs many standard libraries that are themselves gems. They have a fairly simply way that works, but I have some concerns.

The code is around here: https://github.com/rubinius/rubinius/blob/master/kernel/delta/code_loader.rb#L133

Basically, when booting RubyGems or other parts of Rubinius that need code from gems, they temporarily modify LOAD_PATH to contain all gem libs (basically gem/**/lib + LOAD_PATH). This allows requires from those gems to happen without actually activating the gems. In essence, during bootstrap, they're not gems anymore; they're just stdlib broken up into many paths.

After whatever bootstrap has completed, they replace LOAD_PATH with what it was before bootstrapping.

Concerns I have:

1. Some of the files required are still in LOADED_FEATURES.

   On boot of just a -e command line, there are

   This may not be a problem if they are no longer requirable, since subsequent requires should find newer versions. And if that same gem is actually activated later, the file will appear to already be loaded.

   Note that when I tested it, this appeared to only include the "monitor" and "thread" libraries (and their child libs) and the rubinius- libraries you'd expect to need.

2. The code loaded during bootstrap can't be replaced later on, but the requires might fire again anyway.

   The gems/**/lib search is fairly naive, and I believe it would proceed in filesystem order. This could easily choose an earlier installed version of a gem if two versions are installed. As a result, when you later try to activate a newer version of the gem, it will probably require over the top of the old one, probably redefine classes (and warn about redefined constants), and potentially break something that booted using the older version's logic.

   What we'd really want here is a mechanism for saying "temporarily add the newest versions of these N gems to load path so I can boot. But even this has a problem; a user might want to activate an older version.

   There's one more oddity:

3. Many lib dirs remain in LOAD_PATH.

   When I started up rbx -e 'p $:' there were already a couple dozen gems' lib paths in LOAD_PATH, including many that did not appear to have loaded anything. I'm not sure if this is a bug or intentional, but it means many libraries would require directly without going through gem activation, and unless you active a gem manually I think the already-booted version would be seen for requires.

   For the record, here's the LOAD_PATH and LOADED_FEATURES of rbx -e: https://gist.github.com/6928424

[brixen]

All those are implementation details of a work-in-progress. Rubinius will likely load all of RubyGems and its dependencies from pure bytecode.

Since RubyGems architecture currently (and unfortunately) requires it to be loaded at runtime, it (and its dependencies) should just be considered a blob of stuff that magically exists from the user's perspective. Ruby does not have a sensible code reloading facility and #require tries hard to not reload code, so unless a user specified --disable-gems, RubyGems magically exists.

The above, by itself, has no bearing on the discussion of how to provide the "standard library" as gems. The only thing Rubinius needs is a way to wrap loading RubyGems (which it already handles modulo the few rough edges) and ensure that when RubyGems is trying to update itself, Rubinius knows so it can provide the environment it needs.

Rubinius is providing a lot more than just the "standard library" as gems. The express purpose is to make it possible for people to discard the "standard library" at will or to upgrade it or other components without requiring installing a new package. The rubysl meta-gem is a transitional convenience for the standard library. But the perspective of the system providing a core (installed through some package manager) and a bunch of gems will become more pervasive in Rubinius.

[brixen]

The gist above is incorrect for `$LOAD_PATH'

$ bin/rbx -v -e 'p $LOAD_PATH'
rubinius 2.0.0n282 (2.1.0 78f8a2d5 2013-10-09 JI) [x86_64-darwin13.0.0]
["/source/rubinius/rubinius/vendor", "/source/rubinius/rubinius/lib"]

[headius]

All those are implementation details of a work-in-progress. Rubinius will likely load all of RubyGems and its dependencies from pure bytecode.

Well yes, they're implementation details. But that's what we're trying to figure out here. It's useful to know how Rubinius is currently managing its gemified stdlib.

And what you have done is clever. If we choose to go down that path, we can replicate it in JRuby (other than having to make our own JRuby-compatible gemified stdlib, of course).

Since RubyGems architecture currently (and unfortunately) requires it to be loaded at runtime, it (and its dependencies) should just be considered a blob of stuff that magically exists from the user's perspective. Ruby does not have a sensible code reloading facility and #require tries hard to not reload code, so unless a user specified --disable-gems, RubyGems magically exists.

Indeed! The remnants left in loaded features were all largely necessities to boot RubyGems and Rubinius's own internals, so I wasn't too concerned about the loaded features. And as you pointed out later, the load path contamination has been fixed. Kudos for that...it's much cleaner than it was when I made this build.

Rubinius is providing a lot more than just the "standard library" as gems. The express purpose is to make it possible for people to discard the "standard library" at will or to upgrade it or other components without requiring installing a new package.

In essence, you're killing the idea of there being a standard library at all. So there's a debate to be had as to whether there's any value in a set of libraries that are guaranteed to be available whether you specify a dependency on them or not. Or put another way, a set of libraries that are transitive dependencies because you're using Ruby.

The ability to remove the stdlib is not particularly interesting to me. Even now, you can ignore the stdlib and use other libraries to parse XML or JSON, run tests and specs, etc. The fact that the stdlib is there has not been an impediment to using alternative libraries. I don't see that gemifying stdlib and turning it from an implicit dependency to an explicit one does anything to make it easier to replace.

What a gemified stdlib does do, however, is make it easier to version and release individual parts of stdlib on their own. That's a big reason JRuby started gemifying our stdlib years ago (by moving openssl out to a gem) and gemified several additional libraries in 1.7.5. I believe that's also motivation for MRI gemifying libraries for their 2.0 release and working on the "default gem" system. We're all trying to reach the same goal.

But the perspective of the system providing a core (installed through some package manager) and a bunch of gems will become more pervasive in Rubinius.

Are you saying it will become increasingly pervasive in Rubinius in the future, or that Rubinius is the only impl pushing toward this goal? It seems to me we're all doing this right now...MRI in 2.0, JRuby in 1.7.5. And at least on the JRuby side, we've been planning for years to gemify more of stdlib in ways that work across all implementations.

I applaud you and Rubinius for exploring this path toward gemified stdlib. It helps us get perspective on how to continue gemifying the stdlib for JRuby and MRI.