Bundler plugins

[jules2689]

Bundler Plugin System

The Bundler Plugin system allows a developer to integrate into certain points in Bundler's runtime. These include, at the time of writing, before-install, before-install-all, after-install, and after-install-all. There is also an option to add your own bundle commands like bundle viz.

Each of these points allows a plugin to interact with a Bundler::ParallelInstaller::SpecInstallation in the singular install options, and arrays of Bundler::Dependency objects in the -all options.

These provide users with the opportunities to modify the behaviour of bundle install to their heart's content.

I've seen two use cases in practice:

• https://github.com/jules2689/extended_bundler-errors : A bundler plugin that makes gem installation errors more actionable, educative, and all around easier to understand
• https://github.com/shopify/bootboot: Bootboot is a Bundler plugin meant to help dual boot your ruby application.

The former was created by myself and is intended for a large audience (everyone), while the latter was created by Rafael Franca and Edouard Chin - of the Shopify Rails team - and is intended for an organization or project needing to run two versions of a gem at once.

Distribution

The 2 aforementioned plugins presents two different distribution systems. The first errors handling plugin is intended to be installed using bundler plugin install while the latter is intended to be listed in the Gemfile using the plugin "blah" stanza. Both methods have issues that I'll enumerate in this issue.

bundler plugin install

The issue here is that it assumes users know to run bundle plugin install. This is not muscle memory, particularly since there exists practically no plugin gems available for use.

Gemfile's plugin stanza

Once a Gemfile lists plugin in the stanza, it now has to evaluate every single time. bundle install causes the plugin to be install every single time. Likely because it is not added to the .bundle/plugin/index file.

A Common Problem

What plugins exist? In what ways can I extend Bundler? No one can answer these questions. It is up to the plugin developer to market their plugin gem, to little benefit. Many people have never heard of Plugins.

This is even listed here: https://github.com/bundler/bundler/issues/4682

Issues with the plugin index

• The plugin index lists all gems with a full path (https://github.com/bundler/bundler/issues/6943, partial fix in a Ruby Buildpack for Heroku is here: https://github.com/intercom/heroku-buildpack-ruby/pull/4).
• The plugin index can be located at multiple locations (as is normal with bundler), either in ./.bundle/plugins/index or ~/.bundle/plugin/index - it's not obvious what is where though
• The plugin index has an unbounded growth in the hooks section. Likewise if you install multiple versions of the same plugin, it does not get overwritten, it's simply appended.

  commands:
  hooks:
      after-install:
      - "extended_bundler-errors"
      - "extended_bundler-errors"
      - "extended_bundler-errors"
      before-install-all:
      - "extended_bundler-errors"
      - "extended_bundler-errors"
      - "extended_bundler-errors"
  load_paths:
      extended_bundler-errors:
      - "/Users/juliannadeau/.bundle/plugin/gems/extended_bundler-errors-0.3.2/lib"
      - "/Users/juliannadeau/.bundle/plugin/gems/extended_bundler-errors-0.3.1/lib"
  plugin_paths:
      extended_bundler-errors: "/Users/juliannadeau/.bundle/plugin/gems/extended_bundler-errors-0.3.2"
  sources:

• I've had users experience plugins being deleted at the path specified in the index, to which Bundler seems to have issues

Developing/Managing Plugins

• Paths do not work in the bundle plugin install command, but once installed in another way, it ends up in the index forever and just works (TM)

  ~/src/github.com/jules2689/website(master*) ➜ bundle plugin install /Users/juliannadeau/src/github.com/jules2689/extended_bundler-errors
  Fetching gem metadata from https://rubygems.org/.
  Fetching gem metadata from https://rubygems.org/.
  Could not find gem '/Users/juliannadeau/src/github.com/jules2689/extended_bundler-errors' in any of the gem sources listed in your Gemfile.
  
  ~/src/github.com/jules2689/website(master*) ➜ bundle plugin install extended_bundler-errors
  Fetching gem metadata from https://rubygems.org/.
  Resolving dependencies...
  Using bundler 1.17.2
  Installing extended_bundler-errors 0.3.2
  Installed plugin extended_bundler-errors
  

• You cannot uninstall plugins once installed (https://github.com/bundler/bundler/issues/5447)
• There are no way to manage or list plugins (Add bundle plugin list command rubygems/bundler#5467) despite a list command being added (Add plugin list command rubygems/bundler#6120), it does not work.

  ~ ➜ bundle -v
  Bundler version 2.0.1
  ~ ➜ bundle plugin list
  Could not find command "list".
  

Summary

There are 3 main categories to focus on:

• Technical Issues
• Distribution
• Management of Plugins

These 3 categories cause plugins to be more difficult to develop than needed (installing local plugins), difficult to use (technical issues and management), and difficult to find (distribution). This section aims to provide a list of requirements to start to remedy these issues.

Technical Issues

• Fix the full path issue in the plugin index
• Implement a more coherent way to handle the plugin index, that is apparent and evident. Tell the user when a plugin is being installed that a plugin is "Installing globally" or "Installing locally"
  • plugin stanza goes to .bundle/plugins/index
  • bundle plugin install goes to ~/.bundle/plugins/index unless --local flag or something is set
• Make sure the index is not every growing in the hooks section
• Make sure that upgrading a plugin does not cause 2 entries of the same plugin (de-dupe the plugin paths)
• Make sure that the plugin stanza does not cause a bundle install to always evaluate and does not prevent a bundle check from succeeding

Distribution

• On Rubygems.org, list a section for plugins. This can be determined by looking for a valid plugin.rb file in the base of the gem.
• Once that is determined, then allow users to search for the plugins and present the gem in a different show page than a standard gem

I don't expect wide adoption of plugins. It is likely to be a fragmented system, so we should be looking to see how a plugin affects the ecosystem and adopt the ones intended for all users into core where it makes sense.

Management of plugins

• Implement a bundle plugin uninstall command
• Implement, properly, a bundle plugin list command
• Make sure users do not have to touch the index file
• Allow path installs in the bundle plugin install (this should be local only, I'd think?)

[jules2689]

cc @indirect @segiddins @colby-swandale
Here is the plugin write up I finally sat down and wrote.

[segiddins]

cc @asutoshpalai who implemented plugins

[indirect]

@jules2689 thanks! this is super helpful. Would you be interested in writing up an RFC-format document with what you think the next version of plugins should look like? If not, no worries, and we'll try to work on improving the plugin system as we have time.

[sergey-alekseev]

Could a plugin be installed only for a specific group? E.g. plugin 'my_plugin', group: :development or

group :development do
  plugin 'my_plugin'
end

UPD: if I place a plugin under the development group and run bundle install --without=development I get the following error:

NoMethodError: undefined method `full_gem_path' for nil:NilClass
  /usr/local/lib/ruby/site_ruby/2.5.0/bundler/plugin.rb:201:in `block in save_plugins'
  /usr/local/lib/ruby/site_ruby/2.5.0/bundler/plugin.rb:199:in `each'
  /usr/local/lib/ruby/site_ruby/2.5.0/bundler/plugin.rb:199:in `save_plugins'
  /usr/local/lib/ruby/site_ruby/2.5.0/bundler/plugin.rb:69:in `gemfile_install'
  /usr/local/lib/ruby/site_ruby/2.5.0/bundler/cli/install.rb:60:in `run'
  /usr/local/lib/ruby/site_ruby/2.5.0/bundler/cli.rb:235:in `block in install'
  /usr/local/lib/ruby/site_ruby/2.5.0/bundler/settings.rb:143:in `temporary'
  /usr/local/lib/ruby/site_ruby/2.5.0/bundler/cli.rb:234:in `install'
  /usr/local/lib/ruby/site_ruby/2.5.0/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
  /usr/local/lib/ruby/site_ruby/2.5.0/bundler/vendor/thor/lib/thor/invocation.rb:126:in `invoke_command'
  /usr/local/lib/ruby/site_ruby/2.5.0/bundler/vendor/thor/lib/thor.rb:387:in `dispatch'
  /usr/local/lib/ruby/site_ruby/2.5.0/bundler/cli.rb:27:in `dispatch'
  /usr/local/lib/ruby/site_ruby/2.5.0/bundler/vendor/thor/lib/thor/base.rb:466:in `start'
  /usr/local/lib/ruby/site_ruby/2.5.0/bundler/cli.rb:18:in `start'
  /usr/local/bin/bundle:30:in `block in <main>'
  /usr/local/lib/ruby/site_ruby/2.5.0/bundler/friendly_errors.rb:124:in `with_friendly_errors'
  /usr/local/bin/bundle:22:in `<main>'

Bundler version 1.17.3.

[ccutrer]

I've been looking this, and it seems several of the issues mentioned here have been fixed:

• bundle plugin list works
• bundle plugin uninstall works
• Duplicated plugin_paths and hooks no longer occur
• bundle plugin install --local_git works if your gem is in a repo directory

These things are still broken:

• Fetching/installing a plugin defined in the Gemfile every time (Bundle keep "installing" plugins from Gemfile #6630, fixed by Fix plugin installation from gemfile #6957)
• gem uninstall bundler-multilock -v 1.0.3 when the plugin index refers to the global gem can break the bundle until you run bundle plugin uninstall bundler-multilock. However, you have to be careful, because of bundle plugin uninstall wipes out your code if using a path #6953
• There's still no bundle plugin install --path or equivalent (Allow installing plugins from path via CLI #6960)
• No directory of plugins (Update known_plugins.yml and its Rake task rubygems/bundler-site#1222 fixes that)

I'd also like to see a new hook for before/after gemfile evaluation. It seems like a lot of plugins recommend doing Plugin.send(:load_plugin, ...) in the Gemfile so they can extend the DSL in some way, and before_gemfile_eval hook would allow them to no longer need the awkward load. (#6961)

[ccutrer]

Could a plugin be installed only for a specific group? E.g. plugin 'my_plugin', group: :development or

group :development do
  plugin 'my_plugin'
end

#6957 now fixes this as well

[ccutrer]

#6957 also now gracefully handles if a plugin is referencing a gem outside the .bundle directory, and that gem gets removed. It gives a nice error message Plugin(s) bundler-multilock (= 1.0.4) are not installed Run `bundle install` to install missing gems. rather than an obscure stack trace.

[hsbt]

The plugin system described in this issue has already been implemented. Since I don't understand the intention behind creating the issue, I'm closing this.

There's still room for improvement in Bundler's plugin system, so we'd like to discuss those separately and move forward like #6957 and #6961.