Please deprecate (and/or rename) `add_development_dependency`

[e2]

The main reasons:

1. It's used incorrectly more often than not
2. It's not even practical for development nowadays (unless --development-all is used)
3. It's superseded by Bundler's groups
4. There's no distinction between "dev deps", "test deps", "build deps" and "deploy deps" - and these distinctions are getting more and more important every day (especially "test deps")
5. Development workstations don't need the "minimalism" provided by gem install --dev
6. Too easy to accidentally add/remove a dependency from the wrong section (runtime vs dev)
7. Putting development-related gems in a gemspec doesn't make any sense from a dev ops point of view
8. Average development needs are constantly growing and there's already too much stuff in the gemspec
9. Bundler supports easy platform switching during development - while gemspecs are only really useful for managing releases of builds
10. Organizing dependencies between gemspec and Bundler's Gemfile is simply too confusing - and deprecating this method would really help guide people in the right direction

Summary: gem development and testing is not the responsibility of RubyGems - and should never be, because otherwise it's just wasting precious developer hours.

[drbrain]

1. It's used incorrectly more often than not

I've never heard this complaint before. Can you link me to similar complaints?

1. It's not even practical for development nowadays (unless --development-all is used)

If you are developing for gem a you do not need development dependencies for gem b. I don't understand what --development-all would fix here. I guess this ties into complaint number 1 or is a bug in RubyGems. Can you clarify?

1. It's superseded by Bundler's groups

Bundler's groups don't get exported as metadata on gem servers so they can't supersede the Gem::Specification.

1. There's no distinction between "dev deps", "test deps", "build deps" and "deploy deps" - and these distinctions are getting more and more important every day (especially "test deps")

I see this as an argument for adding more dependency types, not reducing the existing number of types.

1. Development workstations don't need the "minimalism" provided by gem install --dev

Deployment does.

Do you mean "development dependencies should be listed in the Gemfile, runtime dependencies should be listed in the specification"?

If so this violates the DRY principle and I am highly unlikely to make such a change to RubyGems.

1. Too easy to accidentally add/remove a dependency from the wrong section (runtime vs dev)

I haven't seen any complains about this. Can you clarify?

1. Putting development-related gems in a gemspec doesn't make any sense from a dev ops point of view

I don't see this as compelling, it is not the only context that RubyGems must consider. (See also my response about DRY?)

1. Average development needs are constantly growing and there's already too much stuff in the gem spec

What do you mean by "too much stuff"?

Note that RubyGems relatively recently added a generic metadata hash to allow authors to be more descriptive with the content in their specifications, so the trend is to add more features to a specification, not less.

RubyGems does not download the full specification when determining which gem to install and instead downloads only a subset of the information available. The full specification is compressed inside the gem and typically consumes 512KB (a single tar block).

1. Bundler supports easy platform switching during development - while gemspecs are only really useful for managing releases of builds

I don't understand how this applies to development dependencies. See also the following

1. Organizing dependencies between gemspec and Bundler's Gemfile is simply too confusing

How is #gemspec insufficient? This seems to be the best way for using bundler along with a Gemfile:

source 'https://rubygems.org'

gemspec

and deprecating this method would really help guide people in the right direction

You have not yet convinced me it is the right direction.

This would be a breaking API change that could not be introduced until RubyGems 3. There is no schedule for such a release.

[e2]

It's used incorrectly more often than not

I've never heard this complaint before.

You'll find many gems with incorrectly/incompletely specified dev deps:

Example 1:

$ gem install cucumber --dev
Fetching: builder-3.2.2.gem (100%)
(...)
Fetching: cucumber-1.3.17.gem (100%)
ERROR:  Error installing cucumber:
    cucumber requires diff-lcs (>= 1.1.3)

Example 2:

 gem install --dev nenv
Fetching: diff-lcs-1.2.5.gem (100%)
Successfully installed diff-lcs-1.2.5
Fetching: rspec-3.1.0.gem (100%)
ERROR:  Error installing nenv:
    rspec requires rspec-core (~> 3.1.0)

Note how ridiculous it is for a developer to include all dependencies of rspec in their gemspec (or be expected to already have them installed).

Example 3:

$ gem install --dev aruba
Fetching: builder-3.2.2.gem (100%)
(...)
Fetching: cucumber-1.3.17.gem (100%)
ERROR:  Error installing aruba:
    cucumber requires gherkin (~> 2.12)

Example 4:

$ gem install --development-all rspec
ERROR:  While executing gem ... (Gem::DependencyResolutionError)
    conflicting dependencies minitest (~> 5.3, development) and minitest (~> 5.2, development)
  Activated minitest-5.2.0
  which does not match conflicting dependency (~> 5.3)
  Conflicting dependency chains:
    rspec (>= 0), 3.1.0 activated, depends on
    rspec-mocks (~> 3.1.0), 3.1.0 activated, depends on
    minitest (~> 5.2, development), 5.2.0 activated
  versus:
    rspec (>= 0), 3.1.0 activated, depends on
    rspec-core (~> 3.1.0), 3.1.0 activated, depends on
    minitest (~> 5.3, development)

(This one actually baffles me to no end)

All 4 examples suggest incorrect usage of the 'development gem' in the gemspec.

Can you point me to a gem that uses rspec AND can be installed on a clean system with the --dev flag?

Of course, the above gems rely on bundler resolving the dependencies.

Can you link me to similar complaints?

No one is probably using gem install --dev (which doesn't matter if people are using Gemfiles anyway).

Some people define rake as a dependency twice - once in the gemspec and once in the Gemfile, to "workaround" problems running rake on Travis (more about that later).

So the usage is obviously incorrect - but the problem is mitigated by using Bundler.

It's not even practical for development nowadays (unless --development-all is used)

If you are developing for gem a you do not need development dependencies for gem b.

Exactly. My point being: there are only two options for dealing with dev deps in gem and both aren't practical:

1. gem install --dev - fails to install dev deps if they have dependencies
2. gem install --development-all - installs every dev dep's deps, but installs a lot more too

I don't understand what --development-all would fix here.

I don't even understand what the use case is for that's still relevant nowadays.

It's the only other command (AFAIK) that makes uses of the 'add_development_dependency' entries.

You're right - and while it would do what gem install --dev fails to do, that's not what it's for.

I guess this ties into complaint number 1 or is a bug in RubyGems. Can you clarify?

Yes. As far as I understand, gem install --dev fails to install deps of deps by design.

So it's a problem either way:

• authors aren't declaring full list of dependencies = problem
• or, gem install --dev doesn't do what people expect = problem

It's superseded by Bundler's groups

Bundler's groups don't get exported as metadata on gem servers so they can't supersede the Gem::Specification.

What's the point of exporting development deps as metadata? I mean practically?

Runtime deps and everything else in the specification - no argument there.

Example: https://rubygems.org/gems/aruba

How is that information relevant? It isn't complete.

Take RSpec for example: https://rubygems.org/gems/rspec

No deps listed, but even with --development-all can't be installed.

There's no distinction between "dev deps", "test deps", "build deps" and "deploy deps" - and these distinctions are getting more and more important every day (especially "test deps")

I see this as an argument for adding more dependency types, not reducing the existing number of types.

From: http://guides.rubygems.org/patterns/#declaring-dependencies

Development dependencies are useful for when someone wants to make modifications to your gem. When you specify development dependencies, another developer can run gem install --dev your_gem and RubyGems will grab both sets of dependencies (runtime and development). Typical development dependencies include test frameworks and build systems.

So we have 4 types of dependencies:

1. runtime (important for users)
2. test frameworks (important for CIs)
3. build/deployment (important for building and releasing gems)
4. developer tools (pry+plugins, guard+plugins, notification, rubocop, etc. - completely optional and often a matter of taste)

Nowadays, tests have become integral to development so much, that they are often more important than developer tools.

Test dependencies and dev dependencies are semantically different (ask any dev op).

It doesn't make sense to install Pry on a CI (where there's no terminal anyway), while doing so may break the build for unecessary reasons (I won't even mention platform deps). With JRuby and Rubinius you likely even won't be able to install the dev tools - no matter how precisely they are defined in the gemspec.

Development workstations don't need the "minimalism" provided by gem install --dev

Deployment does.

If the gemspec had "add_deployment_dependency" and "add_test_dependency" I wouldn't argue - because even as metadata it could be used.

I may be able to install test deps on CI on every platform, but maybe only one platform supports deployment.

I agree the minimalism is useful - which is precisely why the name "development" defined on RubyGems page as "typically test frameworks and build systems" is confusing. It's not uncommon to see people adding Pry as a "development" dependency in their gemspec.

It's so confusing, deprecation is the only thing I can think of.

Personally, I'd love to be able to do gem install --test and not have things like Pry+plugins and Yard installed on the CI.

Technically, to match the def on RubyGems, for my needs the method should be add_test_dependencies and in the Gemfile I can put:

gemspec development_group: :test

gem 'pry', group: :development

And on CI do bundle exec install --without=development and everything works fine.

The problem is naming: I need to map dev -> test TWICE:

• once in the Gemfile above
• once "in my head" when I'm adding files to the gemspec using add__development__dependency

Because for me "development" and "test" are 2 different types of gems (like Pry vs Rspec).

Do you mean "development dependencies should be listed in the Gemfile, runtime dependencies should be listed in the specification"?

That would be perfect, yes. I can use the development_group trick myself, but I'm sure few people know about it.

I can make add_development_dependency useful for my needs. But the "know-how" needed to get this working is not something most developers can afford to spend time to get. They shouldn't need to have to learn that to get a decent setup.

If so this violates the DRY principle and I am highly unlikely to make such a change to RubyGems.

It's the opposite - this actually helps people make things DRYer, because often they put in a rake dependency in BOTH the gemspec AND the Gemfile - because for them "that's what somehow makes things start working".

This is a clean separation of responsibilities - there's nothing "duplicated" to break the DRY principle.

This may just be a conflict of definitions for the word "development". For me, this is sufficient to deprecate the method.

For me, the gemspec dev deps are VERY useful if I use them to define gems for the CI builds (because I prefer them in the gemspec, while Gemfile is much better suited for "real" development deps).

The problem with things like rake and rspec is: they are common to multiple groups (test and development), while e.g rake-compiler is common to e.g build and development. There are 2 workarounds to this:

• put everything in one big "development" list in the gemspec
• use Bundler's groups to organize them in such a way that some can be excluded from installing based on platform (like ronn) or options (like --without=development)

Bundler is better suited for this, because it's state is not version specific (that's what the Gemfile.lock - and it's why it doesn't make much sense for gems). RubyGems is version specific - and that's the point, because it defines releases.

In development mode, there's no version. (and so, e.g. it doesn't make sense to build a gem in order to run unit tests on it).

Too easy to accidentally add/remove a dependency from the wrong section (runtime vs dev)

I haven't seen any complains about this. Can you clarify?

I've made the mistake of adding development deps to runtime deps and vice versa.

Somehow the lines:

spec.add_dependency
spec.add_development_dependency

are just too easy to confuse. For me, the gemspec is much more important than the Gemspec because it affects users. As for the Gemspec I can experiment without breaking anything - which makes it perfect for trying out all sorts of development "conveniences".

Putting development-related gems in a gemspec doesn't make any sense from a dev ops point of view

I don't see this as compelling, it is not the only context that RubyGems must consider. (See also my response about DRY?)

Yes - and that's why I'm suggesting this context can be dropped in favor of putting focus on more practical issues.

What do you mean by "too much stuff"?

Compared to bundler:

source 'https://rubygems.org'
gemspec

Note that RubyGems relatively recently added a generic metadata hash to allow authors to be more descriptive with the content in their specifications, so the trend is to add more features to a specification, not less.

I'm not against adding features at all. Far from it. In fact, adding features usually means needs are changing. So if needs are changing, it makes sense that some thing are no longer relevant, necessary, or even useful.

So deprecations are not only unsurprising, they are to be expected.

And the deprecation I'm suggesting reflects a change of times - where gems have more dependencies than ever (through breaking up responsibilities into smaller gems), development tools are getting more advanced (Pry, rspec) and the productivity of the bleeding edge is prefered over stability (so pulling from git makes more sense than from rubygems), CI builds are more frequend and more demanding (so separating dev gemsets from test gemsets becomes more important).

As a sidenote I recently had a problem installing an older gem. A tiny gem installed for minutes and the --trace showed tar and ssl errors.

Turned out the gems used the rubyforge_home specification option - once removed, everything worked.

I wish that was deprecated and I had seen a message sooner.

RubyGems does not download the full specification when determining which gem to install and instead downloads only a subset of the information available. The full specification is compressed inside the gem and typically consumes 512KB (a single tar block).

By "too much" I mean visually and for editing. As a developer I care more about adding gem deps in the right place - I don't edit the metadata that often.

There's a much higher cognitive load while working with gemspec files. I usually just want to "fill it out" perfectly the first time and never work on the file ever again - that's probably also why I'd prefer to put the development deps in the Gemfile.

Bundler supports easy platform switching during development - while gemspecs are only really useful for managing releases of builds

I don't understand how this applies to development dependencies.

It's not uncommon to switch to non-rubygems sources for dependencies (:git, `:path). And I can tamper with the Gemfile in any way I want and it will have NO effect whatsoever on users installing them. Using the gemspec to do the same is an order of magnitude less convenient.

Also, it depends on how the word "development" is defined. The default Rails distinction of :test vs :development is more intuitive than RubyGems putting :test + :build in the term :development. From a RubyGems perspective, what kind of dependency is Pry? Is it "development"? The RubyGems definition seems to include it, but I'm not sure. (Before the "bundler age", you could just install "pry" and use it without declaring a dependency in the gem).

That's why I'm suggesting the deprecation. Rails has changed how people think about the terms since the original specification field was added.

(gem install --test used to be available but was thrown away - for good reasons and I'm just mentioning as an analogy).

gem install --dev isn't even expected to work anymore (which is why I probably won't find any of the complaints you mentioned - people aren't complaining that it doesn't work simply because either they don't use it, or they don't care).

How is #gemspec insufficient? This seems to be the best way for using bundler along with a Gemfile:

Yes, unfortunately the typical Gemfile with gemspec statement includes "gemspec dev deps" into Bundler's "development" group, and also typically, people expect to install on the CI with the --without=development option.

Symptom: if you add rake using add_development_dependency, it won't get installed on Travis (if using bundler args) and the build will fail.

The workaround people resort to: adding rake also to the Gemfile, which ends up in bundler's 'default' group, so it gets installed. Then people try the same thing with RSpec.

I don't know how to 'grep' the most popular gems, but I'm pretty sure there are lots of popular gems getting this wrong because of the confusion between the terms "development" and "development".

If rake doesn't get installed on CI, people aren't going to investigate further if adding it to the Gemfile solves the problem. (Which is why you won't get complaints - fixing the build takes priority, then people decide it's not worth reporting).

You have not yet convinced me it is the right direction.

I hope I've shed some light and shared my views clearly enough.

If not, feel free to ask further question or for more clarity.

I see 3 "solutions", but none of them are "obvious" (both are actually counter-intuitive in some way not fully documented):

1. assuming "gem dev deps" means "CI build deps", putting everything else (e.g. Pry) in the Gemfile in a :dev_extras group, using bundler install --without=dev_extras on CI

2. assuming "gem dev deps" means "dev extras" (pry, etc.), which means CI-only tools would go into a :test group in the Gemfile , and using bundler install --without development on CI

3. assuming 1), but putting everything else (e.g. Pry) in the Gemfile's :development group, mapping with gemspec development_group: test and using bundler install --without=development

You could blame it on incorrect/bad Travis docs, incorrect/bad Bundle defaults, but I think you'd agree that RubyGems was never designed to perfectly fit into this kind of workflow.

This would be a breaking API change that could not be introduced until RubyGems 3. There is no schedule for such a release.

According to SemVer deprecating is a minor version number change, which SHOULD happen before a major number change so that people can gradually migrate (check semver.org references to "deprecated"). So if this was to be introduced, it would HAVE to be either in 2.x or no earlier than 3.1.

So if this would be deprecated in 2.x, it could be removed completely in 3.0

[drbrain]

With respect to cucumber, etc., it seems that is a bug in RubyGems which should be fixed, not a reason to remove development dependencies. I thought the tests covered such cases but obviously they are inadequate.

For the cucumber case, diff-lcs is a regular dependency, not a development dependency and is not installed, thus the failure. I suspect the others are the same.

With respect to deployment I was unclear. Users do not want development dependencies installed along with a library they need in a deployed application. This is why multiple types of dependencies were added in the first place.

So far there has only been a handful of requests for multiple dependency types besides "runtime" and "development" which in RubyGems covers all development tasks including test-only environments and handy developer tools. RubyGems makes no distinction here.

You misunderstand the DRY principle which is “Every piece of knowledge must have a single, unambiguous, authoritative representation within a system.” (See The Pragmatic Programmer). For a library the gem specification is the single unambiguous authoritative representation of library metadata which includes dependencies.

Note that this is specific to ruby software released as gems and used as a library. For software that is used as an application there is no need for a gem specification thus the Gemfile.lock is where dependency metadata is stored. (Not in the Gemfile as a Gemfile alone does not allow consistent deployments. See http://yehudakatz.com/2010/12/16/clarifying-the-roles-of-the-gemspec-and-gemfile/)

Bundler cannot be used to exclude dependencies of gems it installs. If a gem depends on ronn as a runtime dependency then the platform your are installing on must support it. RubyGems does not support optional or platform-specific dependencies in this manner.

I see in your response that we could do a better job at documentation and could make some improvements around declaring dependencies but I don't see a compelling reason to remove development dependencies.

Overall it seems you have a clear distinction between what RubyGems considers a runtime and development dependency.

That you sometimes remove a dependency from the wrong place is not something that I see removing development dependencies solving. You don't seem to mind the idea of adding additional dependency types which is also something that would prevent you from removing a dependency from the wrong set.

As a gem author it is up to you to take care to ensure that the software you release is fully correct. RubyGems has continually added checks to help authors release gems that have minimal problems, but there is a limit to what RubyGems can check. Ensuring that your dependencies are properly specified in your gem specification is one of those things that is difficult for RubyGems to check statically.

PS: I find bundler entirely inadequate as a tool for developing gems. It requires you to rebuild test suites, documentation generation, release process, correctness checking, etc. for every library you write. I use hoe which has built-in tasks or reusable plugins for all these features. For application deployment it is an acceptable tool.

[e2]

With respect to cucumber, etc., it seems that is a bug in RubyGems which should be fixed, not a reason to remove development dependencies. I thought the tests covered such cases but obviously they are inadequate.

My point being: no one actually cares it's even broken.

For the cucumber case, diff-lcs is a regular dependency, not a development dependency and is not installed, thus the failure. I suspect the others are the same.

Seriously, no one cares! You're time is EXTREMELY precious given the role you are playing in the Ruby world. Instead of working of fixing something no one cares about, I believe it would be better to show a deprecation with a url, and have a clever description protecting you of any "accusations".

Users do not want development dependencies installed along with a library they need in a deployed application.

And CI's are also better off also not having those installed. The problem is dev and test share more deps than runtime and dev do. Since CIs have replaced testers, CI don't require "human tools" (like PRy). And CIs have become "first class citizens" in typical development workflow. Obviously, they can't cry out and fight for their rights to "better working conditions" with "less effort for needless installs of human-only dependencies".

So far there has only been a handful of requests for multiple dependency types besides "runtime" and "development" which in RubyGems covers all development tasks including test-only environments and handy developer tools. RubyGems makes no distinction here.

Neither does bundler - to the point that bundler resolves and installs EVERYTHING - except when the --without option is given (resolving is still done on all runtime+dev gems).

But the --without option is still very useful. E.g. maybe I'm a contributor and I don't want to touch anything related to the gem's docs. Or I'm fixing the docs (e.g. translating), and I don't want any of the build and tools.

Also, many dev tools "essential" for human being may not even be gems in the first place (external repos, tarballs). I don't see a reason why RubyGems should even care.

You misunderstand the DRY principle which is “Every piece of knowledge must have a single, unambiguous, authoritative representation within a system.” (See The Pragmatic Programmer). For a library the gem specification is the single unambiguous authoritative representation of library metadata which includes dependencies.

First, thanks for the quote - it's much better defined than what I had in mind.

And it also just solidifies my point. Simply because runtime dependencies are obviously needed by RubyGems, there's no need to force developers to "log" which tools they used in RubyGems as well (because that's what defining development dependencies is - just developers declaring what they've used for development).

In short: development dependencies and runtime dependencies together are NOT - by any stretch - "a single piece of knowledge". They are two unrealated pieces artificially "mashed" together inside a single gemspec. And test deps included with development deps don't represent a "single piece of knowledge" either. (Any reasonable dev op would confirm).

I believe it would be better to even have a separate file for development dependencies - and (if at all) just have a reference to it in the Gemfile. Because it would be rarely needed by RubyGems (if at all).

Currently, the existence of the add_development_dependencies method "inspires" people to unnecessarily fill it out - "just because it's there". Bundler has a rake task for installing dependencies (rake spec:deps) - maybe there should simply be a "convention" for this in Bundler's gem tasks instead. (Since it relies on Rake, which RubyGems shouldn't care about.)

Note that this is specific to ruby software released as gems and used as a library.

I am constantly aware of this - I'm actually trying hard not to let conventions from the "Rails world" cloud how I'm thinking about this subject (because the rationale there is very different).

Then again gems and apps are getting a lot more similar in terms of the complexity of the development workflow (deploying to hosting provider is not much different from deploying a release to RubyGems. rake release is not much different in terms of outcome from cap deploy).

For software that is used as an application there is no need for a gem specification thus the Gemfile.lock is where dependency metadata is stored.

I see this as yet another reason to delegate the definitions to the Gemfile. The only reason I can think of to not put them in the Gemfile is: if Gemfiles (and Gemfile.locks) are ultimately going to be deprecated in place of gemspecs. Because why let dev dependencies be specified BOTH in the Gemfile and gemspec - especially since now RubyGems uses Gemfile.lock for dependencies

(Not in the Gemfile as a Gemfile alone does not allow consistent deployments. See http://yehudakatz.com/2010/12/16/clarifying-the-roles-of-the-gemspec-and-gemfile/)

Actually, that's the first decent article I came across when I was searching for "good practices" about defining dev deps in gemspecs. And partially the reason I brought this up. Development deps for gems weren't expanded on in the article - probably because they are useless or irrelevant anyway (except when things fail when combined with --without in counter-intuitive ways).

Bundler cannot be used to exclude dependencies of gems it installs. If a gem depends on ronn as a runtime dependency then the platform your are installing on must support it. RubyGems does not support optional or platform-specific dependencies in this manner.

I was talking about development dependencies - ronn is used for generating man pages for gems, but it can't be installed on JRuby (I think it was because of rdiscont). So it's a good example of a development dependency essential for updating docs, but not e.g. for fixing bugs. So if a developer is fixing JRuby-specific bugs, it doesn't make sense for installing dev gems to fial simply because it's listed as a dev dependency.

It's unclear if dev deps in the gemspec are meant to be "optional", "suggested" or "essential". So if development deps were meant to only be in Gemfiles, I'd simply have a :doc group for Ronn and install using bundle install --without=doc on a jruby. Or even have it conditional depending on presence of JRuby - it's a development dependency, so no one will get hurt from it not being installed.

If dev deps are meant to be correct, they should be checked (during gem build for example). If they are meant to be a manifest, then it would be more useful to explain why the dependency is needed (e.g. docs) than expect people to provide version constraints which are never checked in a practical context.

I see in your response that we could do a better job at documentation and could make some improvements around declaring dependencies but I don't see a compelling reason to remove development dependencies.

I don't see a practical reason for development dependencies to exist. I mean if the field became a no-op today, what exactly would that "break" that isn't broken already?

I can't find a single use case as to why dev deps should be in the gemspec, and not a Gemfile (especially since only Bundler can reliably install dev deps right now).

Maybe informational metadata is useful - but then there's no reason to supply version constraints, because enforcing them would be more painful than not (ronn example on JRuby).

That you sometimes remove a dependency from the wrong place is not something that I see removing development dependencies solving.

If dev deps in the gemspec were "deprecated", I wouldn't be able to put dev deps there without feedback telling me not to (I'd have to put them in the Gemfile, obviously). This would make a clear distinction: runtime stuff -> in the gemspec. dev stuff -> in the Gemfile it goes.

With dev gems allowed in the gemspec I have TWO places I can add dev deps to. That violates DRY for me. (and the development_group option is kind of proof of DRY being violated - because it means Bundler is mapping gemspec devs into it's own group).

You don't seem to mind the idea of adding additional dependency types which is also something that would prevent you from removing a dependency from the wrong set.

It's an "I can live with it" thing - if metadata is so important and the gemspec provides the distinctions tools need, then I don't have to bother with a Gemfile anymore. Because I actually need multiple types defined (:dev, :test: :build, :doc), and I can do all that cleanly in a Gemfile right now. If gemspecs supported groups and it was recommended over putting them in the Gemfile, that would work for me too.

(It would still be "strange" to define "gem-version agnostic" dev deps in the gemspec, but if that would be recommended over putting them in the Gemfile for gems, I'd live with it).

But having "just" a "development group" (aside from :runtime) means I have to decide whether :development in the gemspec means: :dev or :build or :test or all of them (including :doc) - and other gem authors are going to make different decisions.

So if the dev deps aren't clearly defined what they should have (as opposed to the homepage field, where I don't put e.g. my phone number), that's why I'm suggesting to deprecate that in favor of something recommended, but maybe a bit more specific and practical.

I'm open to hearing a "right way to do things" or being pointed to docs showing the "the right way" - and Yehuda's article is the best thing out there. And from what I understand, the idea of putting dev deps into the gemspec is wrong.

Just to switch to apps/Rails for a second: the distinctions there are MUCH clearer and more practical. The only issues is Bundler resolves all dependencies anyway (just skips the installs if told to), and many people just put all their gems in both the development and test and production groups until they want to optimize deployment or CI builds. Other times people put things in development, they deploy, they see things broken and then they separate things into groups (which is good, because they can change their decision about deps at any time - and even delay the decision until it's necessary).

Rubygems doesn't allow me to optimize those, because I have no idea what development should semantically mean in the RubyGems. If it's just metadata no one cares about, I can just as well put my dev deps in the Gemfile.

The thing is: most people learn through "herd mentality" - if something is done in a gemspec in one project, they mindlessly "copy" it thinking it's a good practice. So deprecations seems to be the only way to communicate the intention behind the gemspec dev deps.

To me it seems add_development_dependency it's an ancient artifact from a pre-bundler and pre-TDD world. If so, deprecation is the obvious solution. Now it makes as much sense as setup.rb.

After all the investigation and research about how to use "gemspec dev deps" correctly, I still don't understand how specifying them is even useful from even a metadata perspective. It either increases bandwidth unnecessarily, or, if it's optimized not to be served - that just proves it's not important for anything.

As a gem author it is up to you to take care to ensure that the software you release is fully correct.

Yes, that's why I have no objection to defining runtime dependencies in the gemspec (or install time dependencies). I expect installing a gem to work. And I do not expect development dependencies to have ANY impact on installing a gem. If they do, it's a bug or I've screwed up the runtime dependencies.

For some gems I may need Rails for development and testing (not unusual). But who really cares if I do? Only the person who is working on the area of the gem that actually needs it. Maybe it's only good for debugging. Maybe it's used for generating docs (not a strange idea).

RubyGems has continually added checks to help authors release gems that have minimal problems, but there is a limit to what RubyGems can check.

And that's why I'd leave dealing with development dependencies to Bundler. It's because I don't want RubyGems to be checking stuff - especially if I'm using other gems in the form of potentially checked out master branches. I don't care about the version number at all, because on a master branch of a gem can have the same version number - but a totally different behavior (which I'm actually relying on). Or if the gem is no longer maintained, I may need to switch to a fork (which obviously isn't released as a gem).

I don't see a reason for RubyGems to have anything to do with my workflow in regards to development, testing, generating docs, etc. It's only once I've finished development and I want to release - when I even care about version constraints or metadata.

Ensuring that your dependencies are properly specified in your gem specification is one of those things that is difficult for RubyGems to check statically.

I only expect RubyGems to come into play when the user types in gem install, or when they need help or support (basically contact info/website). That alone is awesome. If RubyGems can tell me during gem build that stuff in a gem I'm contributing to is outdated (like the rubyforge_home metadata), I'd be overjoyed.

But otherwise RubyGems only makes development harder if I have specify the deps manually and then manually install them (or add a special rake task to do that). With Bundler I can just put in gem 'rspec' and I'm done. If someone has outdated versions, I don't want to care about them if bundle update can resolve the issue.

PS: I find bundler entirely inadequate as a tool for developing gems. It requires you to rebuild test suites, documentation generation, release process, correctness checking, etc. for every library you write. I use hoe which has built-in tasks or reusable plugins for all these features. For application deployment it is an acceptable tool.

I would have agreed not that long ago. Nowadays I find that if a gem needs that kind of "management", it probably should be split up into smaller gems. It's as if when then documentation doesn't fit in the README - or the code isn't sufficiently simple and descriptive to replace API documentation - then I'm overdoing things.

Ironically, RubyGems makes it so easy to deal with "tiny" gems ... and Bundler takes care of dependencies so well ... releases can happen faster with more confidence, APIs of components can mature faster and independently and each component can have it's own branches, so migration becomes fluid and unobtrusive - yet still any change can be backported without forcing people into major versions they otherwise don't care about, etc.

I've used hoe in the past and it somehow overall discouraged me from creating gems. I was based on great ideas (extracting the specification data), and the plugins were nice - it "seemed" to be more optimized for "managing" gems intended for users who are developers. (The coolest thing about hoe (that I miss) is that I actually can avoid seeing the gemspec altogether.)

And I wish I could "hoeize" existing gems in a less intrusive way that existing gem authors would accept in their project.

To me "adding" stuff to a gem "from the outside" (as opposed to editing existing files as configuration) just seems better - probably because I prefer to add functionality through "revertible commits" and not "config".

I also do things very differently nowadays, which probably explains why I don't see the point of involving RubyGems in development:

1. getting the source - hub clone user/project
2. adding tests to new project - bundle exec rspec --init (plus add Rakefile task - which is a bit bothersome I admit)
3. running all tests - bundle exec rake
4. test config - .rspec
5. releasing - edit version.rb && bundle exec rake release and sometimes publish release notes automatically using Octokit
6. changes - git push -u origin branch_name && hub pull-request (triggers Travis build on multiple Rubies, etc. - I can monitor with hub ci-status)
7. test isolation check - for spec in spec/**/*_spec.rb; do bundle exec rspec $spec || break; done

And those commands are pretty project-agnostic, so it would be hard to justify migrating projects to use hoe.

On the otherhand, if hoe could "lint" projects and "autocorrect" them (without becoming a dependency) - it would be one of the most marvelous tools out there.

Also, since docs usually represent the API for SemVer, if docs change without a major release it's more likely a sign that compatibility was broken.

I don't see a reason for even Hoe to actually bother with the development dependencies. The hoe newb command could just as well use a Gemfile or even Gemfile.lock for installing dependencies. And strangely in hoe, I can't run rake deps:list on an unreleased project.

To summarize, here are options I'd be happy with:
a) dev deps in a separate file (Gemfile would be fine) so I can define them for installing after cloning a project (but not for inclusion in the metadata)
b) separate metadata/groups for dev and test (and probably build/doc - e.g. rake-compiler, ronn), so that bundler install --without=development would work as expected
c) deprecating the method, so a warning is shown during gem build

I believe I understand this issue well enough to explain to other developers how to correctly get RubyGems, Bundler and Travis working as expected. And given the amount of people out there - I can't pass on this knowledge by contacting every person on the planet. I'm not really interested in sending thousands of PRs to fix other people's dev deps (I'd die under email notifications). And it's annoying for me to break away from the PR/fix I'm working on to create a separate PR or commit just fixing the deps.

So it would be really nice if other people just had a way of knowing how to define dev/test/build/doc dependencies to get the results they want both for them and other contributors to their projects.

As a contributor, I'm a "nobody", so people will be suspicious if I submit a PR fixing "something" in their gemspec.

But if RubyGems points them to an article with explanations - I can just point them to that official doc even in the commit. And the patch is more likely to be accepted.

If I have trouble convincing even one of the biggest Ruby heroes out there that this is confusing...

... there's a problem either in my understanding or my communication.

I'm not asking to touch anything that would affect users installing gems (that would obviously be a nightmare for the Rubygems team). This is no "worse" than the "gem dependency defined multiple times" Bundler warning when people have e.g. rake in both the Gemfile and gemspec. In fact the deprecation would help guide people towards correctly avoiding the warning.

According to SemVer it would be perfectly normal to include this in the next minor release - so it will only affect developers who've upgraded RubyGems.

Let me know if I can do anything else to help resolve this (or if there is anything this deprecation could actually "brake").

[drbrain]

I've fixed at least two bugs in development dependency installation (as fallout from the switch to the dependency resolver) in the past six months. There is one or two more that isn't explicitly mentioned in History, so I know the feature gets used.

With respect to time invested, maintaining the feature is something I will continue to have to do until the feature can be removed. Since there is no timeline on removal and, relative to the rest of my workload, the maintenance burden is small I do not find this compelling. Just because I am removing a feature doesn't mean I'm allowed to stop fixing bugs in it as that is equivalent to removing the feature.

There are many different types of users of RubyGems, not just "dev ops" people. I can't make a decision based on the ideals of one group of users unless it is absolutely overwhelming.

There are many gems that are written and released without using Bundler in any way. Forcing users to edit two files to build their gems does not follow the DRY principle regardless of how you wish to define dependencies. You still force users to put one type of gem metadata in one place and another type in a separate place.

It is almost always more work for me to remove a feature from RubyGems than it is to maintain it. Removing a feature involves many carefully orchestrated, documented and announced steps over many months and years. In this particular case it involves writing new code to deal with existing gems with existing development dependencies in a sane manner, something I will need to do for the rest of RubyGems history.

I've read and skimmed about a fifth of what you've written, I don't have the time or energy to read it all, can you summarize down to a three or four paragraphs?

[e2]

I can't make a decision based on the ideals of one group of users unless it is absolutely overwhelming.

Sorry if I suggested that. To me the feature feels "abandoned" by users (given it doesn't work for popular gems).

There are many gems that are written and released without using Bundler in any way.

Yes, and those users are likely manually using gem install instead of gem install --dev on whatever is missing.

Forcing users to edit two files to build their gems (...)

BUILD GEMS??? So that's what I was confused about!!!

It's not "development dependencies", it's "BUILD dependencies" the gemspec defines!

So I can assume:

$ gem install --dev

is REALLY something like:

$ gem install --build-deps

This makes LOTS of sense to me now!

I wish there was at least an alias to add_development_dependency such as:
add_build_dependencies.

An in Gemfiles I could use e.g.:

gemspec development_group: :build

gem 'foo', group: :development # (etc.)

I'm sorry I was so confused about this and it took me this long to make the distinction I needed.

My replies:

With respect to time invested, maintaining the feature is something I will continue to have to do until the feature can be removed.

I understand completely.

Since there is no timeline on removal and, relative to the rest of my workload, the maintenance burden is small I do not find this compelling.

My goal here is to reduce confusion without creating extra work for you.

Would aliasing 'add_build_dependencytoadd_development_dependency` be too much to ask for?

Just because I am removing a feature doesn't mean I'm allowed to stop fixing bugs in it as that is equivalent to removing the feature.
It is almost always more work for me to remove a feature from RubyGems than it is to maintain it.
Removing a feature involves many carefully orchestrated, documented and announced steps over many months and years.

I understand those - that's why I though deprecation would be a good way of "testing the waters" without risk or maintenance costs.

Thank you sincerly for your time and insights!

Thanks and have a warm, happy Christmas!

[chulkilee]

+1 to mark add_development_dependency deprecated and remove in the next major release (v2)

Agree that it might be useful in pre-bundler era. However things changed and any ruby projects should use bundler - including gem development. I don't know what is the merit of having dev dependency in gem spec itself and in gem source.

If it's extraneous feature, then it would be better to deprecate and then remove it to focus on other key features.

With respect to time invested, maintaining the feature is something I will continue to have to do until the feature can be removed.

I appreciate contributors' efforts on this feature, but I don't think we need to consider such sunken cost. Of course we have to get more feedback and then take phaseout process when decided.

There are many gems that are written and released without using Bundler in any way. Forcing users to edit two files to build their gems does not follow the DRY principle regardless of how you wish to define dependencies. You still force users to put one type of gem metadata in one place and another type in a separate place.

I get your point but I'd argue that using a gem and developing a gem is different use case. In bundler era all ruby developers are expected to use bundler in their environment. dev dependencies can be in such environment by Gemfile. It doesn't need to be in gemspec.

There are many different types of users of RubyGems, not just "dev ops" people. I can't make a decision based on the ideals of one group of users unless it is absolutely overwhelming.

I think all *users use just runtime dependency. dev dependency is for developer of each gem.

I think the main point is how seriously we consider bundler as a "standard tool"...

[evanphx]

I don't see any reason to deprecate it. Having development dependencies is still super useful and there is really no reason to remove them.

[e2]

@evanphx - how so? I've always believed that, but now I can't find a single "good" reason where it isn't using the "wrong tool for a given job".

The point I 100% agree with is that given the "age" and scale of RubyGems, deprecation is simply not worth the actual costs.

[e2]

TL;DR - I agree deprecation is costly, so I think a "new" tool (alongside Bundler and RubyGems) would be a better place for "new features" and "deprecations".

@chulkilee - thanks, you totally understand me!

I don't know what is the merit of having dev dependency in gem spec itself and in gem source.

And without Bundler, there's no guarantee you're actually using the right versions of build tools to build the gem anyway (while rake install can use Bundler implicitly).

I appreciate contributors' efforts on this feature, but I don't think we need to consider such sunken cost. Of course we have to get more feedback and then take phaseout process when decided.

I understand that just keeping RubyGems working as it is requires a ton of effort few people are even capable of appreciating. Also, there are at least thousands of people "relying" on the feature (even for strange purposes), so any "feedback" will be enormous in quantity (i.e. tons of "complaints").

I think a better option would be a "new" tool that would integrate Bundler and RubyGems functionality in an "integrated" and "recommended" way. Package managers are a tough problem - especially when it comes to backward compatibilty. While it would be confusion to have another tool (given there's Bundler and RubyGems already), but it does give millions of people a choice (backward compatibility vs new features).

Also, a lot of gems out there are "ancient" - adding a deprecation to them would only cause maintainers and users to waste focus.

I think all users use just runtime dependency. dev dependency is for developer of each gem.

I can't agree more.

Gems are for "use", while repositories are for development. (In a perfect world, we'd all be using the master branches of everything in production).

Not to mention - almost every development dependency can be optional (contributing/developing does not necessarily imply you have to be able to build a gem yourself on your own workstation). And so if dev/build deps are optional - why should they be expected to be part of a "released specification" (gemspec)?

I think the main point is how seriously we consider bundler as a "standard tool"...

I'd disagree that it's even about bundler - you can have a Rakefile so that e.g. rake install:deps uses gem install -v <version> <gem> and other system commands to make sure the dev environment is properly set up (including checking out other repositories, submodules, toolchains, OS packages, IDE plugins, etc).

A gemspec is just too restrictive for dev deps - you can't lock to a specific version of Golang for example (e.g. zeus gem requires golang properly set up to build it) and you can't use checked out repositories. Even plain Rake is is more flexible and also more "standard" for this.

[evanphx]

@e2 What features are you thinking a new tool needs? We've been adding new things people need to Rubygems fine.

Because dev_deps is unable to articulate the universe of development needs doesn't mean it's a bad option. It's not uncommon when working on a gem for the first time to install the development dependencies and the best way they're articulated is via dev_deps.

[e2]

TL;DR - Bundler Rakefile shows that gemspec devdeps don't help with a real-world problems

@evanphx - a new tool would "absorb" paradigm changes faster than the heavily relied on RubyGems which already has a massive legacy or expensive (backward-compatibility) features.

As for dev_deps in a gemspec - I believe the fact that no extremely knowledgeable and experienced Ruby "guru" (e.g. you or @drbrain) can articulate a single "good" use case for it is ... case in point.

It's not a bad option technically (I'd just say it's plain inferior given other options) - it's just "cargo cult". It's like a configuration option that no one uses, but everyone specifies because "other people do it".

Rake and Bundler are much better for setting up the dev environment - whether those deps are in the gemspec or not is just an implementation detail. There's just no advantage to having those in a gemspec - only disadvantages.

This is especially true for platform-specific dev tools (Rubinius, JRuby, etc.).

It's pretty much the same story as with specifying test files in a gemspec. People do it but ... what's the point?

It's not uncommon when working on a gem for the first time to install the development dependencies and the best way they're articulated is via dev_deps.

Not true - the best way to install deps is by what's recommended in the docs. E.g. even Bundler (which can't use itself to install deps) has custom code (for rake spec:deps):

https://github.com/bundler/bundler/blob/f2c712b6f5a16ea8f81b432537d2852bbea2df67/Rakefile#L53-L75

And because it uses the gemspec (a mistake in my opinion), it breaks the DRY principle (that @drbrain talked about), because it has to remove ronn and rdiscount on JRuby. This is a clear example of why the gemspec is a bad place to store gem deps - except maybe for gems that don't have documentation to generate, etc..