Pre-RFC: Global Package Interop

[charlespierce]

Starting this as an issue, since there are a number of questions and I'm not sure of a full approach at the moment.

Motivation

We have a number of not-quite-related issues around the interaction of global package installs that break Volta's transparency by not working as users expect. Some of these include:

• Cannot find module when sub-package depends on the root package volta#762 - Dependency can't find parent project because of directory structure
• ts-node peerDependancy not found volta#652 - One global tool can't find another even when both are installed
• Support for non-binary global installs volta#555 - Access to global libraries through NODE_PATH
• Leave Shims on the PATH when executing a tool volta#492 - Global tools calling other tools

These issues, while not extremely common, are very confusing for users. There also aren't any good workarounds at the moment to make them even sort-of work, so users who run into them are generally stuck and forced to disable or otherwise go around Volta. Additionally, our interception of npm i -g and yarn global add is a confusing situation, especially when people are following the instructions all over the internet, which instruct them to use npm i -g or similar to install a tool.

The goal of Volta is to be transparent, so we need to work out a model that handles global installs the way users expect while maintaining the UX improvements of Volta.

Existing Solutions

Existing node version managers, such as nvm or asdf-nodejs handle global installs by using the existing npm infrastructure, which installs into the node directory. The result of this is that whenever the users install a new Node version, they also have to re-install all of their global tools for the new node version.

One of the features of Volta is that we associate each tool with a Node version, so that we can ensure that they continue to work, even when users switch default Node versions (or work in a project that has a different Node version set).

Details

Constraints

• Global packages should have an associated Node version We already do this, and it should be preserved. Associating a Node version with a global package allows us to set up the environment correctly when calling that package, regardless of whether or not the user is still on the same version of Node they were when they installed it. This is one of the major benefits that Volta provides, so we should preserve that behavior.
• Global binaries should be able to call other global binaries With a "standard" npm or Yarn setup, all of the tools that are globally installed are put onto the PATH directly, so they can all call each other without issue. We add the associated Node version to global tools, so we likely need to go through the shim logic, but that shouldn't stop tools from being able to call other tools (or themselves recursively).
• Global libraries should be allowed While not extremely useful, since they aren't generally accessible from arbitrary scripts, we should nevertheless allow users to install global packages that don't have any binaries. This is important for things like peer dependencies and yeoman generators.
• Volta should provide a way to determine the path to a global package Similar to how we have volta which to "unwrap" the Volta shims, we should also provide some command that will locate a globally installed package and return the path to that package. This will allow users to pass that path to NODE_PATH as needed.
• Global packages should be able to require globally installed libraries npm installs all global packages under a single node_modules directory, so when calling require from one package, it automatically can see the other packages that were installed globally. We should preserve this behavior as much as we can; the main difficulty I see here is native modules and differing associated Node versions.
• Global packages should leverage Node module resolution Our directory structure currently doesn't have any folders named node_modules, so the Node module resolution algorithm is completely unable to detect our packages, even from within a package's dependencies. There shouldn't be a need to completely reinvent the wheel here, we should do our best to leverage the existing resolution algorithm.

Possible Solutions

I don't have a fully-formed solution, but my current thinking is around leveraging an actual npm i -g command with the --prefix option to direct the install into a specific directory. That way we get the node_modules directory automatically, to take as much advantage of the Node module resolution algorithm as we can.

If it weren't for the existence of native modules, we could easily point all of the global installs into a single directory and then all of our problems would be solved. Unfortunately, since we need to work across varying Node versions, native node modules prevent us from using a single repository for all of the packages.

If we keep each package in its own silo, to preserve the associated Node version, perhaps we should provide the ability to install multiple packages as a batch. That way we could be sure that they could access one another and that they have a consistent associated Node version.

[mikrostew]

If we keep each package in its own silo, to preserve the associated Node version, perhaps we should provide the ability to install multiple packages as a batch. That way we could be sure that they could access one another and that they have a consistent associated Node version.

I don't have a fully thought-out solution for this whole issue either, but something like this ^^ is the key to getting globals to work together well. We currently treat all global installs the same way, but maybe we need to make a distinction between how we handle global binaries vs. global libraries?

Global binaries: Users expect to install these packages in a standalone way, and call the associated executables from the command line or a script. This is what we currently support.

Global libraries: Users expect to install these along with some already-installed global binary (like yeoman generators), so that the global binary can find the library. Right now, if a package doesn't have any executables, this fails.

I'm not sure that there is a super clear-cut distinction though, because some global "libraries" also contain their own binaries (like ts-node).

I don't know what the best syntax would be, but like you say it would be nice to have a way to install global libraries and binaries together as one unit, with the same Node version, something like volta install --group ts-node typescript. And it would be even better if we could detect when the user volta installs a new version of any of the packages in a group, with a new Node version, and upgrade everything in the group together.

One complication is that if we want to intercept npm i -g-type commands, to seamlessly install things with Volta, we may need to prompt the user about installing in a group with an already-installed package. That breaks the seamless illusion, and could be confusing.

[dherman]

I really like the direction this is going—I think you're onto something here! I especially appreciate the problem statement about how confusing and off-putting it is to users when we break the documented behavior of other tooling (npm i -g, Yeomen generators, global binaries that call each other).

At a high level, I think the value add Volta is trying to offer for globals is that you don't have to think about the matrix of which globals are installed for which Node versions. But your new idea here is that we can achieve that by installing the same set of globals for all versions of Node you ever install. That's a slightly lower-level explanation, and not necessarily one you have to explain to every user, but it's a very easy way to explain to people who have come to understand the way Node and npm installations work.

One idea that came up on a video-chat today: as long as we keep track of all the globals you have installed, we can just-in-time install them all every time you change your default Node version. That way even global libraries without binaries work (as opposed to doing the just-in-time installation when you call a global binary).

Also I wanted to say, I think the idea of hooking commands like npm i -g to add a --prefix (and track some extra Volta metadata on the side) is a pretty lightweight modification, and doesn't really fundamentally change the behavior of npm i -g. In particular it still respects the underlying directory layout and inner-workings of a Node/npm installation.