[FR]: link outside WORKSPACE for dependencies during local development (~= `pnpm link`)

[sjbarag]

What is the current behavior?

Currently, rules_js fetches packages from either:

1. A remote location (https://registry.nodejs.org/..., git://github.com/...) with the Bazel downloader
2. The user workspace (//pkg/left-pad), which requires the package to be pulled into the workspace and its dependencies readable by npm_translate_lock.

Attempts to link outside the Bazel workspace to an NPM package located elsewhere on-disk get rejected with
invalid link_package outside of the WORKSPACE.

Describe the feature

The npm link workflow is pretty common in the node community, as it allows a dev to modify a third-party package and use it without having to vendor that package.

As a contrived example motivation (replace left-pad with any other package):

1. I've found a pathological case in left-pad performance that my application highlights
2. I'd like to contribute a fix back to upstream left-pad. While working on that fix, I want to use the locally-changed copy in my application as a sort of "real world" gut-check in addition to the left-pad test suite. Just to make sure my pathological case is indeed getting fixed.
3. I don't want to vendor left-pad for any number of reasons:
   1. left-pad has a complicated/obscure build system that makes it non-trivial to vendor and build with rules_js.
   2. left-pad takes bugfixes incredibly fast, so my vendored copy would only exist for a few hours after my PR goes out.

In the current state, there appears to be no way to build left-pad in a separate directory and use it in my rules_js-built package(s).

With vanilla pnpm link (or npm link or …), the workflow is roughly:

1. pnpm --dir ~/bitbucket.org/my-org/my-app link ~/github.com/left-pad/left-pad to use a custom left-pad
2. Change and rebuild left-pad.
3. Change and rebuild my-app. It includes my left-pad changes.

Ideally, the workflow would be pretty similar under Bazel + rules_js. For local development it feels reasonable to break hermeticity, akin to npm_translate_lock's current use_home_npmrc), but I'd be happy to add an extra flag or two to achieve this.

---

Relevant slack conversation: https://bazelbuild.slack.com/archives/CEZUUKQ6P/p1689358470077099

[sjbarag]

For a non-contrived motivation, included here to avoid cluttering the main description:

1. My large Bazel-built open-source application $app has a major release every six months.
2. That application shares UI components with a closed-source hosted webapp.
3. That hosted webapp supports multiple major-versions of the OSS $app. I'd like those shared UI components to be versioned independently of the OSS $app, such that the shared UI components themselves support multiple major versions of $app.
4. Keeping those shared components in a separate repo with separate history feels the most natural to achieve that, but makes local development of $app and the shared UI components very difficult.

[twheys]

If you link a package link pnpm link lodash, the dependencies in the pnpm.lock file get modified to something like

      lodash:
        specifier: 4.17.4
        version: link:node_modules/lodash

However, rules_js doesn't like this and errors out

ERROR: .../BUILD.bazel:6:22: no such package 'node_modules/lodash': Package is considered deleted due to --deleted_packages and referenced by '//:.aspect_rules_js/node_modules/lodash@0.0.0'

Seems like this could work hermetically since linking changes the lock file. Would be really nice if we could get support for this feature, since it's a very common workflow for developers who maintain 3rd party packages in their workspaces. I'd be happy to make a PR if someone from aspect could give me a little guidance of how this might be implemented. Please let me know.

[alexeagle]

I think it makes sense to allow you to do this.

We've also discussed a bazel wrapper (like Aspect CLI) could give an easier way to apply patches to third-party libraries - imagine you have edits in anything that bazel downloader fetches and could easily go from the repo where you made the edits to a patch file Bazel applies. (issue 828 in our internal monorepo)