Feat: could roku_modules be installed as symlinks?

[elliot-nelson]

Context

We are working through a proposal on what a Roku application monorepo could look like, built on rokucommunity tooling, and using Rush as the build orchestrator / monorepo tooling. Rush is built around TypeScript, and since many of the rokucommunity tools have similar goals to the TS ecosystem, a lot of stuff "just works".

We rely on PNPM to install packages and provide workspaces, which is how a Roku app would "depend on" a Roku library in the same repo (using the workspace:* dependency notation). I've prepared an example commit to show how this looks, using bsconfig file mapping and just swapping from path references to a workspace reference.

We would like to take this a step further: the above approach works great, but every time you add library x to your dependency list, you need to go and manually add multiple lines of x file mappings to your bsconfig. It looks like ropm would be one way to remove that requirement.

The problem

The issue is that ropm doesn't work quite like we'd want it to in this case, and I'm interested in options.

At first, I was thinking I just needed to change ropm to symlink instead of copying:

src/source/roku_modules/foo/     -> ../node_modules/foo/source/
src/components/roku_modules/foo/ -> ../node_modules/foo/components/
src/images/roku_modules/foo/     -> ../node_modules/foo/images/

Then I realized this doesn't quite work because an ropm install isn't just an ropm install, it's actually modifying files as it copies. So it's not really something we could add to the "rush install" step in our repo. Ideally, these files would never be in the source folders, and would get written into the destination at build time just like the rest of the bsconfig-mapped files.

I'm opening up this ticket without having a clear picture of exactly what I want yet -- maybe a better idea on how people are using the tool? I suppose we could make it part of the build script, e.g.:

{
  "scripts": {
    "build": "ropm install && bsc"
  }
}

This seems a bit messy because you are overwriting a bunch of files in src/ every time you run, but if all of those files are gitignore'd, maybe that's OK. It would give us the primary goal: we could depend on library foo, and then those files would automatically be available in the usual roku_modules/ folders without any extra file mappings.

Anyone toyed around with similar ideas?

[philippe-wm]

The thing with ropm is that you should assume that you are linking transpiled and namespaced (with a configurable prefix) source.

In a monorepo you might choose to have a standard namespace, and emit these files in a lib folder (to use Rush terminology) of the library project, then you could symlink this lib folder around under the (gitignored) roku_modules folder.

So overall it's probably feasible but that would require to have a very different ropm mode: with one command for namespacing sources (which would be part of the library's build task, after a potential brighterscript transpilation), and one command for creating the symlinks.

[elliot-nelson]

Thanks @philippe-wm!

As I experiment more with this, I think treating ropm strictly as a build step makes more and more sense to me... the command ropm copy && bsc basically does what I want.

The only issue with this command is that attempts to run:

npm ls --parseable --prod --depth=Infinity 

This blows up because it doesn't understand pnpm's symlinks, but if I can replace it with:

rush-pnpm ls --parseable --prod --depth=Infinity

Then I think the command would work fine.

Perhaps all I need is a way to configure what base package manager to run.

[philippe-wm]

What would possible be an interesting ropm feature would be to be able to use it (through JS API) to process an arbitrary source location into namespaced code in an arbitrary target location.

[elliot-nelson]

That would work!

I think we have two processes we'd like to manage:

One: the library build

In /libraries/acme-lib, do a "ropm build" which namespaces everything in /src into /lib folder.

So you end up with a fully namespaced and ready to use:

libraries/
  acme-lib/
    lib/
      source/
      components/
      etc/

Two: the application build

Here, if the lib folder is already set up, we only need to symlink:

apps/acme-app/source/roku_modules/foo -> ../node_modules/acme-lib/lib
... (and so on) ...

Ideally these two steps could be executions like ropm build --library and ropm link --app (or something), but if there was a JS API we could write our own entry points.

[TwitchBronBron]

I've been thinking about this recently. The BrighterScript file API will enable a lot of additional plugin functionality. I was considering writing a ropm plugin, so that when running a brighterscript language server or build, the ropm plugin would actually read the files and translate them in-memory. This could also then work for your in-active-development ropm packages as well, and would mean the ropm modules would never be copied into your source code.

The file api isn't quite ready yet, but do you think this theoretical approach would solve all the above issues?

[philippe-wm]

@TwitchBronBron does the file API run early enough to provide a whole library (and potentially transitive dependencies)? Can diagnostics, code navigation,... point to the original source file? (or rather be omitted as we'll probably want to lint them individually)

[TwitchBronBron]

Yes to all of those things. :) You can read the proposed docs in the file api PR here, but to summarize, brighterscript now officially supports having plugins contribute files at various stages of the lifecycle.

Diagnostics need to be tied to a "File" object in the program, so as long as the files were added to the program, the srcPath can be the original file path from node_modules and the pkgPath & destPath can point to the desired location in the app.