Reorganize Package Exports / Imports

[ryanflorence]

tl;dr

• We are creating undue friction without technical reasons with our package structure and exports
• We can simplify consumption of remix if we re-arrange some code and packages if:
  • packages only export what makes them unique (specific implementations)
  • packages do not re-export things that work universally
• Consumers just import it from the package where it originates
• We expect the consumption side to be easier to understand, match the actual dependency graph of our code, and remove the artificial friction for portable code

Prior art

• docs(decisions/0003): package for shared server features #3262
• 🗺 Improve how we expose and name types #4839
• Audit of server runtime pkgs: https://docs.google.com/spreadsheets/d/1VDwtasw8qntPPzJwk_NQLkMpz6Qn1wfIXwlisPC_fNk/edit#gid=0

Background

When we created Remix we had never seen anybody build a web framework that supported multiple JavaScript runtimes. To do it, we built our code around the Web Fetch API (:hat-tip: Cloudflare Workers) instead of node.js APIs

We had two goals:

1. Not make app devs think about this
2. Have types return the actual types of the polyfills for node.js, instead of the DOM types

We failed at (2). The types have always just been the DOM types. If I remember correctly, we anticipated needing to pass the polyfill into Remix runtime packages which influenced our current design. We ended up just adding the polyfills to global.* instead. I think this also influnced our decision to "import from the runtime" cause we didn't expect to have universally useful functions like json.

Magic Exports

For (1) we first did magic exports with some node_module shenanigans (detected the runtime in package.json, created fake node_modules/remix to import everything from). That turned out to be a bad idea and messed up yarn, pnpm, lerna, turborepo, kiwi (loljk) and friends.

Today: Adapter + React + Runtime

After bailing on magic exports, our goal was to minimize the number of @remix-run/* dependencies in package.json

Most app code would only need to import from two places:

• @remix-run/{rendering-lib} react, and future others
• @remix-run/{js-runtime} node, cloudflare, etc.

And then

• @remix-run/dev as a dev dependency
• @remix-run/{adapter} to set up a server

While you still have four dang packages, at least the code in app/ only imports from two.

While this has been fine (alright, it's been mid) it also introduces undue friction where there is technically no reason to have the friction. We also do extra work in Remix itself to add the friction!

We create our own demons.

Problems

Abstractions don't know where to import things

Imagine an auth abstraction that is published to npm or in a big project's monorepo that's shared across multiple Remix apps deployed to various places (one on node, one on oxygen, one on cloudflare, etc.).

Now imagine it simply wants to redirect. Where does the abstraction import redirect from?

import { redirect } from "@remix-run/???";
import { isAuthenticated } from "./utils";

export function getAuth(request:Request) {
  let auth = await isAuthenticated(request);
  if (!auth) {
    throw redirect("/login")
  }
  return auth;
}

There is no technical reason redirect comes from @remix-run/node or @remix-run/cloudflare, etc. The runtime is irrelevant, redirect is universally useful and built around Response.

Today, folks use server-runtime but that's undocumented, considered internal, and has got a bunch of abstractions for creating runtime packages that should be irrelevant to apps.

Multi-runtime apps

Jacob and others are deploying Remix apps that are multi-runtime! Some routes are handled by an edge runtime while others go to a node runtime. This is likely to become a common architecture.

In these apps the whole idea of "pick a runtime" doesn't even make sense anymore. Consider a simple cookie session:

import { createCookieSession } from "@remix-run/node";

Not only does this not make sense because cookie sessions have nothing to do with node or cloudflare, but it will break the routes that run at the edge runtime because @remix-run/node imports fs/promises.

The only option today is to import from the actual source of createCookieSession which is server-runtime.

I think we did this because we expected to be injecting the fetch API polyfill into the runtimes so you'd need to the import these things from the package with the right polyfill. But we put them on global.* instead, so there's no reason to import them from the runtime. It's accidental, artificial friction we added as we blazed this multi-runtime trail.

Documentation and Demos

While this is not the reason for this proposal, it is a symptom of the problem. Our docs do this everywhere:

import { json } from "@remix-run/node"; // or cloudflare, etc.

Re organizing our exports to simply export what is unique to the package would clean this up.

Types

Types are also strange:

import type { LinksFunction } from "@remix-run/node";

Why does that come from the server runtime? Meanwhile this function comes from the react package?

import type { ShouldRevalidateFunction } from "@remix-run/react";

LinksFunction is has nothing to do with the runtime, and ShouldRevalidateFunction has nothing to do with React.

These should both be coming from the same package as they are Remixisms, not react, node, or cloudflare.

Proposal

• Identify "Remixisms". Universally usable functions, types, etc.
  • Pedro already did this with his spreadsheet
• Move all Remixisms into one single package to be used by apps and third parties.
• In v2, stop re-exporting Remixisms from runtime packages.
• Ensure all package only export what makes them unique, no re-exports.
• Most of the code in this new package will come from "server-runtime" (since runtimes re-export a ton of things), but we'll keep server-runtime to continue to hold the "internal shared code" between runtimes (like createCookieFactory, etc.)

While we can bikeshed this, I think the best package name to use is the "remix" package.

Let's consider the use cases previously outlined, first is the authentication library that simply wants to redirect.

import { redirect } from "remix";
import { isAuthenticated } from "./utils";

export function getAuth(request:Request) {
  let auth = await isAuthenticated(request);
  if (!auth) {
    throw redirect("/login")
  }
  return auth;
}

It's obvious where to get redirect. The multi-runtime app (deployed to both an origin node server and an edge runtime) also has no issues with importing a Remixism:

import { createCookieSession } from "remix";

All of our docs can simply import from "remix":

import { json } from "remix";

And the types clean up too:

import type {
  LinksFunction,
  ShouldRevalidateFunction,
  MetaFunction,
} from "remix";

After Pedros audit, we found that only a handful of APIs aren't universally useful. So instead of us doing extra work with our packages and adding artificial friction by re-exporting everything from runtime packages, consumers can simply import those handful of APIs directly from the package that implements it.

// only works in node
import { createFileSessionStorage } from "@remix-run/node";

// only works in cloudflare
import { createKVSessionStorage } from "@remix-run/cloudflare";

No magic exports shenanigans, no package.json module/export field shenanigans, simply import the package that defined it, and put universally useful things into a package to be used universally.

Summary

Again, we had never seen a multi-runtime framework before, we did our best and it worked great! But now we know the constraints better and can simplify both internal and consuming code.

In the end a typical Remix app won't have more packages than it did before:

Most app code will continue to only need to import from two places:

• @remix-run/react
• remix

And then:

• @remix-run/dev as a dev dependency
• @remix-run/{adapter} to set up a server

If an app wants something specific to a runtime like createKVSessionStorage, they'll add it:

• @remix-run/cloudflare

I think this is totally worth it.

Upgrade path:

In v1 the runtime packages will re-export stuff from "remix", in v2 the won't. A codemod should be pretty easy too.

[sergiodxa]

Imagine an auth abstraction that is published to npm or in a big project's monorepo that's shared across multiple Remix apps deployed to various places (one on node, one on oxygen, one on cloudflare, etc.).

Now imagine it simply wants to redirect. Where does the abstraction import redirect from?

In Remix Auth I had to import from the unique shared package, server-runtime, same for Remix Utils

Also in Remix Utils Typed Cookie and Typed Session I had to wrap the Cookie or Session Storage objects because each runtime provide an implementation and server-runtime only the base

[kiliman]

Your meme game is on point! 😍

I worked around this import circus by creating a script that re-exports from the Remix packages. This way I simply import { whatever } from '~/remix'

https://github.com/kiliman/rmx-cli#-gen-remix

[cliffordfajardo]

This is going to make it even easier for folks coming into the web/js ecosystem for the first time for Remix to use it 💿
We've already had folks with no UI/frontend backgrounds at my job adopt and spin up Remix servers to build their own tools ✨ yay DX

[mjackson]

Right now the server runtime packages use a sort of classical inheritance model. Although packages don't actually "extend" one another, that's the current model. What you're proposing @ryanflorence is that we ditch inheritance for the composition model, which is also a valid way to do things.

I'd like to outline the two mental models and the trade-offs of switching to a different model.

Inheritance

In this model (the current model), server-runtime acts like a "base class" for node, deno, cloudflare, and potentially others (bun, fastly, any new JS runtime). node/deno/cloudflare/etc re-exporting everything from server-runtime fits this model; subclasses inherit all properties from their superclass and override them as needed. As an abstract superclass server-runtime is considered "private". Although it is useful for developing server runtime packages and 3rd party Remix-y libraries like remix-auth, it is never installed in an app's package.json.

Composition

In this model we expose basically everything in (or most of) server-runtime directly to users. In your proposal, this is theremix package. Apps install server-runtime as a dependency and grab stuff directly from there as needed, pulling from node/deno/cloudflare/etc for stuff that is specific to those platforms.

What you're proposing (e.g. import { json } from "remix") could be done today if people installed server-runtime directly and imported from there.

Trade-offs

The main advantage of the inheritance model is that you have one less dependency in the app's package.json. This is a BIG one, not because of the number of entries in dependencies (who cares) but because of how imports work in your app.

• You don't have to guess about whether to import from server-runtime or node. You know what platform you're building for, and you import everything from there.
• It's easy for deno/node/cloudflare/etc to "override" some method from server-runtime and expose their own platform-specific variant
• VSCode will not autocomplete imports from transitive dependencies, so it won't autocomplete from server-runtime.

Conversely, the main advantage of the composition model is that it's easier to write a portable import statement. This is useful mostly in writing docs. Porting your app to a completely different JS runtime (or "multi-runtime apps", as you called them) is an explicit non-goal.

In talking with @pcattori about this he noted that the composition model is less of a maintenance burden on us, since we don't have to remember to re-export everything from each of the concrete implementations. But we should be able to write a unit test for this to eliminate this burden.

It's important to note that in the composition model we lose all the benefits of the inheritance model, which is a pretty big loss IMO.

I'm doing my best to be as objective as possible here about the trade-offs. Please let me know if I've missed any.

[pcattori]

I agree with how you've defined "Inheritance" and "Composition" for packages, but I disagree with how the trade-offs are presented. I only see one potential trade-off loss if we abandon inheritance-style packaging, and in practice I think it's not even a loss at all. On the other hand, composition-style packages have clear trade-off wins that we know actually exist in theory and in practice.

---

You don't have to guess about whether to import from server-runtime or node. You know what platform you're building for, and you import everything from there.

👆 I see a couple reports pretty much everyday about people using @remix-run/node even though they are running in a Cloudflare environment, so I don't think "You know what platform you're building for" is necessarily accurate. I think a better framing is "You know that you are using Remix so you import things from remix (or @remix-run/server; naming tbd). If you want to use file session storage, you look that up in the docs and see that it's available either under node or deno and not cloudflare. You only have to start caring about your runtime when you actually need features from it. I think this greatly simplifies the mental model for users, especially beginners. Progressive disclosure makes for the best learning curve.

It's easy for deno/node/cloudflare/etc to "override" some method from server-runtime and expose their own platform-specific variant.

👆If there are things that need to be specialized to a specific runtime, that runtime can export a more specific version. For example, we want to take advantage of Node streams in json, @remix-run/node can export a Node-specific json. remix can export a Web-standards version of json:

import { json } from "@remix-run/server" // web standards version, works anywhere

import { json } from "@remix-run/node" // node streams version if you _really_ need it for some reason

I think needing runtime-specific variants will be either non-existent or exceedingly rare.

Even for the json and streams case I use as an example, I don't think this is an actual problem anyone has surfaced. The only confusion I hear about regarding streams is users unaware that their platform doesn't support streaming.

VSCode will not autocomplete imports from transitive dependencies, so it won't autocomplete from server-runtime.

👆Good to call this out as a misconception. Just want to emphasize that this isn't an advantage of the inheritance model, since both models behave the same with regard to autoimports. They both autoimport correctly.

Conversely, the main advantage of the composition model is that it's easier to write a portable import statement. This is useful mostly in writing docs. Porting your app to a completely different JS runtime (or "multi-runtime apps", as you called them) is an explicit non-goal.

👆I think the main advantage to portable import statements isn't writing docs, its in understanding how Remix works. I think importing from remix or @remix-run/server creates a better affordance that conveys that something isn't runtime-specific. I think it helps the user (esp. beginners) create an accurate mental model, without magic or indirection, that helps them do the right thing with Remix. Docs being simpler is a side-effect of this approach.

Package inheritance costs innovation tokens

I'd like to argue that "importing from where something is defined" is the vanilla, default, no-op way to do packages. The way I see it, we are explicitly choosing to do something non-standard by adopting the inheritance model and this is evident in that we need to do extra work (e.g. maintain re-exports and disambiguate imports in docs) to support it.

So in my view, I use 0 innovation tokens in the composition approach. That's just how packages are normally. Whereas the inheritance approach does use an innovation token and extorts a maintenance cost.

Trade-offs as I see them

To me, the only thing we lose when abandoning the inheritance approach is automatically giving users the most optimized version of an overloaded function like json.

import { json } from "@remix-run/node"
// ^ inheritance model: I always import from my runtime package and get the most optimized thing

// but how do I get access to the web-standards version of `json` if I want it?

In theory, that sounds good. But again, I don't think this rare; the only example I can think of is Node streams and I don't have evidence that anyone needs json to use Node streams and that web-standard streams are insufficient. And the people who care about this would know enough to opt-in to the Node-specific json function in the composition paradigm:

import { json } from "@remix-run/server"
// ^ I start by using the web-standards version of things
// I think most/all usage of `json` will be like this

import { json } from "@remix-run/node"
// ^ I'm really in the weeds on streams and know that I want Node-streams
// still possible and ergonomic with composition model
// not automatic, but rather explicit

Plus what if I'm on Node but I really want to use web-standard streams? With the inheritance model, there isn't a good place to grab a web-standards version of json. The "overload" nature of inheritance works against that.

Plus there's ongoing work to optimize web-standard streaming.

tl;dr: I can't think of one good example where we would get benefits from inheritance-style packaging. Maybe streams, but I'm not convinced. But we are using innovation tokens to do inheritance-style packaging and paying real costs for it, both in terms of making usage more confusing and also making maintenance more error-prone.

[mjackson]

In Remix Auth I had to import from the unique shared package, server-runtime, same for Remix Utils

@sergiodxa I think this is ok. We should recommend to 3rd party Remix-y libs to import directly from server-runtime as needed.