Branch Bundles: better serverless support and infinite runtime scale

[ryanflorence]

tl;dr

• Remix apps on serverless are "fat lambdas" (arc, aws, netlify, vercel, cloudflare, etc.)
• With some bundler work we can deploy "skinny lambdas" and get better serverless support

Background

The ethos of serverless is to have small functions, not the whole app, on each function. This is good for UX to speed up cold boots, good for developers for debugging (logs are function specific, etc.), and resilience (one function might be busted but the others aren't). There are a bunch of other reasons too but that's all barely inside my wheelhouse, so I don't have much more to say other than that.

Point is, we could support serverless much better than we do today with Branch Bundles.

Proposal

remix build --branch

module.exports = {
  branchServerBundles: true,
};

Instead of creating a single server bundle, this would generate a server bundle for every branch of the route tree. Each of these branches could then be deployed to a separate lambda.

Consider the following routes:

root.tsx
routes/
  _index.tsx
  about.tsx
  products.$pid.tsx
  account.tsx
  account._index.tsx
  account.orders.tsx
  account.orders._index.tsx
  account.orders.$oid.tsx

This would result in the following server bundles:

build/
  root--_index.js
  root--about.js
  root--products--$pid.js
  root--account.js
  root--account--_index.js
  root--account--orders.js
  root--account--orders-_index.js
  root--account--orders-$oid.js

The name is irrelevant (though we might want a convention to make it easy to set up your infrastructure), it's just to illustrate the code contained in each bundle. While it looks like they're just the bundle for every route, they will actually contain the code of the parent routes as well. For example, root--account--orders--$oid.js would have all four routes bundled into it. Similarly, root--account--order.js would be identical except it would not include $oid.js.

Each bundle needs the parent routes because all parent routes are needed to render a full HTML document.

Once these bundles are created, you'd need to set up your infrastructure to deploy each bundle to its own lambda and serve the right lambda for different URLs to the app (I've done this with arc and it's easy but verbose, for vercel we'd probably build into their file system .build API, not sure what netlify looks like).

Since each bundle also needs the server module that sets up an adapter, we'd probably want to make that part of the bundling process, too.

Pretty much infinite runtime scale here with this and serverless. No route branch will ever get tremendously big because they are naturally restricted to the cognitive processing of a human looking at the screen. While apps can grow enormous in breadth, routes have a natural depth size limit because it represents a single UI the user is looking at.

I know the Vercel team is very interested in this, maybe we can get their help with the work.

[kasperpeulen]

While apps can grow enormous in breadth, routes have a natural depth size limit because it represents a single UI the user is looking at.

What if you want to make a web app like this one?

I think this is where nested routing becomes too limiting to express the current location of your app. In nested routing you alway have one active child per parent: /parent/child/grand-child/grand-grand-child

In this vscode example, you really have like 10 isolated apps, within one screen. I think it should be possible to describe the state of all those little isolated apps without limits, to really scale this idea, but also remix in general.

For example a syntax like this:

/parent1/child1/grand-child1&parent2/child2/grand-child2&parent3/child3&parent4/child4/grand-child4

[sergiodxa]

I would use search params instead of nested rotues, the user is on /source-control?commitGraph&panel=gitlents-commits&commit=SHA

So the route is source-control, the rest are controlled with search params. There can also be a parent route for rendering the left sidebar with the icons for the navigation.

[ryanflorence]

@kasperpeulen feels off topic to me since vscode in the browser is a ton of code in the browser, not the server that sent it. I'd never try to server render something like this.

[kasperpeulen]

@ryanflorence But a lot of apps are in the browser now right, and could benefit from Remix.

I understand that vscode might a bit far fetched, but say you have a dashboard that shows all kind of metadata from a github repository in customizable panels. This dashboard let's the user easily keep in sync with what is new in the repo. (Just saying something). That could also look very similar like this:

You would not make this app in Remix? But just fetch all data from the frontend? And except all waterfalls/loaders and less than ideal performance?

[tom-sherman]

This is... interesting. Cloudflare worker specific thoughts below :-D

If you speak to most people deploying cloudflare workers they'll recommend combining many concerns into one worker ie. a nanoservice. This is beneficial for performance in that environment.

This proposal does solve the issue of max worker size in this environment, which is good! But in order to get the maximum benefits these sub-workers should be invoked as services.

This is a little difficult, but not impossible with workers, Pages would require a bit more thinking I believe as currently it's assumed that a remix app is a single pages function.

[tom-sherman]

In the end, it's a tradeoff though. Between cold start, warm start, and worker size. Ideally I would like to be able to apply any splitting policy as is fit for my use case. For example I would like to combine all of my logged out routes into one worker, and all of my logged in routes into another.

[tom-sherman]

But in order to get the maximum benefits these sub-workers should be invoked as services.

Oh, I think i've misunderstood how this proposal works. We're not talking about executing child workers here, each route group is deployed to a different worker!

I think what I assumed this proposal was about should be considered though, it gets the best of both worlds.

[tom-sherman]

I have a concern about deploying updates. Deploying multiple lambdas/workers at the same time, in a transactional manner, sounds like a very tricky thing indeed.

There is going to need to be some top-level lambda/worker or some CDN routing to switch between the old and new deployments I believe. This is verging on blue/green deployment territory. Introduces a lot of extra complexity to deployments in these environments.

This is more of a comment for the teaching of this option, I think, as this kind of complexity is going to need to be explained in the docs and guides.

[shamsup]

This is a very interesting proposal, but I have some thoughts as an AWS Lambda user.

First of all, I am not super opposed to the fat lambda that gets deployed currently. It is a bit at odds with the least-privileged security model, but generally speaking, it keeps access patterns predictable to optimize provisioning for cold starts. I think a more workable alternative to this extreme is being able to opt into a separate bundle for a specific route, which should make sense after considering these points:

Concerns

1. Cold starts: Separate functions for each route combination makes it almost guaranteed that users hitting a page that is infrequently used will experience a cold start. Even navigating from a parent to a child route could trigger another cold-start. This problem doesn't exist with the fat function because your entire user base is hitting the same pool of functions for the whole app. Any page will perform just as well as any other page, with some users possibly experiencing a cold start randomly here and there.

2. As @tom-sherman noted - coordinating deployments: A change to a piece of code in one parent route could functionally change many Lambdas. With IAC tools like CloudFormation or the asynchronous nature of workers pushing out to all edge locations on Cloudflare, there's a scenario where any given function could be out of sync with the other functions and the UI code served to the user. There would need to be a clear path to solve this (ie how to implement blue-green or a pre-built deployment pipeline).

3. Deployment sizes: While the functions may be smaller, duplicating route code across many functions (think the code+dependencies for a parent route) will significantly increase the overall size of deployments. Instead of a single 2-3MB bundle in our case, we could have 15 (actually many more in our case, ~75 with this model) 0-1MB bundles. In reality, most of our routes are sharing the same dependencies and could be deployed together (same DB models/client, same auth services, same date libraries, etc), the exception being that route that is generating a PDF or resizing the profile image when it's uploaded. It seems weird to deploy 15MB of code instead of 3MB, just to save 2MB for most executions. Extra code increases cold start times, but cold starts should only really be a problem using this deployment model.

Possible Benefits

1. Least-priveleged security model: Routes will only need permissions to resources that they use (or their parent routes use, which is kinda weird and maybe bad). Currently any route could access the resources that should maybe be protected by any other route, like the entire application requires my Stripe live key instead of only the function that handles that route.

2. Small deployments per-function: I know that Cloudflare Workers are pretty limited in the size of the code you can deploy to them, so this would allow most routes/apps to easily fit within Workers. Cold starts in Lambda would be generally faster with less code as well.

3. Resource tuning: Most routes shouldn't require many resources and tuning most routes separately from that route with the PDF generation or image processing could come with cost savings. This is not our experience, since network is the primary bottleneck in our main routes, while CPU is the bottleneck for that route, so we need to provision to meet our network needs in the primary routes anyway.

In summary, I think this approach adds a lot of complexity with very few possible benefits. I think a Happy Place for this idea could be opting for specific routes to get their own bundle instead of every route. This significantly reduces the complexity while still gaining the benefits in the majority of cases. It's much easier to decide that the PDF generating route should be tuned separately from the rest of the app, or the payment processing route can have secrets separate from the rest of the app, or the image processing library can be loaded for a specific route but excluded from the bundle for the rest of the app.

[ryanflorence]

I think a Happy Place for this idea could be opting for specific routes to get their own bundle instead of every route.

That's a great idea. Branch every route, just specific routes, or none.

[tom-sherman]

When I use arc or vercel or netlify or firebase functions the deployment coordination problem is completely handled.

I can imagine Arc working, I think they swap API gateway versions under the hood for you - essentially blue/green deployment.

Similar with Netlify/firebase, they work on a git push so I imagine their abstraction can update the lambdas transactionally.

Cloudflare Workers on the other hand requires a separate API call to update each worker, and the rollout of them to all edge locations is done in parallel - they all reach the PoPs at different times. Cloudflare Functions may be a different story, it would be good to check if this is the case there too.

[tom-sherman]

Folks in the cloudflare discord informed me that Cloudflare Pages functions are all bundled into the same worker, so this proposal couldn't help folks deploying to that environment.

Combined with the comment above (which rules Workers out as well, without additional steps) I don't think this proposal can really help folks deploying to Cloudflare at all.

[tom-sherman]

Another issue, even if this did split, is that Cloudflare has a low limit on the number of workers you can have in your account (max 500 on the paid plan).

[ryanflorence]

I don't think this proposal can really help folks deploying to Cloudflare at all

The capability of splitting up the bundles is not the same concern as a deployment target supporting it easily.

[gut4]

If it will be possible to branch specific routes it will be amazing.
Also it may be helpful for complex apps that require Service Worker
for specific routes (https://twitter.com/ryanflorence/status/1603399586764455936)

[TooTallNate]

The branchServerBundles: true proposal is great to see! However, I think it's a little too "prescriptive". I'd like to propose an alternative syntax and behavior. In short, I'd like to be able to define multiple server bundles and declare which routes will be compiled in to those bundles.

I'm calling it serverBundles as a new config option in remix.config.js:

  serverBundles: [
    { serverBuildPath: 'build/build1.js', routes: ['routes/index', 'routes/about'] },
    { serverBuildPath: 'build/build2.js', routes: ['routes/generate', 'routes/error'] }
  ],

remix build would take this configuration and generate one output file per bundle entry, which would contain only the specified routes.

This is actually more low-level than branchServerBundles, because you could achieve the same result by specifying one route per bundle, but this proposal is also more flexible because it gives the invoker the ability to keep multiple routes in a single bundle. For example, at Vercel, we might want to initially utilize this feature by producing two "fat" bundles - one for Node.js Serverless and one for Edge Runtime, and the bundles would only contain the routes that have been configured to execute on those specified runtimes. Later down the line we may want to add more granularly split bundles to address function size limits and enable infinite scalability.

Let me know what you think, I hope you like 😄

[TooTallNate]

I was actually able to implement this rather quickly, so here's a PR if you like the direction of this proposal:

#5479

[SpencerDuball]

I didn’t notice, but does your PR include wildcard matching as well by chance? Would be nice if you could effectively breakup all routes into one output bundle with some sort of pattern matching.

[tom-sherman]

@SpencerDuball #5479 (comment)

[TooTallNate]

Wildcard isn't currently supported, but would be easy to add, possibly using minimatch.

[barbinbrad]

Personally I prefer this direction. Thanks for working on this.

[izznat]

From reading the concerns mentioned above, it seems edge platforms that support safe multi functions deployment will benefit the most from this feature. Such as Vercel Edge Functions and Netlify Edge Functions. The reasons are:

1. They don't have cold start problem.
2. They have much smaller limit of script size compared to NodeJS based serverless function.
3. They handle multi edge functions deployment safely as long that they are located in the same project.

Therefore, I want to propose additional changes to better fulfill the requirements of those platforms. I will use Vercel Edge Function with Vercel Build Output API v3 as the imaginary constraints to demonstrate the proposal in action.

Taken from Vercel Build Output API docs for edge functions, below is the required file system structure:

1. Functions must be placed inside .vercel/output/functions directory.
2. Each function is represented by a directory that ends with .func in its name.
3. .func directory must have two files:
   • Entry-point script that contains the function code itself.
   • vc-config.json to configure the edge function
4. Entry-point script may imports .js, .json, .wasm file that will be bundled at deploy time.

We need a way to customize the placement of each server bundle and also emit other files that are required by the platform such as vc-config.json. There are two solutions that I could think of:

1. We will add a new option called serverBuildSideEffect that accept function with route entry object as the the argument and return nothing. It will be run after every bundle builds. Inside the function we will emit other required files such as vc-config.json. Then extends serverBuildPath to also accept function that also accepts the route entry object but must return the build path for that route. Maybe we should set serverBuildSideEffect and serverBuildPath function to only run once by providing all the the builds route entry object combined as a single object as the argument.

2. We do the side effect of emitting other files inside serverBuildPath function. But, it kinds of strange that serverBuildPath function also emit files instead of only returning the bundle path.

Personally, I prefer the first solution because it is clear what to expect from the option name. Also, for consistency sake, I think branchServerBundles should be renamed as branchServerBuild.

[tom-sherman]

emit other files that are required by the platform

Could this not be done as a step after the remix build, instead of being built into remix itself?

Eg. You build the functions to the relevant folders, you then loop over the folders and insert an appropriate vc-config.json.

Remix could emit a manifest to help with iterating over these outputs.

[WesleyYue]

Could this not be done as a step after the remix build, instead of being built into remix itself?

I think Vercel is doing exactly that already vercel/vercel#9504

[izznat]

emit other files that are required by the platform

Could this not be done as a step after the remix build, instead of being built into remix itself?

Eg. You build the functions to the relevant folders, you then loop over the folders and insert an appropriate vc-config.json.

Remix could emit a manifest to help with iterating over these outputs.

Yes we can. But, now we need to maintain another script. I expect the platform adapter package to include their ownserverBuildSideEffect function and serverBuildPath function. Then we just need to import and use them in remix.config.js file. Maybe named something like createServerBuildSideEffect and createServerBuildPathResolver.

[mmachatschek]

This proposal would be very appreciated. I really like the idea of @TooTallNate #5405 (comment) so that multiple routes can be bundled together or splitted out into separate files.

having a mix of serverless node functions and edge functions on vercel would greatly benefit from that. Edge functions have no cold start so they can be split into separate functions. Serverless functions suffer from cold starts so they can be bundled together

Edit: + we have a working POC of this made by @TooTallNate

[clovis1122]

I'm very interested in this functionality. Additionally, I wonder how hard it is to implement in userland. I've seen the routes split when deploying to Vercel but no idea how they do that.

[mmachatschek]

This seems to be related to React Router Roadmap