Node API for compiler and dev server

[pcattori]

Relates to #4838

How things are today: "CLI as an API"

Today, users can only interface with the compiler and dev server via CLI commands:

• remix build : run the compiler, usually with NODE_ENV="production" for prod builds
• remix watch : run the compiler in watch mode AND run a Websocket server to broadcast change notifications to the browser for live reload
• remix dev : does what remix watch does plus also runs a dev server to handle requests

External dev servers (i.e. server.ts instead of running remix-serve build¹) often run two processes in dev mode²:

{
  "scripts": {
    "dev": "remix build && run-p \"dev:*\"",
    "dev:node": "cross-env NODE_ENV=development nodemon --require dotenv/config ./server.js --watch ./server.js",
    "dev:remix": "remix watch"
  }
}

The CLI (remix watch) is currently the only way to run (1) compiler watch mode and (2) Websocket server broadcasting from @remix-run/dev for an external server.

Issues with "CLI as an API"

The main limitation of this approach is that the live reload loop (from remix watch) runs in a separate process from the actual dev server in server.{js,ts}.

Remix users are painfully trying to work around this limitations in a couple places, notably:

• feat(cli): update a touchfile upon successful (re)build in watch mode #4307
• Purge cache on file changes on dev servers #4453

Unfortunately, this makes race conditions possible between (1) the Websocket server managed by remix watch sending out a broadcast for a change and (2) whatever mechanism is used to detect that a Remix (re)build happened in server.ts.³

How does remix dev do it correctly?

We don't run into this issue with the official Remix Dev Server (i.e. remix dev).
Why?
Because remix dev does not run the live reload loop in another process.
Instead it directly calls the compiler.watch function, so that it can use callbacks to be directly notified by the live reload loop when a (re)build occurs.

Proposal

To fix these issues, let's expose a Node API for the compiler and dev server.
That way, external servers can call compiler.watch or devServer.liveReload within server.ts to coordinate with the live reload loop, the same way remix dev does it.

Compiler API

export type BuildOptions = {
  mode: "development" | "production" | "test";
  target: "cloudflare-workers" | "node14";
  sourcemap: boolean;
  onWarning?: (message: string, key: string) => void;
  onBuildFailure?: (failure: Error | esbuild.BuildFailure) => void;
};

/**
 * Compiles the Remix app into the browser build (`public/build/`) and server build (`build/`)
 * Analogous to CLI command `remix build`
 */
export const build = async (config: RemixConfig, options: BuildOptions): Promise<void> => {/* ... */}

type WatchOptions = BuildOptions & {
  // a bunch of callbacks to coordinate with watch loop
  onRebuildStart?(): void;
  onRebuildFinish?(durationMs: number): void;
  onFileCreated?(file: string): void;
  onFileChanged?(file: string): void;
  onFileDeleted?(file: string): void;
  onInitialBuild?(): void;
};

/**
 * Watches the filesystem for changes to the Remix app and rebuilds when changes are detected.
 * Returns a promise for an async function to close the watcher.
 */
export const watch = async (config: RemixConfig, options: WatchOptions): Promise<() => Promise<void>> => {/* ... */}

Good news!
The compiler/ directory in remix-dev/ already exposes this exact API! ✅

Now we just need to expose that in remix-dev/index.ts:

export * as compiler from "./compiler"
// OR
export { build, watch } from "./compiler"

Dev server API

/**
 * Analogous to CLI command: `remix watch`
 */
export const liveReload = async (config: RemixConfig, options: WatchOptions): Promise<void> => {/* ... */}

/**
 * Analogous to CLI command: `remix dev`
 */
export const serve = async (config: RemixConfig, options: WatchOptions, port?: number) => {/* ... */}

Ideally, (1) the Websocket server code from the watch command and (2) the express dev server from the dev command should be moved to a devServer/ directory within remix-dev/ that exposes the proposed API.

Then the CLI can access these via:

import * as devServer from "../devServer"

export function watch = (/* all the same CLI args as before */) => {
  // parse command line args
  liveReload(/* ... */)
}

export function dev = (/* all the same CLI args as before */) => {
  // parse command line args
  serve(/* ... */)
}

I have an example implementation of this interface in the BYOC prototypes.
But its really just moving a bunch of dev server logic out of the CLI commands.ts file into a better home.

Update: PR #4528 implements liveReload and serve, so these would just need to be exported from devServer/ in remix-dev/index.ts

CLI

The CLI commands could be tightened to just be build and dev.

Note that the current remix watch command is confusing because it is not the analog to compiler.watch, but rather wraps compiler.watch with a Websocket server for broadcasting changes to the browser.

Footnotes

1. remix-serve build seems misleading as build command name implies that this command compiles the app, but in reality it merely serves the existing built app. I think a better name would be npx remix-server start or maybe npx remix serve (if we can call out to @remix-run/serve from within @remix-run/dev). ↩

2. Example taken from the express template ↩

3. You can see the author of Purge cache on file changes on dev servers #4453 try to hack around the race condition by introducing devServerBroadcastDelay: 100 ↩

[pcattori]

@jacob-ebey and I chatted about this and he suggested exposing a Middleware API for the dev server as well. Something like:

import express from "express"
import remixDevServerMiddleware from "@remix-run/dev"

let app = express()
app.use(remixDevServerMiddlware({
  config,
  liveReload, // boolean
  loadContext: {/* user defined */}
})