feat: split up API Routes + use .nft.json files to make builds fast (#2058)

* feat: split up API Routes

Brings back the functionality that was reverted in
#1731, but under a flag.
This will be utterly slow in building,
so let's try to speed that up in the next step!

* feat: load includedFiles for every page

* refactor: extract function config logic

* refactor: extract flag into own definition

* feat: use "none" bundler for split-up api routes

* feat: list some more dependencies

* feat: use NFT to trace common required files

* refactor: clean up a wee bit

* fix: please don't include /sh

* feat: enable flag by default, so tests use it

* feat: add a naïve packing algo

* feat: write rough sketch for packing lambdas

* refactor: add constructor for APILambda

* feat: pack handlers together into bundles

* fix: linter

* feat: exclude some heavy unneeded files

* fix: trigger CI again, now that it supports `none` bundler

* feat: remove code for old mechanism

If we'll be doing a staged rollout,
and our test suite covers most of the cases,
we should be able to roll this out
without a self-serve opt-out mechanism.

* fix: remove test for deleted code

* fix: ensure that react doesn't try to load development build

* fix: move test directory into repo, so node_modules can be read

* fix: snapshot with API redirects

* fix: remove .only

* fix: don't assert on _api_*

* fix: another test

* fix: apply NODE_ENV=prod to right generated handler

* feat: remove nft tracing

* feat: put change behind flag again

* feat: source flag from plugin input

* fix: add default value for featureflags

* fix: default flag to true for testing

* fix: revert changes to lockfiles

* fix: eslint it/test

* fix: lint

* fix: revert distracting change

* fix: now that we don't use nft anymore, we don't have to copy over node_modules

* fix: swallow require.resolve errors for unit tests

* fix: remove timing logs

* fix: lint name

* fix: lint

* fix: isr needs _document.js

* fix: add _app for ISR

* fix: correct wrong output of some npm versions

* fix: integration test

For some reason,
ZISI doesn't like relative paths in this integration test.
We can fix it by using absolute paths.

Since this PR moves API routes out of __netlify_handler,
we can no longer make the assertion on x-nf-render-mode.

* fix: assemble npm package path correctly, also for monorepos

* fix: try what happens if we use next-netlify server

* fix: check what happens when we skip revalidation

* fix: dont let revalidate request time out

* fix: send x-nextjs-cache header to prevent error

* fix: update error message in test

* fix: resolve relative paths based on data in required-server-files

* fix: test

* fix: keep manually-added `included_files`

* fix: try something

* fix: don't include unneeded _app pages in ISR

* fix: finalize includedFiles before writing it onto netlifyConfig

* chore: update comment

* fix: exclude sass file in monorepos

* Update packages/runtime/src/helpers/functions.ts

* chore: remove comment

* fix: update flag impl

* refactor: use getRequiredServerFiles

* chore: add comment on route[0]

* fix: set NEXT_SPLIT_API_ROUTES in netlify.toml

* fix: put updated revalidate behaviour behind flag

* fix: supply splitApiRoutes in getHandler

* fix: better run your code before committing it and embarrassing yourself

---------

Co-authored-by: Nick Taylor <[email protected]>

Author: Skn0tt

.github/workflows/e2e-appdir.yml

@@ -6,6 +6,9 @@ on:
   push:
     branches: [main]
 
+env:
+  NEXT_SPLIT_API_ROUTES: true
+
 jobs:
   setup:
     runs-on: ubuntu-latest

.github/workflows/e2e-next.yml

@@ -8,6 +8,9 @@ on:
   schedule:
     - cron: '0 0 * * *'
 
+env:
+  NEXT_SPLIT_API_ROUTES: true
+
 jobs:
   setup:
     runs-on: ubuntu-latest

.github/workflows/test-integration.yml

@@ -9,6 +9,9 @@ concurrency:
   group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }}
   cancel-in-progress: true
 
+env:
+  NEXT_SPLIT_API_ROUTES: true
+
 jobs:
   build:
     name: Integration tests

.github/workflows/test.yml

@@ -11,6 +11,9 @@ concurrency:
   group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }}
   cancel-in-progress: true
 
+env:
+  NEXT_SPLIT_API_ROUTES: true
+
 jobs:
   build:
     name: Unit tests

.gitignore

@@ -152,4 +152,6 @@ packages/*/lib
 cypress/screenshots
 
 # Test cases have node module fixtures
-!test/**/node_modules
+!test/**/node_modules
+
+/tmp

cypress/e2e/default/revalidate.cy.ts

@@ -33,7 +33,7 @@ describe('On-demand revalidation', () => {
     cy.request({ url: '/api/revalidate/?select=5', failOnStatusCode: false }).then((res) => {
       expect(res.status).to.eq(500)
       expect(res.body).to.have.property('message')
-      expect(res.body.message).to.include('Invalid response 404')
+      expect(res.body.message).to.include('could not refresh content for path /getStaticProps/withRevalidate/3/, path is not handled by an odb')
     })
   })
   it('revalidates dynamic non-prerendered ISR route with fallback blocking', () => {

demos/default/netlify.toml

@@ -10,6 +10,7 @@ CYPRESS_CACHE_FOLDER = "../node_modules/.CypressBinary"
 # set TERM variable for terminal output
 TERM = "xterm"
 NODE_VERSION = "16.15.1"
+NEXT_SPLIT_API_ROUTES = "true"
 
 [[headers]]
 for = "/_next/image/*"

demos/middleware/netlify.toml

@@ -3,6 +3,9 @@ command = "npm run build"
 publish = ".next"
 ignore = "if [ $CACHED_COMMIT_REF == $COMMIT_REF ]; then (exit 1); else git diff --quiet $CACHED_COMMIT_REF $COMMIT_REF ../..; fi;"
 
+[build.environment]
+NEXT_SPLIT_API_ROUTES = "true"
+
 [[plugins]]
 package = "@netlify/plugin-nextjs"
 

demos/nx-next-monorepo-demo/netlify.toml

@@ -3,6 +3,9 @@ command = "npm run build"
 publish = "dist/apps/demo-monorepo/.next"
 ignore = "if [ $CACHED_COMMIT_REF == $COMMIT_REF ]; then (exit 1); else git diff --quiet $CACHED_COMMIT_REF $COMMIT_REF ../..; fi;"
 
+[build.environment]
+NEXT_SPLIT_API_ROUTES = "true"
+
 [dev]
 command = "npm run start"
 targetPort = 4200

demos/static-root/netlify.toml

@@ -3,6 +3,9 @@ command = "next build"
 publish = ".next"
 ignore = "if [ $CACHED_COMMIT_REF == $COMMIT_REF ]; then (exit 1); else git diff --quiet $CACHED_COMMIT_REF $COMMIT_REF ../..; fi;"
 
+[build.environment]
+NEXT_SPLIT_API_ROUTES = "true"
+
 [[plugins]]
 package = "@netlify/plugin-nextjs"