Initialize code at server startup and make it available in ctx

[c100k]

Feature request

Is your feature request related to a problem? Please describe.

In almost all the applications, we need to write code that needs to be run only once when the server starts in order to initialize "global" elements:

• Connecting to a database
• Starting some jobs
• etc.

In the current version, we don't see how to do it. There are some hacky ways of initializing these but they do not seem 100% reliable. (i.e. : https://www.codeoftheprogrammer.com/2020/01/16/postgresql-from-nextjs-api-route/)

Describe the solution you'd like

It would be very handy to be able to define a file where all this initialization code can be added following the example of _app.js or _document.js.

It could be called _init.(js|ts). When present, the next command would execute it before starting the actual server. Here is what it would look like:

// Here is the signature of the function, it should not accept any arguments
// It returns a Promise with custom context that next will set in `ctx`
export default async function _init(): Promise<ICustomContext> {
    // Connect to a PostgreSQL database
    const dbClient = await connectToDatabase('psql://...');

    // Start some jobs
    setInterval(() => {
        // eslint-disable-next-line no-console
        console.log('This is a global setInterval that would run while the server is running');
    }, 1000);

    // The returned object will be available in pages through `ctx`
    return {
        dbClient,
        someOtherValue: 'GlobalSetting',
    };
}

Describe alternatives you've considered

As an alternative, we can implement global code initialization using NodeJS.Global but it seems too hacky : https://www.codeoftheprogrammer.com/2020/01/16/postgresql-from-nextjs-api-route/.

Additional context

Might be related in some way to #8969

[mattcarlotta]

You may want to use a custom server if you need to initialize custom jobs/services vital to a connection; otherwise, off-load them to a microservice. However, if you're trying to avoid a custom server, another approach I've found is to store the pg-promise connection into a runtime configuration variable. While this also a bit hacky and also won't work with serverless, it at least avoids polluting the global/window spaces.

This will also work if you're using pg-monitor as it can only monitor one connection at a time without calling detach.

Click to expand example

database/index.js

const bluebird = require("bluebird");
const monitor = require("pg-monitor");
const pgPromise = require("pg-promise");

const initOptions = { promiseLib: bluebird, noWarnings: false }; // Database options
const pgp = pgPromise(initOptions); // initialize pg-promise with options

const { NODE_ENV, DB, DBHOST, DBPASSWORD, DBPORT, DBUSER } = process.env;

if (NODE_ENV === "development") {
  monitor.attach(initOptions, ["query", "error"]);
} else {
  monitor.attach(initOptions, ["error"]);
}

module.exports = pgp({
  database: DB,
  host: DBHOST,
  password: DBPASSWORD,
  port: DBPORT,
  user: DBUSER,
});

database/connection.js

const getConfig = require("next/config");

const { db } = getConfig().publicRuntimeConfig;

module.exports = db;

next.config.js

require('dotenv').config()
const db = require("./database");

module.exports = { 
  publicRuntimeConfig: { 
    db 
  }
};

[ItsMrAkhil]

But, Vercel doesn't support custom server deployments.

[c100k]

Mmmh thanks @mattcarlotta, I hadn't seen this next/config solution. It looks better than polluting global indeed.

I've had a look at the docs. Is there a reason why you would put your db in publicRuntimeConfig instead of serverRuntimeConfig and therefore making it available in the client ?

[mattcarlotta]

Depends on how you're using db. If you're using getInitialProps, and calling the db directly, then you'll want publicRuntimeConfig. If you're using getServerSideProps, then you'll want to use serverRuntimeConfig. Otherwise, if you're just making AJAX requests to the api pages, and the db logic is used within the api page, then you'll want to use serverRuntimeConfig.

[c100k]

So after some testing, it turns out one cannot easily do what you recommend @mattcarlotta because it is impossible to have a next.config.ts if you develop in TypeScript. Therefore, you need your whole chain to be JavaScript otherwise you can't require/import your code in next.config.js. 😕

[adorsk]

Some other use cases for something like an 'preServerStart' hook:

1. Setting up a store or DB that uses the filesystem. For example, a leveldb instance or a sqlite instance.
2. Reading data from a file.
3. Setting up a local 'scratch' dir, relative to the build file.

These use cases sometimes come up when making small tools or hack projects. For cases like these it would be nice to not have to use a custom server, but to instead hook into the server start-up.

[jmoo]

So after some testing, it turns out one cannot easily do what you recommend @mattcarlotta because it is impossible to have a next.config.ts if you develop in TypeScript. Therefore, you need your whole chain to be JavaScript otherwise you can't require/import your code in next.config.js. 😕

I was able to hack together a solution for this that works extremely well at the moment but due to relying on nextjs internals it may break in the future. It works with typescript and hot module reloading.

Here's what you'll have to do to get this working:

1. Create a ./src/bootstrap.ts that will handle everything you want to do on server start.
2. Get your server-side typescript code into the webpack toolchain and externalize all of your imported modules.(Generalizing this to automatically import all of your server-side code is left as an exercise for the reader)

    // next.config.js example
    module.exports = {
        webpack: (config) => {
            if (config.output.path.match(/server$/i)) {
               const entryConfigFunction = config.entry;
               config.entry = (context) => entryConfigFunction(context).then((entry) => {
                    entry['foo-service'] = './src/foo-service.ts';
                    entry['bootstrap'] = './src/bootstrap.ts';
                    return entry;
                });
            }
            // you will need to replace this regex with a function that fixes the import paths once you have modules in nested directories
            config.externals.push(/foo-service|bootstrap/);
         }
     }
   

3. Create a custom server.js that will require your compiled application bootstrap.

// server.js example
const nextToolchain = next(...);
const handler = nextToolchain.getRequestHandler();

const boot = () => {
   const app = require('./.next/server/bootstrap');
   app.boot();
}

nextToolchain.prepare().then(() => {
   boot();
   server = createServer(handle);
   server.listen(parseInt(process.env.PORT, 10), process.env.HOST);
});

4. At this point production builds should work but to get the dev server working, you will need to hook into the next internals and reload your application bootstrap after every compilation.

// server.js
nextToolchain.prepare().then(() => {
   boot();
   server = createServer(handle);
   if (dev) {
      require('next/dist/build/output/store').store.subscribe(async (state) => {
         if (!state.loading && !state.typeChecking && !state.errors && !state.warnings) {
            boot();
         }
       } 
    }
    server.listen(parseInt(process.env.PORT, 10), process.env.HOST);
});

[amille44420]

Thank you, I spent quite some time looking for a way to catch hot reload for development purposes and you got it working. I wished we could have a proper and documented way to add/customize the entry points with hot reload capability.

In my case I needed to couple the next server build & sources with :

• automated/scripted data migrations
• workers for asynchronous jobs

[taoqf]

I just need an entry to setup logger.
I wish there is an entry file, like a _startup.js file, there I could do the app.prepare() things.
may be init DB, configure the logger, even add middlewares in that file.

[chris-schra]

well, using typescript it's not that elegant, but we did:

Install @babel/register

In your next.config.js:

require('@babel/register')({
    extensions: ['.js', '.jsx', '.ts', '.tsx']
})

Inject into the webpack config (abbreviated for clarity):

webpack: (config, { buildId, dev, isServer, defaultLoaders, webpack }) => {
     if(isServer)
     {
       if(config.entry && !entryModified)
       {
         entryModified = true;
         const oldEntry = config.entry;
         config.entry = async () => {
            return oldEntry().then(require("./server/index"))
         }
       }
     }
    return config;
}

this loads "our" bootstrap process in ./server/index.ts where we init crons and mongoose. And yes, so far it seems to be working because our /api routes use that connection 😆

EDIT: That's only for dev server, please see below

[chris-schra]

Here's a gist for a cleaner example, working with dev and production:
https://gist.github.com/chris-makaio/4c26924bcc9d6229fb5a3720f9d50b17

packages to install:

• ts-node
• @babel/register

start inside your project root with ts-node --project tsconfig.server.json server/index.ts

debug with TS_NODE_PROJECT=tsconfig.server.json node -r ts-node/register server/index.ts

[ddegroot1985]

Gist not found

[ahbnr]

Gist not found

He changed his user name. That's the reason for the 404.
You can still find the gist under https://gist.github.com/chris-schra/4c26924bcc9d6229fb5a3720f9d50b17

[mrgoonie]

WORKAROUND here, my friends!

1. Create a start up script, for example: /server/index.js with whatever you want to do here, for example:

console.log("SERVER is initialized!");
// then, database initialized...
// then, cron job initialized...
// blah blah...

2. Create /api/startup.js with:

// import that startup script from server side here:
require("server/index");

const handler = async (req, res) => {
	res.statusCode = 200;
	res.status(200).json({ status: 1, message: "OK" });
};

export default handler;

3. If you don't have any custom server file, create one. Then after the server startup script, make a HTTP request to the above API endpoint:

const request = require("request");
...
request("http://localhost:3000/api/startup");
// You will see: "SERVER is initialized!"

That's it!

(Why thumb down? 🤔 It's hacky obviously, I know, but this workaround works safely)

[revero-doug]

it doesn't work safely since the server listens before there's any sort of guarantee that /api/startup is called before anything else

[Pig-Cola]

I hope this solution will solve your problem.

This is an example of the startup function in next JS with not custom server

This repository run startup function without using custom server.
Before and after the next JS is loaded, it can be executed respectively.

// before next loading

console.log(process.env.ENV_TEST, '[expected]: undefined') // undefined

// after next loading

const command = process.argv[2]
if (['dev', 'start'].includes(command)) {
  require('./node_modules/next/dist/bin/next')
  check(init)
}

// function
// function
// function

// next loading check
/** @param {Function} fn */
function check(fn) {
  setTimeout(() => {
    if (!('LOAD_CHECK' in process.env)) {
      setTimeout(check, 0, fn)
      return
    }
    fn()
  }, 50)
}

// after init function
function init() {
  switch (command) {
    case 'dev': {
      // here dev script
      console.log('run dev')
      break
    }
    case 'start': {
      // here start script
      console.log('run start')
      break
    }
  }
  // commons script
  console.log(process.env.ENV_TEST, '[expected]: Hello!!') // 'Hello!!'
}

For more information, see the app.js in this repository

[Max4nd]

Any updates on this issue? I would rather not hack in a solution like some of the answers above but it seems that it's the only answer at this point in time :(

[Max4nd]

My current workaround is using a custom server that sends a request to an endpoint (pages/api/foo) and the logic in foo will be run on startup. This is super hacky but seems to work well.

[Max4nd]

I just realised this was @mrgoonie solution! Give him cred for the idea haha

[jonathanasquier]

I'm currently building a next app with a apollo-graphql server and a webhook handler
Works pretty well but I'm out of luck when trying to initialize and share a startup context between my pages/api/graphql and my pages/api/webhooks
I'm using a singleton (I don't like singletons) for the context that is initialized on each page once, only if the route is called at least once 😩
So no context sharing, no fail fast on boot, no graceful shutdown on an unhandled error...
I tried to use a custom koa server with a apollo server handler and a webhook handler to share a startup context in the koa ctx, but it's very sloppy.
These are very big tradeoffs...
🙏

[aMoniker]

This seems like a sensible, feasible feature that lots of people want. Could someone on the Next.js team tell us if this will be implemented, and if not, why not?

[ItsMrAkhil]

It's been more than 2 years since this issue was created. I do not understand why the Vercel team is looking into this issue.

[HelloKashif]

After trying all the methods, I've found using a custom server to be the best and most reliable solution.

What most people seem to be wanting out of this is to implement long-running tasks/background processes that just won't work on Vercel due to the ephemeral/serverless nature of the deployment. So if you still need it, then a custom server (deployed outside Vercel) would be the best option, imo.
Vercel will obviously never implement this because custom server-related code can't be deployed there; hence not a priority for them.

Running a custom server is trivial. Instead of doing next start you would do node server.js and then run whatever code you want there.

//server,js
const express = require('express')
const next = require('next')

startBackgroundProcess()
doWhatever()

const app = next({ dev: false })
const handler = app.getRequestHandler()

await app.prepare()

const server = express()

server.all('*', handler)

server.listen(3000)

[mytchallb]

That's what I also found to be best! My startup file looks like this:

// Force production mode
process.env.NODE_ENV = 'production';

const app = require('next/dist/cli/next-start');

app.nextStart({
  port: process.env.PORT || 3000,
});

Also ade a post on some common issue with Next.js/Plesk.

[leerob]

Hey folks, wanted to mention this solution. I know it's not perfect, but it should unblock you from running code before the Next.js server starts without needing to eject out of the default server.

node --require ./server-preload.js ./node_modules/.bin/next start
#    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
#    Load our preload before running `next start`

Source and credit: https://jake.tl/notes/2021-04-04-nextjs-preload-hack

[bonniss]

Hi @leerob, how can I use this with ./server-preload.ts?

[mattduggan]

When using standalone output, the modules used by the initialization script that are not imported in source are not included as they cannot be traced. Is there a way to configure next to include these modules?
To clarify, I'm using Next 12. Next 13 instrumentation hook is too buggy at the moment to consider upgrading for my purposes.

[j4tmr]

+1 Still no native solution

[yf-hk]

It's been 3 years now and it isn't likely to be addressed by Next team any time soon. It's kind of annoying since Next is getting more popular as a all in one backend server as well, especially after tools like tRPC and Blitz.js emerge.

[anthonyalayo]

I believe...there is now a solution?

Last week, version 13.3.2 made changes for:
Add support for async instrumentation register: #48575

Looking at the history, version 13.2.0 added the initial support:
feat: add instrumentation hook: #46002

Checking out the docs from last week:
https://nextjs.org/docs/pages/building-your-application/optimizing/instrumentation

If you export a function named register from this file, we will call that function whenever a new Next.js server instance is bootstrapped. When your register function is deployed, it will be called on each cold boot (but exactly once in each environment).

Note: This feature is experimental. To use it, you must explicitly opt in by defining experimental.instrumentationHook = true; in your next.config.js.

I haven't tried it myself yet, but this may be what we were looking for.

[anthonyalayo]

@mickeld23 I'm encountering the same thing

[jtlapp]

This isn't doing what I expect. register is called twice in dev mode and not at all in production mode.

And when I only import or require server-side modules from within register, it still complains about server-side modules not being available (e.g. path and fs).

[tkmcmaster]

I had to wrap it in a if (process.env.NEXT_RUNTIME === "nodejs") {} to get rid of the complaining about path and fs. But it's working for me now in production mode. I haven't tried dev mode. That was a recommendation from the docs https://nextjs.org/docs/pages/building-your-application/optimizing/instrumentation.

[tkmcmaster]

Ok. After some testing I found the change. In Next.js 13.2.4, instrumentation starts at npm start. Starting in 13.3.0 it runs on the first web call rather than server start. The first web call can be a page OR api call that will trigger the instrumentation code. Then in 13.4.2, it now runs twice on the first API call.

That's what I've found from some testing.

[anthonyalayo]

Very interesting, I wonder if @leerob can chime in.