Feature request: make logging possible on ejected custom server

[Janpot]

Feature request

Is your feature request related to a problem? Please describe.

Using custom server interferes with other logging in my app.

Describe the solution you'd like

A way to silence next and an interface to tap into ready, recompile, error events.

const app = next({ dev: true, quiet: true })
app.on('ready', myLogger.info);
app.on('recompile', myLogger.trace);
app.on('error', err => myLogger.error('something went wrong', { err }));

const handle = app.getRequestHandler()
// ...

[Janpot]

@timneutkens I would like to find a way to integrate the logging of the request handler into my logging/monitoring infrastructure. I guess it would need some refactoring on next side to be able to hook into this. I don't mind trying to cook up a PR for this if this is something you'd like to add. Right now I see two directions this can be implemented:

1. provide a logger instance in the style of things like pino. If you want to overwrite logging, you can just pass your own.
2. "silence" next with a verbose: false flag or stdout: null (or another stream) or something. And add events that report on the different stages of the app compiling, compiled, error,...

What are your thoughts on this?

[ravinggenius]

Is there any interest in this? I have a custom server, and I'd really like to unify my log output. I'm personally fine with either approach. Passing in a custom logger sounds easier for Next users (developers), though adding a silent option and emitting logging events would be more flexible.

@timneutkens, @rauchg Thoughts? If bandwidth is an issue I can probably get something working in my spare time over the next few weeks.

[bensaufley]

More than this, I'm using pino-http for my logging and it attaches a bunch of metadata to every log entry; error logging would be much more useful if I were able to get that information on errors. A simple error handler would be better than nothing, but I'd prefer something even more granular.

It seems to me that Next.js is intercepting these errors so neither Sentry.io (which we use for error reporting) nor our server-level try/catch receive the error to handle or log it. This is really inconvenient. It seems like the ideal solution would be just not to catch these errors in a custom server (maybe a flag… catch (e) { if (process.env.NEXT_HANDLE_ERRORS === 'false') throw e; }) and let my top-level server catch them and handle them.

[adamhathcock]

Allowing some way to attach a customizable logger to Next's logging would be great for me.

I log JSON to stdout for Docker and use Next as a custom server.

[oswaldoacauan]

Did anyone found a workaround? We are trying to integrate Rollbar and Datadog and this would be great.

[ravinggenius]

My "workaround" is to just filter out non-JSON lines. Using roarr:

npm run start | npx roarr filter --context 0 --exclude-orphans '{ "context.logLevel": { gt: 10 } }' | npx roarr pretty-print

My solution for pino is similar, but I had to use jq to filter offending lines.

[Ianfeather]

Just want to bump this issue. Not being able to have structured logs emit from next is a real limitation. Would love to see an api like:

const app = next({ dev: true, logger: myLogger })

and happy to work on this if you'd be willing

[shakedlokits]

Bumping it again, this is important, would be happy if it would exist.

[kraiovsky]

+1 this would be good improvement

in the meantime, how do you guys log response body with your pino or pino-http solutions?

[ravinggenius]

I added an Express middleware to log each request. pino-http makes this easy.

This works because each route hits the server, whether rendering on the server (HTML) or client (JSON).

[bensaufley]

That doesn't handle this problem though, because as soon as inside your Express handler you hand off the request to next, pino has no jurisdiction and everything gets outputted to stdout or stderr again. You can log the initial request but as soon as you call handle or whatever, logging is taken out of your hands.

[kraiovsky]

I ended up creating a custom serializer for next req/res: file
a higher-order function that takes that logger, and wraps around API handler: file
example of usage: file

project repo

if there is some interest, I could convert it into a package that would take other loggers and more middlewares to run.

[daviduzumeri]

+1 to this being sorely missed; we've got stdout logging to DataDog with structured logs for all of our other services and now it's just constantly interspersed with Next.js's line logging being noise.

[PsyGik]

Using a custom server ejects from the Next.js defaults and de-optimizes your application from things like Automatic Static Optimization.

I've seen this statement almost everytime there is a mention of a custom server, (including the docs), but never a detailed explanation of how and why it disables Automatic Static Optimization.

[meotimdihia]

@leerob: Setup next.js with a custom server is easy.
I think the problem is how to print logs with self-hosted Next.js.
And no one wants to disable Automatic Static Optimization.

[leerob]

Is it possible to configure the way Next logs when not using a custom server?

By default, logs and errors are printed to STDOUT / STDERR. You can still persist and store these logs without modifying anything using next start. To customize logging, using libraries like Pino or related, you can do something like this as a workaround for now:

node --require ./server-preload.js ./node_modules/.bin/next start
#    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
#    Load our preload before running `next start`

Where this is an example of the preload script. Agreed this could be easier.

detailed explanation of how and why it disables Automatic Static Optimization.

Check out https://nextjs.org/docs/advanced-features/automatic-static-optimization as well as the source code for this feature to understand more how it works 😄

[ValentinH]

Thanks for the answer, we are indeed using the preload way for the moment.

[mAAdhaTTah]

@leerob Sentry's nextjs plugin prepends a sentry.config.server.js file to the webpack entry, which gets loaded before nextjs boots up. I attempted to plug the above example into that file, but get an error related to this line:

const nextBuiltInLogger = require("next/dist/build/output/log")

When it starts to monkeypatch the built-in logger, it's returned undefined. I think generally speaking, it would be nice for this kind of monkeypatching to be available in a more blessed way, but in the meantime, do you have any idea why these files would be coming back undefined in that case?

I've currently got a few "hacks" that require a custom server and I've been looking at moving it back into the built-in server but can't get the hacks patched in the way it's described in that gist. Any thoughts as to why and whether this should be considered a bug? It would be nice to have a single preload for Sentry + DD + logs rather than 2.

[PsyGik]

Check out https://nextjs.org/docs/advanced-features/automatic-static-optimization as well as the source code for this feature to understand more how it works 😄

I did read it Lee. And to make sure I understood things correctly, I tried out the Custom Server example locally.

If the above is not the case, Next.js will statically optimize your page automatically by prerendering the page to static HTML. [...] next build will emit .html files for statically optimized pages. For example, the result for the page pages/about.js would be - Automatic Static Optimization

To validate the above, I did this:

❯ npm run build

> [email protected] build
> next build

info  - Using webpack 5. Reason: Enabled by default https://nextjs.org/docs/messages/webpack5
info  - Checking validity of types
warn  - No ESLint configuration detected. Run next lint to begin setup
info  - Creating an optimized production build
info  - Compiled successfully
info  - Collecting page data
info  - Generating static pages (6/6)
info  - Finalizing page optimization

Page                             Size     First Load JS
┌ ○ /                            1.65 kB        65.4 kB
├ ○ /[color]/c                   269 B            64 kB
├ ○ /404                         3.17 kB        66.9 kB
├ ○ /a                           262 B            64 kB
└ ○ /b                           262 B            64 kB
+ First Load JS shared by all    63.8 kB
  ├ chunks/framework.64eb71.js   42 kB
  ├ chunks/main.a3a79a.js        20.2 kB
  ├ chunks/pages/_app.a8eaee.js  801 B
  └ chunks/webpack.672781.js     766 B

λ  (Server)  server-side renders at runtime (uses getInitialProps or getServerSideProps)
○  (Static)  automatically rendered as static HTML (uses no initial props)
●  (SSG)     automatically generated as static HTML + JSON (uses getStaticProps)
   (ISR)     incremental static regeneration (uses revalidate in getStaticProps)

❯ ls .next/server/pages
404.html     500.html     [color]      _app.js      _document.js _error.js    a.html       b.html       index.html

And it looks like it is indeed generating static files. So I hope you'd understand why I am confused, when "Using a custom server ejects from the Next.js defaults and de-optimizes your application from things like Automatic Static Optimization"? 😅

[Agontuk]

I've seen this statement almost everytime there is a mention of a custom server, (including the docs), but never a detailed explanation of how and why it disables Automatic Static Optimization.

I'm also curious to know how this works, current documentation does not explain this. Let me give 2 examples here so that someone can clear this up for me and others.

Case 1

I only need custom server to add an extra line that logs my route to the monitoring service, and let Next js handle the rendering

// server.js
const { createServer } = require('http')
const { parse } = require('url')
const next = require('next')

const dev = process.env.NODE_ENV !== 'production'
const app = next({ dev })
const handle = app.getRequestHandler()

app.prepare().then(() => {
  createServer((req, res) => {
    const parsedUrl = parse(req.url, true)

    // log to monitoring service
    logToMonitoringService(parsedUrl.pathname)

    // does this disable ASO ?
    handle(req, res, parsedUrl)
  }).listen(3000, (err) => {
    if (err) throw err
    console.log('> Ready on http://localhost:3000')
  })
})

Case 2

I want to render some of the routes manually, and let Next js handle the other routes

// server.js
const { createServer } = require('http')
const { parse } = require('url')
const next = require('next')

const dev = process.env.NODE_ENV !== 'production'
const app = next({ dev })
const handle = app.getRequestHandler()

app.prepare().then(() => {
  createServer((req, res) => {
    const parsedUrl = parse(req.url, true)
    const { pathname, query } = parsedUrl

    if (pathname === '/a') {
      // this should disable ASO, since we're rendering manually
      app.render(req, res, '/a', query)
    } else {
      // this shouldn't disable ASO, since we're letting next js handle it
      handle(req, res, parsedUrl)
    }
  }).listen(3000, (err) => {
    if (err) throw err
    console.log('> Ready on http://localhost:3000')
  })
})

As far as I understand, automatic static optimization should be disabled only for the manual routes in case 2. Because when I run next build, it is already generating static html for routes that don't use getServerSideProps. That means ASO should work for those routes even if I'm using custom server (but letting the Next js handle the rendering).

[leerob]

Further, I want to mention the solution for this - which is where next start is headed. We're working to support OpenTelemetry inside Next.js. This includes tracking, logging, and adding metrics.

That will be the intended solution for this 👍

https://opentelemetry.io/

[trevoro]

Hey @leerob; it's been over a year and this thread is pretty old. I think people have some very valid use cases for wanting to be able to set something as basic as a logger within next. While I appreciate the plan was / is to use opentelemetry this is sort of orthogonal to that goal and opentelemetry has nothing to do with being able to silence or capture logging output when running nextjs in test mode, for example. Can we just get some kind of acknowledgement that the effort someone might put into contributing to this open-source project won't go unnoticed? It's pretty clear that a large part of the community would be grateful for such a feature.

[sandcastle]

We're working to support OpenTelemetry inside Next.js. This includes tracking, logging, and adding metrics.

@leerob, this is great news! 🙌

Is there anything the community can do to support this effort?

[kamranayub]

It sounds like there's no appetite to merge this proposal? #22587

One additional thing we added ourselves was tying the logError call to the req context/res context since we like to add a correlation ID for all logs occurring during the request pipeline. All instances of logError have RequestContext or res in scope that they could pass to logError so that seems pretty doable. Patching console.error would not allow this so that's one benefit of making logError better.

I'd love to help move that PR in until we get OpenTelemetry, especially it being Hacktoberfest. But it sounds like that appetite is low considering #4808 (comment)

[raimohanska]

I found a pretty simple workaround for this when using a custom server.

import nextJs from "next"

async function createNextServerWithCustomLogging() {
  const server = nextJs({})
  const innerServer = (await (server as any).getServer()) as any
  if (!(innerServer.logError instanceof Function)) throw Error("Assertion fail")
  innerServer.logError = (err: Error) => {
    console.log("Custom handler", err)
  }
  return server
}

[trevoro]

Hey Team; I'm not excited about preloading or patching global console messages so I'm happy to see that the plan is to adopt OpenTelemetry.

I think something that would make it easier would be knowing what kind of timeline when initial OpenTelemetry support is supposed to land even on a super coarse scale like 'This year' or 'maybe next year', etc. I'm not asking for a commitment just a rough idea. :) 🙏

[leerob]

Thanks for the shout here @trevoro. I do have an update. We have been working on the OpenTelemetry integration into Next.js actively. I do want to note, however, this will require using next start, not an "ejected" server.

To reiterate, using next start (whether self-hosted, deployed through Docker, deployed to Vercel, etc) supports all Next.js features. We do not recommend using the ejected server, as the majority of reasons for ejecting over the past 3 years have now been built back upstream into Next.js (for example, advanced routing patterns through Middleware, layouts support in the app directory, and more).

[ScottAwesome]

Any word on when the OpenTelemetry will land?

I'm getting exhausted managing loggers between SSR environments, middleware, and the browser. There's no transparent isomorphic / universal logger on the market currently that works everywhere - edge functions / middleware, nodejs, and the browser - that doesn't require path hacks or extensive wrapping, and log drains aren't always enough.

Even trusty pino is becoming incompatible with Next.js (doesn't work in middleware or edge runtimes)

[weyert]

The only use case for using a custom/"ejected" server is https during localhost and OpenTelemetry everywhere. Are there plans to pass a certificate to the next start?

[Rowno]

Another use case is if you need to have Websocket endpoint for a realtime chat app etc.

[david-alpert-nl]

any update on OpenTracing support? or ability to inject server startup code?

is this it? 🤔 https://nextjs.org/docs/advanced-features/open-telemetry

[DopamineDriven]

Following. Trying to determine how to best get a dd-trace integration into next without compromising performance to provide richer insights than @datadog/browser-rum is currently able to do in isolation (though the session replays are pretty 🔥)

I know there is a server preloading method using a start script flag to load a file instantiating dd-trace first but it seems like there should be a clearer way to configure telemetry than this, especially for enterprise Vercel orgs

Should I use the preloading method for the time being or is there a better way?

[rohit-gohri]

We just set NODE_OPTIONS="-r dd-trace/init" environment variable and it works

[leerob]

Following up here, OpenTelemetry support landed already, but we're going to continue working on observability next.

https://nextjs.org/docs/app/building-your-application/optimizing/open-telemetry

[trevoro]

@leerob Is this version in Next 12, 13, or Next 14?
The documentation never makes it obvious what version a feature exists in.

[leerob]

I believe it was first introduced in 13.4.

[karlhorky]

Is OpenTelemetry also the suggestion for displaying the swallowed errors / exceptions in Next.js Custom Servers?

• app.prepare() failure crash the program but error message hidden #47211 (comment)

Or is this a bug?