defer -> async of next scripts caused a page speed drop

[danlunde]

We recently upgraded our next.js stack from 9.1.4 to 9.2.2, and everything was looking great until we noticed a meaningful dip in our page speed score and our First Contentful Paint metric.

Last September (represented by the yellow line) we enabled the deferScripts experiment with great success and was happy to see defer become the default Timer@2757826 so that we wouldn't need to keep the "experiment". But then the default was switched back to async 32c390a and since the experiment had been stripped out, we no longer have a means to opt-in to using defer.

According to #10143 (comment) it was after investigations and recommendation of the Chrome team, but I'd love to better understand what that investigation yielded and why async is the better alternative, particularly since it has an adverse affect on page speed.

Could we get the deferScripts experiment back?

[jgubman]

@Timer any updates on getting the deferScripts experiment back? People are also asking for it on the PR

[prashantchothani]

Need the defer option for scripts.

[jgubman]

@Timer any thoughts?

[Siggnja]

@jgubman Have you done any tests after PSI updated to lighthouse v6 and see if defer vs async has an effect on it?

[jgubman]

Yes, unfortunately our speed scores haven't changed

[mintyfloss]

We are seeing the same thing. We are currently pinned to 9.1.7 and see our scores drop dramatically when upgrading to 9.4.4. All metrics are about the same except for FCP which doubled as well. It went from 2.1s to 4.7s.

I compared the generated source and the only difference is the number of JS files being referenced. Because version 9.2 changed the way chunks are generated, our page is now seeing an additional 19 JS files.

Our next app has 11 different pages. If I delete all the pages except for the one I'm testing, Next generates much less JS files, and our scores are back to normal. I wonder if there's a way to customize how these chunks are generated.

[jgubman]

Since we haven't heard back, we went ahead and extended NextScript to defer scripts via this component:

class DeferredNextScript extends NextScript {
  getScripts() {
    return super.getScripts().map(script => {
      return React.cloneElement(script, {
        key: script.props.src,
        defer: true,
        async: false,
      })
    })
  }
  getDynamicChunks() {
    return super.getDynamicChunks().map(script => {
      return React.cloneElement(script, {
        key: script.props.src,
        defer: true,
        async: false,
      })
    })
  }
}

After replacing <NextScript /> with <DeferredNextScript /> in _document.js, we immediately noticed a jump in our PageSpeed scores, especially the FCP score. Here's an updated graph (the DeferredNextScript hack is the last couple of data points on the graph). @Timer @Siggnja we'd LOVE to see the experiment turned back on or at least a reason why we can't defer our scripts?

[prateekbh]

@jgubman we're amidst data collection phase to move to defer scripts. I would be very interested to experiment with you app as well.
Is there an before/after web page test we can take a look at? or a link to repo if it is open source?

[jgubman]

Hey Prateek, I replied to your email but it bounced. We don't have any public links that use the default async implementation at the moment, but I'll hack something together that'll switch implementations based off the presence of a query param.

I'll ping you when that's ready.

[jgubman]

@prateekbh We've been collecting the timing data that drives these graphs for awhile. Unfortunately we haven't been keeping track of every metric on the lighthouse page, but we do have desktop and mobile metrics for speed score, FCP, TTI and FMP.

A sample that also shows which URLs the timings are associated with is here:
https://docs.google.com/spreadsheets/d/1VB4nnGahx9fkZZIB2FquTRhMv9RNv5RSbeDID_KPy9Q/edit?usp=sharing

the 8/31 run was pretty much the default Next.js <NextScript> implementation and the ONLY change to the 9/2 run is we introduced the <DeferredNextScript /> component in the _document.js as I mentioned above.

[jgubman]

Hi Prateek, we made the change, if you tag defaultNextScript=1 to the end of any of our query strings you'll get the default <NextScript /> behavior. Omitting that query string param will get you the hack I have above that adds the defer attribute to nextscripts. Here's a sample page w/ the WPS outputs:

https://www.ign.com/articles/marvels-avengers-review-2?defaultNextScript=1 -- Mobile score 9, Desktop score 34
https://www.ign.com/articles/marvels-avengers-review-2 -- Mobile score 20, Desktop score 52

Default Next Script:

DeferredNextScript:

[prateekbh]

This is weird, I am getting very similar results for both. Are those results consistent for you?

[jgubman]

Strange, I consistently get a mobile score of 10 on that page when I run it with the query param on PageSpeed Insights. Agreed, the only meaningful difference in lab data scores I can see is First Contentful Paint, which is always around 5s with the default <NextScript /> component and closer to 1s with the <DeferredNextScript /> component

[prateekbh]

@jgubman, in your screenshots the only major difference is FCP which shouldn't be dependent on scripts anyway but only on TTFB. Doing a few runs on webpage test to see whats going on

[dvelasquez]

which shouldn't be dependent on scripts

This is not entirely true @prateekbh . Scripts can affect the FCP, depending on how they are loaded.
A normal <script> will stop the execution, parsing and rendering of HTML until it gets downloaded and executed.

A <script async> will not block the parsing and rendering of HTML during the download of the script. But if the script get completely downloaded while the browser is still parsing or rendering, the current process will be blocked until this script execution finishes.

A <script defer> will not block the parsing and rendering of HTML. Instead it will wait until the HTML is fully rendered to start executing it (DOMContentLoad).

What is the problem here then?: the <link rel="preload"> in the head will immediately start the download of the script therefore, when the <script async> is parsed, the file is already downloaded and it will be executed, effectively blocking the render of the page (and affecting the FCP).

Here is how Chrome is treating the scripts link. In this case, async is stopping the parser from execution.

This might be different for other browsers, but if you are running lighthouse, it will be Chrome

[dvelasquez]

The problem here is the coexistence of <link rel="preload"> and <script async>.

Using both at the same time is what is causing the problems in the FCP.

So, we have two ways of address this problem:

1. Use <script defer>
2. Don't add <link rel="preload"> in the Head.

This is my TypeScript code for each case:
DeferredNextScript is more complicated, it needs some types that are not externally exposed in NextJS

import { NextScript } from 'next/document'
import React from 'react'

type DocumentFiles = {
    sharedFiles: readonly string[]
    pageFiles: readonly string[]
    allFiles: readonly string[]
}

function dedupe<T extends { file: string }>(bundles: T[]): T[] {
    const files = new Set<string>()
    const kept: T[] = []

    for (const bundle of bundles) {
        if (files.has(bundle.file)) {continue}
        files.add(bundle.file)
        kept.push(bundle)
    }
    return kept
}

export class DeferredNextScript extends NextScript {
    getScripts(files: DocumentFiles) {
        return super.getScripts(files).map((script: JSX.Element) => {
            return React.cloneElement((script), {
                // eslint-disable-next-line @typescript-eslint/tslint/config
                key: script.props.src,
                defer: true,
                async: false,
            })
        })
    }
    getDynamicChunks(files: DocumentFiles) {
        const {
            dynamicImports,
            assetPrefix,
            devOnlyCacheBusterQueryString,
        } = this.context

        return dedupe(dynamicImports).map((bundle) => {
            let modernProps = {}
            if (process.env.__NEXT_MODERN_BUILD) {
                modernProps = bundle.file.endsWith('.module.js')
                    ? { type: 'module' }
                    : { noModule: true }
            }

            if (!bundle.file.endsWith('.js') || files.allFiles.includes(bundle.file))
            {return null}

            return (
                <script
                    defer={true}
                    key={bundle.file}
                    src={`${assetPrefix}/_next/${encodeURI(
                        bundle.file
                    )}${devOnlyCacheBusterQueryString}`}
                    nonce={this.props.nonce}
                    crossOrigin={
                        this.props.crossOrigin || process.env.__NEXT_CROSS_ORIGIN
                    }
                    {...modernProps}
                />
            )
        })
    }
}

HeadWithoutPreload is easier, since is just override this two methods to return an empty array.

import { Head } from 'next/document'

export class HeadWithoutPreload extends Head {
    getPreloadDynamicChunks () {
        return []
    }

    getPreloadMainLinks () {
        return []
    }
}

[svassaux]

@dvelasquez do you see a way to load all those scripts after the window.onload event?

I have implremented this sucessfully so far but that does not yield the expected results, my google pagespeed mobile score would not go over 50. Id I disable NextScripts as a whole I have a 100, but I need to keep them because I am using some js still...

[dvelasquez]

@dvelasquez do you see a way to load all those scripts after the window.onload event?

I have implremented this sucessfully so far but that does not yield the expected results, my google pagespeed mobile score would not go over 50. Id I disable NextScripts as a whole I have a 100, but I need to keep them because I am using some js still...

@svassaux it depends a lot of the reasons why you are getting that performance score.

This modified versions NextScript can help reducing the FCP & LCP but affect the TTI and TBT.

If you want, you can share the lighthouse results or your page so I can take a look on that.

Scripts loaded with defer will load after DOMContentLoad, which is not exactly what are you looking right?
Do you want to run those scripts in the window.load event instead?

[mrafei]

@dvelasquez
I have successfully implemented your solution with the second method and it had excellent result on FCP. Is there a tradeoff to it that I am not aware of? Seems like it has some negative effect on TTI and TBT as you mentioned but I don't understand the reasoning behind it. Can you explain to me how this is working and why Nextjs doesn't do this by default?
thanks.

[dvelasquez]

Answering your questions @mrafei

Is there a trade-off?:

Yes, it might affect your TTI and TBT, depending on how much javascript or other resources are you loading during and after the hydration.

Can you explain to me how this is working?

Here I explain a little more in deep why this happens: #11120 (reply in thread)

Long story short, is a race condition. My recommendation is the following:

• Good <script defer> +<link rel=preload>
• Good <script async> alone or with <link rel=prefetch>
• Bad <script async> + <link rel=preload>

Let me know if is not clear enough and I will try my best to explain it ;)

why Nextjs doesn't do this by default?

Hard question. I'm not a member of the core team or involved in NextJS, so take my answer with a pinch of salt: Is not easy to make a one size fits all framework or application, so there will always be cases that the authors of the framework have not given a thought. In this particular case, I think this is a widespread problem of the JavaScript ecosystem: JavaScript obesity. This issue is hardly noticeable if you are using only NextJS, but if you (like me) are running a big application, with thousands of items, multi-language, caring about accessibility, having your own design system and on top of that, loading GDPR libraries, marketing tags and showing ads... well that is a case that they might not be doing examples or testing themselves.

This problem is only noticeable if you are shipping a lot of javascript (over 250kb).

[dnaranjo89]

Google is rolling out the usage of the page experience signals in ranking in May 2021. Improving LCP will directly improve the site's ranking in Google Search.

Deferring the load of these scripts seem to make obvious improvements in that metric, are there any updates around why was that option removed or whether there are any plans to add it again?

[jordie23]

Has there been an update to this anywhere?

[bimusiek]

Is there any official way to enable defer option for next scripts? It improved our score as well and I wonder if those workarounds are fine long term. We would love to see official support for deferred scripts.

[spetroll]

Since #26021 (June 11th) scripts are loaded with defer as default again.

[mkotsollaris]

Is there a way to make NextScript async or vanilla (neither async or defer)?