revalidate's Cache-Control values

[jaksenko]

So I just tried to use Incremental Static Regeneration because it is a nice fit to our use-case: we have an internal CMS that gets updated only once a few days. To speed up the page we set revalidate: 3600. We use CloudFront as our CDN. However, after few attempts to verify that the page actually gets refreshed in 1 hour I noticed that Next.js sets Cache-Control: s-maxage: 3600, stale-while-revalidate header. This is very problematic if our server sits behind CloudFront, because now we have 2 layers of cache, each respecting stale-while-revalidate directive. So request coming after 1h gets old response from CF and CF requests new content in background. Next.js returns old content and refreshes its cache. After another hour CF still returns and old content, but in the background it finally gets a new content from Next.js, although if that content has changed again it still displays outdated one. The cycle never ends.

I don't understand why Next.js does that proactively without ability to opt-out. Nobody else uses CDN except for me?

I tried to use On-demand Revalidation as well, but that one is even crazier - a page with getStaticProps without revalidate key gets Cache-Control: s-maxage=31536000, stale-while-revalidate - how Next.js can even think about setting such header to a page url that obviously doesn't change in between builds? Anyone who accesses the page will forever see the old version of it, no matter how many releases behind.

Am I missing something extremely important here? Because I have a feeling that Next.js can't be that counter-intuitive.

[jamesopti]

Any update from the Vercel team on this one? Does this change with Next13?

[benjaminDanis]

@jaksenko I was investigating a similar issue.

I found this from NextJS Docs

// This value is considered fresh for ten seconds (s-maxage=10).
// If a request is repeated within the next 10 seconds, the previously
// cached value will still be fresh. If the request is repeated before 59 seconds,
// the cached value will be stale but still render (stale-while-revalidate=59).
//
// In the background, a revalidation request will be made to populate the cache
// with a fresh value. If you refresh the page, you will see the new value.
export async function getServerSideProps({ req, res }) {
  res.setHeader(
    'Cache-Control',
    'public, s-maxage=10, stale-while-revalidate=59'
  )

  return {
    props: {},
  }
}

If the request is repeated before 59 seconds, the cache value will be stale ... in the background a revalidation request will be made ...

According to web.dev, a request made outside of the window of max-age and stale-while-revalidate will deem the cache stale and abandon it for the fresh value. That would be useful in getStaticProps, but I haven't seen any option to specify that for when it runs on revalidation.

All Cache-Control headers specified in next.config.js are overwritten in production, so thats not an option either.

[MartinDavi]

None of what I wrote here is true, so you can ignore the rest of this comment. Middleware, like the next.config.js headers prop seems unable to set Cache-Control header. I can set other custom headers and see them on the response, but not Cache-Control.

Another option is to manually add the Cache-Control header in middleware.

We're using SSG with on demand revalidation - so HTML files are generated at build time, and regenerated when the content in the CMS is changed. We also have a CDN in front of all of this which is going to respect the s-maxage value before checking back in with nextjs for fresh content.

Because we're using getStaticProps and not getServerSideProps, we don't have access to the request to set the cache-control headers.

We don't want to use the revalidate prop because - while that does correctly set the s-maxage value - it is pulling double duty and also triggering a regeneration of the html file if a request comes in outside the s-maxage window. We don't want this because we're doing on demand ISR, and the html file should theoretically always be in sync with the CMS content if we do it right.

In my middleware.ts:

export function middleware(request: NextRequest) {
  const response = NextResponse.next();
    response.headers.set(
      'Cache-Control',
      'public, s-maxage=600, stale-while-revalidate'
    );

    return response;
}

So if I understand everything correctly (which I very well may not), the CDN will get a request from the client. If it is outside of the s-maxage window, it will serve that client the cached version because of stale-while-revalidate, and then go and ask nextjs for fresh content. Next.js will respond with the html file for that path which should be either exactly the same if the content hasn't been edited, or a fresh html file that was regened thanks to on demand ISR.

I am still figuring all of this out so please don't take any of this as gospel, it's more to throw it out and see if anyone comments.

[jamesopti]

I'm curious if we can use this caching technique to persist pages across deploys while revalidating them in the background via dynamic ISR.

For us it's much more important for the page to load instantly than to have it be always perfectly up to date. Refreshing in the background for the next requester is just fine.

[MartinDavi]

We're going to probably clear the CDN and regenerate all pages each deploy, but theoretically if you did neither:

• CDN would continue serving cached content.
• Once outside the s-maxage window it would phone home.
• If using the revalidate prop in getStaticProps, nextjs would regen the html file at that first request, which would use the code from the latest deploy.

Right? I'm thinking out loud but wouldn't that work?

[MartinDavi]

Hey @jamesopti it seems you're unable to set Cache-Control headers in the middleware. I can set other custom headers, but Cache-Control either gets ignored or overwritten by next.js somewhere down the line.

My comments here are moot, and I'm back to the drawing board.

[mrkaluzny]

@MartinDavi I'm experiencing the same issue, but using Digital Ocean's App Platform where CloudFlare cache is added on top. ISR works in the background as expected but won't clean the cache, tried the same approach as you did, let me know if you have any other ideas, I would love to get this working :)

Which version of Next.js are you using?

[ericdrosas87]

I think I'm seeing this same behavior in our Next.js v13.0.1 web app. We're using Google Cloud's Cloud CDN which caches the frontend and backend responses and so far our only lead is to disable the CDNs. Has this been addressed by Vercel?

[mrkaluzny]

Nope, I talked to Digital Ocean team about the issue, and basically CDNs seem to overwrite the cache headers, our solution was moving back to Vercel platform.

It's not technically a Next issue -> it's more of an infrastructure issue.

[MuhammadTalhaKhan007]

I was also facing the same issue and I ended up adding just one line export const revalidate = 0 at the start of each server side page and my issue resolved.

[yongshun950824]

Have the similar issue as well when the infra migrated from CloudFlare to Imperva. After that, when browse to certain path such as: /login, it is not showing the HTML but something like minified JS script (believe that it should be proceed by Next.JS) with the response headers as:

cache-control: s-maxage=31536000, public
content-type: text/x-component

Suspect that this is more likely that some CDNs are not handle the caching for the webpages in Next.JS application correctly. Especially that we try also when bypass the mentioned CDN (so that to access to the ELB & server that host the application), the reported issue is not happening.

At this moment, we asked Imperva side to bypass certain pages in CDN but mentioned that this is not a good approach.

[leerob]

It's now possible to customize cache-control as of v14.2.10. Please upgrade and give it a try!

#69802
Related: #70674

[mrkaluzny]

@leerob Thank you for this update 👍