Caching in Dev Mode

[mdugue]

Summary

Hi, I would like to use built-in caching in npm dev mode. In my project I have a Server Component which is querying GPT API. The API response is slow and billed on a per request basis, so quering every time HMR kicks in is not a convenient solution. I can imagine that many similar situation exists, where requests are not "cheap" regarding time or money.

For me, neither app dir revalidate nor react cache() nor some custom runtime-cache would work during development (though perfectly once built for production).

I found in https://stackoverflow.com/questions/69388338/nextjs-using-a-memory-cache-in-dev-does-not-work the hint to build a custom file based "cache" but was wondering if there is a more "idiomatic nextsjs" approach.

Is there anything I am missing? Any hints?

Additional information

Operating System:
      Platform: darwin
      Arch: arm64
      Version: Darwin Kernel Version 22.4.0: Mon Mar  6 20:59:28 PST 2023; root:xnu-8796.101.5~3/RELEASE_ARM64_T6000
    Binaries:
      Node: 16.18.1
      npm: 8.19.2
      Yarn: 1.22.19
      pnpm: 8.2.0
    Relevant packages:
      next: 13.3.0
      eslint-config-next: 13.3.0
      react: 18.2.0
      react-dom: 18.2.0

Example

import 'server-only'
import { Configuration, OpenAIApi } from 'openai'

const openai = new OpenAIApi(new Configuration({apiKey: process.env.OPENAI_API_KEY}))

// Server Component rendering a TLDR via GPT for a given text
export default async function GptTldr(props: { text: string }) {
	const response = await openai.createCompletion({
		model: 'text-davinci-003',
		prompt: `${props.text} tl;dr`,
		temperature: 0.7,
		max_tokens: 372,
		top_p: 1,
		frequency_penalty: 0,
		presence_penalty: 1,
	})
	return <>{response.data.choices[0].text}</>
}

[mdugue]

did this problem never arise anybody else or is my issue unclear 🤔?

[dsuttonpreece]

I'm also having this issue. Every time the next page is rebuild it fires a new request irrespective of cache settings. I understand why they do this in development mode, but would be good if you could force caching across page builds.

[mdugue]

Hey @dsuttonpreece, welcome on board 😅. I was wondering why so little people seem to stumble upon this topic. Do you have any plan already how you want to solve this in your case?

[dsuttonpreece]

Unfortunately I don't have a workaround. The api I'm trying to use offers a free mock api, so I'm just switching to that for local dev. Then going to use SSR caching in prod to try limit the number of calls we use. Luckily the data we're interested in from the api doesn't change very often so I don't think it's going to be a huge issue in prod.

Would love to hear if you come up with something though!

[VogelVlug]

Would also love a solution for this. Maybe a flag in the config to turn it on/off?

[johnsimeroth]

Looks like this topic hasn't received any attention from the next team yet, but I would also like a way to do this.

[quintesse]

I'd like to have a solution too. In our case it's not solely about caching though, it's about trying to create a single worker thread to do specific work. Firing up multiple versions of that can cause serious issues. But if we can't create singletons, we can't reliably determine if there's only a single worker or not.

[zoul]

Can you do something like locking a file exclusively and the first worker that succeeds in acquiring the lock may continue?

[quintesse]

thing is that AFAIK on unix-like systems there's no way you can "lock" a file. There are other ways to get "locks" but then we'd have to write specific code for specific OSes which seems like a bit too much work for something so basic.

[zoul]

FWIW, I have created a custom memoization function that can be used to turn any other function to its cached variant:

import { Cache } from "file-system-cache";
import crypto from "crypto";

const sharedCache = new Cache({ ttl: 60 * 5 });

type Producer<T> = (...args: any[]) => Promise<T>;

export function memoize<ReturnType, F extends Producer<ReturnType>>(
  originalFunction: F,
  key: string = originalFunction.name
): F {
  if (!key) {
    console.error("Cache key is empty, things will probably break.");
  }
  const memoized = async (...args: Parameters<F>): Promise<ReturnType> => {
    const compoundKey = hashDigest([key, ...args]);
    const cachedValue = await sharedCache.get(compoundKey);
    if (cachedValue) {
      return cachedValue;
    } else {
      const freshValue = await originalFunction(...args);
      sharedCache.set(compoundKey, freshValue);
      return freshValue;
    }
  };
  return memoized as F;
}

function hashDigest(params: string[]): string {
  const shasum = crypto.createHash("sha1");
  shasum.update(params.join(":"));
  return shasum.digest("hex").slice(0, 10);
}

It’s used like this:

const originalFunction = /* call some API or whatever */
const cached = memoize(originalFunction); // now use this instead of originalFunction

I would still prefer some built-in solution in Next, though.

[AlexLomm]

Would be nice to make the memoize function have the same signature to react's cache, that way something like this would become possible:

import { cache as reactCache } from 'react';

// ...

// substitute the default `cache` with memoize in dev env
export const cache =
  process.env.NODE_ENV === 'production' ? reactCache : memoize;

[rszalski]

I'm in the same boat, using a Snowflake SDK client.

Tried following workarounds like this Prisma doc would be totally acceptable but this seems to no longer work. Not sure if this is due to App Router vs Page Router differences or simply something changed in one of the bigger Next.js releases.

Another use case, beyond cost (which I totally understand) is being rate limited. Assuming the reload does clear cache so that it doesn't have a memory leak, when dealing with connections it might still result in a client being blocked for abusing.

The fewer differences between environments the better IMO.

See also:

• https://stackoverflow.com/questions/75272877/how-to-prevent-next-js-from-instantiating-a-singleton-class-object-multiple-time
• https://stackoverflow.com/questions/64712126/nextjs-initialize-an-object-once
• Global variables #15054

[sudhir4ev]

I can confirm the solution prescribed in Prisma doc for preserving instances in development mode works only when using pages router. Its absolutely useless when using app router. 😠

[AaronLayton]

I have come across this as well, I can't really test the caching strategy in dev mode without somehow turning this on

[Andrewjeska]

Would also love this feature. I get 1000 API calls per day.

[AaronLayton]

@leerob this would be a great feature when developing locally so we can test things. (also a DevTools for NextJS would be great, hint hint 😉 )

[magaldima]

This would be very helpful!

[mix3d]

Throwing my hat in the ring that this would have saved a few hours of troubleshooting in my dev env. I want to test that the caching is working correctly, I shouldn't have to do that "in prod".

[laurencefass]

Is this possible in 14.1? Are there any other open related tickets or response from core devs?

[amirrezaDev1378]

This is really needed in development mode.

[cw-1988]

yep running into db timeout issues when adjusting my css...

[nhducit]

We have the official Redis custom cache handler, but we cannot use it in the local environment. How can we verify the caching work?

https://github.com/vercel/next.js/blob/canary/examples/cache-handler-redis/README.md

[AaronLayton]

All caching in NextJS 15 is opt-in now, so this will probably solve your issue
https://nextjs.org/blog/next-15-rc#caching-updates

[mix3d]

No, the issue is that we want to enable caching *in Dev mode, locally*. We shouldn't have to deploy just to confirm caching is working, or blow up a rate limited API while developing.

…

On Fri, May 24, 2024, 2:14 AM Aaron Layton ***@***.***> wrote: All caching in NextJS 15 is opt-in now, so this will probably solve your issue https://nextjs.org/blog/next-15-rc#caching-updates — Reply to this email directly, view it on GitHub <#48481 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AALH4WI6F6Y6SEKG5Z7Q7KDZD3ZHZAVCNFSM6AAAAAAXBFHZBCVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TKNBUGE2TM> . You are receiving this because you commented.Message ID: <vercel/next. ***@***.***>

[baba43]

I keep stumbling across this problem.

It means that you can't develop your application the way it should actually be developed. If you want to use caching sensibly, you can't do this because you can not test and you can not benefit from it.

For example, I don't want a token to be generated again and again to save the API. The normal cache would actually be ideal for this. Unfortunately, it doesn't work during development and so I have to find some stupid workarounds.

Caching should ALWAYS work locally. We theoretically have native functions in the browser to skip the cache. Where possible, you should use them.

For me, this (and this: #58736) has been one of the most important pain points when using Next.js because you write so much about caching and while I would love to use it properly, you make it so hard and annoying to actually implement it in a beneficial way.

[markedwards]

I'm confused. I have also set up the Redis cache handler pattern as recommended in docs, but enabled it in dev because I need to build and dev against the cache.

I see this message when starting next in dev mode:

Next.js does not use the cache in development mode. Use production mode to enable caching.

However, the cache is populated and used everywhere we're using unstable_cache(). I haven't tested fetch caching. So it does seem to be partially enabled in dev. Is there any documentation on what actually happens here? Does anyone know?

[Ynng]

I'd really appreciate this feature!
I only get 1000 API calls a month so I usually cache it for 1 hour, having the same caching in dev built in would be amazing

[AaronLayton]

using unstable_cache in dev mode works

[psiho]

Well, it seems it was some kind of glitch. I upgraded, deleted .next folder and started it again. unstable_cache works as expected now. Thanks for making me retry. I definitely want batching revalidateTag that arrived in 14.2.11

[psiho]

Weird. I'll check changelogs and try to figure out what is causing this.

Off-topic:

The one issue I have not yet solved in 14.2.13 is that I have a route handler for which revalidateTag() no longer works. This is a little bit of a weird case -- its a route which provides a streaming response for use with the EventStream API, and just before the stream closes the route makes a DB mutation and calls revalidateTag(). This stopped working in 14.2.11, obviously related to the changes to revalidateTag there. I have a feeling it's some race condition and the revalidation work gets killed now.

However, this is consistent in dev and prod builds, so I don't think it's any evidence of a change regarding unstable_cache support in dev mode. In all other cases, caching works normally for me in 14.2.13.

Many of mine stopped working too. Back to 14.2.5 version. Maybe related to this: #70446

[markedwards]

Many of mine stopped working too. Back to 14.2.5 version. Maybe related to this: #70446

Thanks for the pointer there, will track it.

[psiho]

The one issue I have not yet solved in 14.2.13 is that I have a route handler for which revalidateTag() no longer works. This is a little bit of a weird case -- its a route which provides a streaming response for use with the EventStream API, and just before the stream closes the route makes a DB mutation and calls revalidateTag(). This stopped working in 14.2.11, obviously related to the changes to revalidateTag there. I have a feeling it's some race condition and the revalidation work gets killed now.

However, this is consistent in dev and prod builds, so I don't think it's any evidence of a change regarding unstable_cache support in dev mode. In all other cases, caching works normally for me in 14.2.13.

Just to keep some answers here in this thread, I've detected exactly what does not work for my revalidateTag() on v14.2.11 and above. I'm using tRPC routes on my server side and revalidateTag() does not work there anymore. Tested even new v15 version and experimental useCache() and none work. Last working version i v 14.2.5. However, this is the case only with tRPC routes. If I create a classic Next route with just revalidateTag() and call it... it works. If I use server action with just revalidateTag() and call it from a button in frontend... it works. But calling server action from tRPC route also does not work. For me, only workarround I can think of is to create standard api route for revalidating cache and using fetch() in tRPC routes to call it, but it seems sub-optimal. Another alternative is to switch to server actions and next-safe-action which will give me 95% of what I need. So, I'm affraid I'm dropping tRPC after all :\

[psiho]

And just to close this part of thread, finally a solution. I was using unstable_httpBatchStreamLink and with streaming responses revalidateTag() does not work because headers are sent before this point. Changing it to httpBatchLink resolves the issue, although loosing streaming

[andrewLapshov]

Is there a solution to this topic?

[unstubbable]

Depends on your use case. One of the use cases that were discussed here is now handled by https://nextjs.org/docs/app/api-reference/next-config-js/serverComponentsHmrCache.

[daveycodez]

This feels..awkward

[VitaliyPotapov]

I'm using Next v15.2.1 and for me data cache works locally.
How I check:

1. Enable fetches logging in the next config:

     logging: {
       fetches: {},
     },

2. Create server component that sends fetch request with revalidate time 60 seconds:

   // page.tsx
   export default async function Page() {
     await fetch('https://example.com', { next: { revalidate: 60 } });
     return <h1>Hello</h1>;
   }

3. Start dev server:

   npm run dev
   

When I refresh the page by browser reload button, I see that the request to example.com is cached:

If I remove revalidate option from the fetch call, then page reload triggers real request every time:

// page.tsx
export default async function Page() {
  await fetch('https://example.com');
  return <h1>Hello</h1>;
}

[shide1989]

As mentioned above I think this works in page mode. not in app mode

[VitaliyPotapov]

I use app router mode.

[OwenMelbz]

i guess no movement on this? horrible when developing with slow endpoints and you cant enable it

[unstubbable]

How are you fetching those endpoints exactly? In server or client components? With fetch or with something else?

[OwenMelbz]

using fetch() in server components - obviously that means ugly page loads whilst the page is loading, at least in client components we can put loaders onto those components.

We wouldnt move all them to client components as this is needed for stuff like open-graph data and page titles etc.

At the moment we're using const revalidate = 60 and const dynamic = 'force-static' for prod, which is fine - but during development its painfully slow

[unstubbable]

Those fetches should be cached during HMR updates though. Is that not sufficient for your dev workflow? Are you doing full page reloads?

[OwenMelbz]

If its something stored in useState then sure HMR works sometimes, obviously if there's a syntax error, or updating some global context or non-react files such as http clients etc which require full page reloads it doesnt work - but then when was testing the UX between pages - trying to share data to make sure we don't need to refetch certain things e.g. say we've got a user profile with several pages - why fetch the user again when you change between their profile pages - but when you use SSR this is more tricky.

Plus I actually spent a good few hours debugging an issue which was caused by not having caching on dev - the prod site was acting differently which I never saw in dev and cost time, I visited a page that had ISR on it, which triggered a background job, when the websocket event came in to say it had finished, i used router.reload() but I didnt see the change on prod - but i saw it on dev - this is because ISR had cached the page before the job ran and was just delivering the cached page, instead I had to then make an api route and call revalidatePath before calling router.reload() - so was just a pain. When coming from a background of using established tools like Laravel these things feel like they're just missed