Add ability to encrypt NEXT_DATA in html, to avoid data being scraped

[ndeviant]

I've found a way to encrypt data by overriding NextScripts method, which a hacky way. But I cant really find a place where can I add decryption step in client.js file. Can we have this opportunity out of the box? It could be disabled by default, and enable if user set some env variable, NEXT_DATA_SECRET foe example.

[rafaelalmeidatk]

I haven't tested it but if this data you want to encrypt is the one you return from gSP/gSSP/gIP, you could do it in directly in these methods.

That said, even if NEXT_DATA is encrypted you would have to decrypt it in the client-side, and if someone really wants to scrap your app they will figure out the method and scrap it anyway

[ndeviant]

Yep, I understand that, also they could scrap some data from the content itself, as it usually happends. But you may have something sensitive in props which is not gonna be rendered in hmtl, and having at least some protection would be nice. They'll have to go through js files in order to find a secret, and in minified code it would be a torture, I hope :)
At least make a life harder for scrapers than find id="__NEXT_DATA__" >>> JSON.parse

[blech75]

At least make a life harder for scrapers than find id="__NEXT_DATA__" >>> JSON.parse

@ndeviant agreed! see my comment below.

[...] having at least some protection would be nice.

it's not really about "protecting" content, imo. it's about making it more opaque. imo, __NEXT__DATA__ is akin to a private method; it exists only for Next.js and its implementation for hydrating client-side state (right?). others should be unconcerned with it and should ignore it. (of course, this being the web, we can't directly prevent others from ignoring it.)

my point is, while advertising this private data as <script type="application/json"> is correct and honest, it's perhaps a little too revealing. we can't rely on bots/parsers to ignore/disregard this tag. it's effectively plain-text, so its content is scraped and interpreted in whichever way the bot desires. let's keep it private.

/cc @leerob

[hugejile]

It's very important to encrypt sensitive data in some cases, for example, client side js rendering user's account, access token to avoid spider's Crawling, proxy caches.

[leerob]

Correct! If you're loading sensitive data, you should likely use client-side rendering (e.g. fetching in useEffect) instead of static-site generation. Encrypting the contents of your pre-rendered HTML is not the correct solution.

[Amorem]

Hi Lee,
Client-side rendering brings other problem, exposing APIs...
Can you develop why encrypting/compressing NEXT_DATA is not a correct solution ?
Thanks !

[DennieMello]

We have another problem, Google indexes NEXT_DATA, displaying warnings in the console, besides, the effect for SEO is not clear from repeating all the text on the page.

[leerob]

@Amorem You can mask external URLs using API routes as a proxy.

We have another problem, Google indexes NEXT_DATA, displaying warnings in the console

I don't believe this is accurate. Could you provide more info?

[blech75]

We have another problem, Google indexes NEXT_DATA, displaying warnings in the console

I don't believe this is accurate. Could you provide more info?

@leerob: a few years ago i observed bots crawling path-like strings in __NEXT_DATA__ JSON.

first, with path-like strings. one of the bits of data, nested deep within __NEXT_DATA__ looked like this:

{
  "id": "/foobar-content-item-123",
  "type": "teaser",
  "data": { /* ... */ }
}

/foobar-content-item-123 was an internal ID and completely unused/exposed anywhere else in our content, let alone linked on our public site. regardless, we ended up getting tons of requests to that path on our domain, which generated tons of traffic/load (😠), to which we responded with an HTTP 404. our quick fix was to hash that value, and, sure enough, the traffic went away. 💥

second, with URLs. we've occasionally had issues with editorial staff inadvertently using 'raw' (non-optimized/transformed) image URLs in various non-website contexts. we mostly addressed that via education, but a non-trivial amount of raw image URL access remained. 🤔

recently, we noticed the billing for our image CDN (Cloudinary) was getting a bit (too) high, so we decided to revisit this issue. we remembered that raw image URLs are present in __NEXT_DATA__, which could (probably?) explain why the raw image URL access persisted. 😞 (we dynamically add context-appropriate transforms in our front-end code, so we need the original URLs.)

(image requests don't hit our infra, so we can't directly see the access logs/patterns. Cloudinary's reporting is very coarse, so that doesn't help, either. i expect we'll eventually engage with their support to see if we can get more details.)

sorry that both of these are a bit anecdotal with no publicly visible results of the crawl. we are in process of digging deeper into the CDN image URL issue now, which is why i'm here. :-)

note that our particular issue is not necessarily concerned with leaking sensitive/secret data in __NEXT_DATA__; we've (thankfully) got that covered! 👍

what would help us, is if __NEXT_DATA__ were a bit more opaque in-page. my first inclination, of course, would be to base64 encode it, but that would increase the payload size. (maybe rot13? 🤣) maybe such encoding would be opt-in or a dev/prod switch? i have yet to investigate how __NEXT_DATA__ is assembled and parsed, or if it's possible to do such encoding as an "extension" without touching next's core.

[pateljoel]

why NEXT_DATA exists? can't there be an option to bake data into HTML in nextjs?

[leerob]

#15117

[hugejile]

OK, 2 years passed, there's still no official solution, I wrote a package for myself in 4 hours, encrypt props with des or base64, it's not a complete solution, but solved my problem! https://github.com/hugejile/with-next-encrypt

[greatcrypter]

that is useless

[leerob]

If you are still looking for a solution to this, we recommend incrementally adopting the Next.js App Router, which allows you to keep this entire payload on the server (in a Server Component).

[LukasBombach]

@leerob We tried this for SEO purposes. Instead of a large chunk of NEXT_DATA, we now get a bunch of chunks that contain __next_f.push wherein we find similarly duplicated contents. #42170 comes to the same conclusion.