Dynamic Sitemap Updating in Production

[joonarafael]

Summary

Hello,

I was wondering if it's possible to update the sitemap.xml and/or robots.txt file(s) dynamically (through sitemap.ts & robots.ts) after build process, in production?

Say we have blog posts or news articles, for example, and as more posts are published, the "list of valid URLs" (or something similar) would update. How would I then update the sitemap? Is it possible?

My idea is to add two endpoints (with proper authentication) that, when called, regenerate said files for my application. Initial testing with this kind of approach did not work. My endpoints called the real functions to generate sitemap.xml / robots.txt but nothing was updated.

Currently I'm just rebuilding the complete application again every 24h through a GH Actions Cron job. The cron job could instead of rebuild just call the endpoints.

Additional information

No response

Example

No response

[icyJoseph]

According to the docs, https://nextjs.org/docs/app/api-reference/file-conventions/metadata/sitemap:

sitemap.js is a special Route Handlers that is cached by default unless it uses a dynamic function or dynamic config option.

So you should be able to export one of these, https://nextjs.org/docs/app/api-reference/file-conventions/route-segment-config, such as revalidate, to have it regenerate at runtime.

Alternatively, as the docs say, a sitemap file is a special Route Handler, so you could just make a Router Handler, that returns XML, and revalidates every now and then. #52831 - though that shouldn't be necessary.

I'll run a couple of local tests with revalidate to see it if works, but it should. Confirmed, it does re-run when you export something like export const revalidate = 30 (every 30 seconds).

[joonarafael]

Thank you so much! I'll give it a shot.

[icyJoseph]

Awesome. Do remember that this revalidation uses stale-while-revalidate, which means that a request must arrive after the time you set, to get the revalidation triggered.

[gmoniava]

sitemap.js is a special Route Handlers that is cached by default unless it uses a dynamic function or dynamic config option.

The link "dynamic function" link above does not reference to a valid section anymore, could you update the link, or maybe mention what those dynamic functions were?