Sitemap index

[omerp-explorium]

Goals

When using the multiple sitmap approach, Nextjs should produce a root sitemap that will point to all sitemap urlsets e.g /sitemap.xml

Non-Goals

Background

As you can see in the image, we should also have this sitemap pointing to multiple sitemaps

Taken from Google documentation on large sitemaps

Proposal

I propose Nextjs will keep track of all the generated sitemaps, and generate the proposed sitemap of sitemaps xml.

What do you think?

[Fedeorlandau]

I made a PR that adds support for sitemap-index.ts which you can use to dynamically add your sitemaps to the sitemap index file.

#61391

[MartineBoulanger]

Thank you for this! I was looking for a solution since this sitemapIndex() is still a WIP, and your solution works perfectly!

[rkuang9]

Wouldn't sitemaps generated this way be ignored by crawlers? If it's /games/sitemap/1.xml, the sitemap may only point to link within its current level, which is /games/sitemap.

[tomcru]

Not sure I understand your question @rkuang9
The sitemaps aren't ignored by crawlers and all properly processed by Google, etc.

Our link structure is like this atm and the sitemap index is on /sitemaps/index.xml

[rkuang9]

It's a matter of directory levels. According to https://developers.google.com/search/docs/crawling-indexing/sitemaps/large-sitemaps#sitemap-index-best-practices and https://www.sitemaps.org/protocol.html#location, your links in sitemaps/games/1.xml will only be recognized if they point to sitemaps/games. Of course this directory is dedicated to sitemaps and not game content. It seems the way Next.js handles sitemaps do not fit sitemap standards.

[tomcru]

The first article linked is only talking about sitemap indexes, which is correct in our case as all sitemap indexes are in the /sitemaps/ directory. It's saying, however, that you shouldn't have your index at /sitemaps/index.xml and then link to another sitemap that's in another directory like /somewhere-else/sitemap.xml.

I'm also not sure if link 2 is enforced, I've never run into any problem. It would mean I could only link to game pages that are in /sitemaps/games/.... But this is not the case as Google is clearly recognising the sitemaps and their included urls (on /games/...)

Also this has nothing to do with next.js, as you can choose the directory/path you want your sitemaps or indexes to be yourself.

[antoinerousseau]

I don't understand the utility of generateSitemaps without a sitemaps index, am I missing something?

[hermbo]

Agreed, seems like this should just be automatic. Then you only need to provide google with the index file.

[CaptainN]

Next.js has been filling up with half baked "features" - I'm not sure why...

[shadee22]

Yes, we also need a feature to generate a NEXTJS sitemap index. Right now, it only generates URLs.

[antoinerousseau]

Meanwhile, I went with my own custom index at src/app/sitemap.xml/route.ts:

import { env } from "~/env";
import { getLastMod } from "~/lib/sitemap";
import { generateSitemaps as getArtistSitemaps } from "../api/sitemaps/artists/sitemap";
import { generateSitemaps as getEventSitemaps } from "../api/sitemaps/events/sitemap";
import { generateSitemaps as getVenueSitemaps } from "../api/sitemaps/venues/sitemap";

export const revalidate = 3600; // cache for 1 hour

function getFileName(id?: number) {
  if (id == null) {
    return "sitemap.xml";
  }
  /**
   * https://nextjs.org/docs/app/api-reference/file-conventions/metadata/sitemap#generating-multiple-sitemaps
   * In production, your generated sitemaps will be available at /.../sitemap/[id].xml. For example, /product/sitemap/1.xml.
   * In development, you can view the generated sitemap on /.../sitemap.xml/[id]. For example, /product/sitemap.xml/1. This difference is temporary and will follow the production format.
   * See the generateSitemaps API reference for more information.
   */
  return env.VERCEL_ENV === "development"
    ? `sitemap.xml/${id}`
    : `sitemap/${id}.xml`;
}
function getLoc(path: string, id?: number) {
  return `${env.URL}/api/sitemaps/${path}/${getFileName(id)}`;
}
function getSitemap(path: string, id?: number) {
  return /* XML */ `<sitemap><loc>${getLoc(path, id)}</loc><lastmod>${getLastMod()}</lastmod></sitemap>`;
}
function getSitemaps(ids: { id: number }[], path: string) {
  return ids.map(({ id }) => getSitemap(path, id)).join("");
}

export async function GET() {
  const xml = /* XML */ `<?xml version="1.0" encoding="UTF-8"?>
    <sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
      ${getSitemap("cities")}
      ${getSitemaps(await getEventSitemaps(), "events")}
      ${getSitemaps(await getArtistSitemaps(), "artists")}
      ${getSitemaps(await getVenueSitemaps(), "venues")}
    </sitemapindex>
  `;

  return new Response(xml, {
    headers: {
      "Content-Type": "application/xml",
    },
  });
}

[SemihOzden]

I have implemented same logic in my next.js project and It worked. Thanks in advance.

[jonathanlal]

legend thank you - elegant solution

[Sammeeey]

agreed, we need proper sitemap generation (incl. sitemap-index)

[pondorasti]

For anyone looking for a alternatives, I would recommend the server side utils from https://github.com/iamvishnusankar/next-sitemap?tab=readme-ov-file#server-side-sitemap-getserversidesitemap. You can plug them in into a route handler and it probably has all the features you would ever need.

[rtorrente]

I don't really understand the goal of generateSitemaps if it's just to generate the available url without indexing them.
Paginated sitemap are growing indefinitly on a news site as post are created. It's just not possible to add new sitemap in google console because there is no alert like "New sitemap generated". Especially since, when using generateSitemap, the basic sitemap.xml file returns a 404, where indexes would be more appropriate. It seems more like a bug than a new feature at this level.

At least, MetadataRoute.Sitemap have to accept both url or sitemap link to allow dev to reference another sitemap file and generate an xml <url></url> or <sitemap></sitemap> accordingly.

[tomcru]

You create a Sitemap index that links to all your sitemaps - then you just add the index to google search console and it's able to find all of them. Like so: https://gamegator.net/sitemaps/index.xml (if that's what you mean)

[rtorrente]

Yes but it need to create a route and generate manually XML and we can't use he sitemap.ts at the root app/ directory

Because sitemap.ts only generate <url> XML tag and not <sitemap> XML tag

[heymartinadams]

Maybe this can help @rtorrente https://raddy.dev/blog/multiple-sitemaps-in-nextjs-app-router-sitemap-index/

[dsbrianwebster]

Sharing the approach we’ve been using:

We have a script that runs at the end of our build process. This script scans the app directory for all instances of sitemap.ts, resolves their URLs based on relative paths, and generates a single sitemap index at the root.

We like this approach because, once it’s in place, all we need to do is create sitemap.ts instances following Next.js file conventions, with no additional work required. It’s also copy n' paste reusable across projects. Additionally, it even handles sitemap.ts files that use generateSitemaps and adds all the generated sitemaps to the index.

Here’s how you might integrate it into your package.json:

  "scripts": {
    "build": "next build && pnpm sitemap:generate",
    "sitemap:generate": "dotenv -c -- npx tsx ./scripts/sitemap.ts",
  },

And here is the script...

import { promises as fs } from 'fs';
import path from 'path';

import { SITE_URL } from '~/app/_lib/constants';

/**
 * 🗺️ Generates the main sitemap index file.
 *
 * This function creates a sitemap index file by discovering and aggregating
 * sitemap files generated across the project. It is temporary until Next.js
 * offers native support for sitemap indexing.
 */

const ROOT_DIR = process.cwd();
const APP_DIR = path.join(ROOT_DIR, 'app');

async function main() {
  try {
    const sitemaps = await findSitemapFiles(APP_DIR);
    const SITEMAP_INDEX_PATH = path.join(ROOT_DIR, 'public', '/sitemap.xml');

    const createSitemap = (url: string) => /* XML */ `
      <sitemap>
        <loc>${url}</loc>
      </sitemap>`;

    const createSitemapIndex = (urls: string[]) => /* XML */ `
      <?xml version="1.0" encoding="UTF-8"?>
      <sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
        ${urls.map(createSitemap).join('')}
      </sitemapindex>`;

    const xml = createSitemapIndex(sitemaps);
    await fs.mkdir(path.dirname(SITEMAP_INDEX_PATH), { recursive: true });
    await fs.writeFile(SITEMAP_INDEX_PATH, xml.trim());

    console.log(`Sitemap index generated with ${sitemaps.length} entries.`);
  } catch (error) {
    console.error('Error generating sitemap:', error);
  }
}

/**
 * 🔎 Recursively finds 'sitemap.ts' files and extracts sitemap URLs.
 *
 * Traverses directories from the specified start point, looking for 'sitemap.ts' files.
 * If a file contains a `generateSitemaps` function, the function is invoked
 * to retrieve sitemap URLs. If absent, a default sitemap URL is constructed.
 *
 * Known Limitations:
 * - While this does consider Next.js conventions for folders like route groups, do not place
 *   'sitemap.ts' under folders like route groups untill the following seperate issue is resolved:
 *    https://github.com/vercel/next.js/issues/68403
 *
 * @param {string} dir - The base directory to begin searching for sitemap files.
 * @returns {Promise<string[]>} A promise that resolves to an array of sitemap URLs.
 */
async function findSitemapFiles(dir: string): Promise<string[]> {
  let sitemaps: string[] = [];
  try {
    const files = await fs.readdir(dir, { withFileTypes: true });
    for (const file of files) {
      const fullPath = path.join(dir, file.name);
      if (file.isDirectory()) {
        // ingore private folders since all its child segments are opted out of routing
        // https://nextjs.org/docs/app/getting-started/project-structure#route-groups-and-private-folders
        if (!file.name.startsWith('_')) {
          const nestedSitemaps = await findSitemapFiles(fullPath);
          sitemaps = sitemaps.concat(nestedSitemaps);
        }
      } else if (file.isFile() && file.name === 'sitemap.ts') {
        const relativePath = path
          .relative(APP_DIR, dir)
          .split(path.sep)
          .filter(
            (segment) =>
              // Exclude specific Next.js folder conventions that should not be part of the resolved path
              // https://nextjs.org/docs/app/getting-started/project-structure#route-groups-and-private-folders
              // https://nextjs.org/docs/app/getting-started/project-structure#parallel-and-intercepted-routes
              // https://nextjs.org/docs/app/getting-started/project-structure#dynamic-routes
              !/^[(\[@]/.test(segment),
          )
          .join('/');
        const sitemapFile = await import(fullPath);
        if (sitemapFile.generateSitemaps) {
          const result = await sitemapFile.generateSitemaps();
          result.forEach((item: { id: number }) => {
            sitemaps.push(`${SITE_URL}/${relativePath}/sitemap/${item.id}.xml`);
          });
        } else {
          sitemaps.push(`${SITE_URL}/${relativePath}/sitemap.xml`);
        }
      }
    }
  } catch (error) {
    console.error('Error finding sitemaps:', error);
  }
  return sitemaps;
}

main();

[Zenith2198]

This is a somewhat clean solution that worked for me. When deploying a NextJS application to Vercel, I had to change a few things.

I was unable to get the SITE_URL from my envs, and had to hardcode it as the production domain. This was fine, because I only really need the sitemap to be correct on production.

I also changed the package.json scripts as follows:

{
    "scripts": {
        "build": "next build",
        "postbuild": "npx tsx ./sitemap-index.ts"
    }
}

Loading dotenv in the script wasn't working for me, so I removed it. I also utilized postbuild to run the script.

You could consider making this into a standalone package. I think it would be very useful to people as an alternative to next-sitemap that integrates more cleanly with the built-in NextJS sitemap tools.

[Araon]

Ah yes, just what we needed—another 'half-baked feature' in Next.js that requires us to write 200 lines of custom scripts to make it actually useful. Love how we keep reinventing the wheel, but with extra steps. Can’t wait for the next ‘simplified’ feature that needs my entire engineering team to waste 1hr on.

[antoinerousseau]

What's the point of this comment? Is bitching really useful?

[programming-with-ia]

+1

[abel-officener]

+1