Proposal for Adding Sitemap Index in Next.js Root and Automatically Adding Pages and Sitemaps to Indexed Sitemap with robot.txt

[Shani-Sinojiya]

Goals

Proposal for Adding Sitemap Index in Next.js Root and Automatically Adding Pages and Sitemaps to Indexed Sitemap with robot.txt

Proposal

Introduction:
In modern web development, having a sitemap is crucial for search engine optimization (SEO) and better indexing by search engines. However, manually managing sitemaps and updating them whenever new pages are added can be cumbersome. This proposal suggests incorporating functionality into Next.js to automatically generate a sitemap index in the root directory and dynamically add pages and their corresponding sitemaps to this index.

1. Implementation Overview:

• Integration with Next.js Build Process: Integrate the sitemap generation process into Next.js build process, ensuring that whenever the application is built or deployed, the sitemap is updated accordingly.
• Dynamic Sitemap Generation: Implement logic to dynamically generate individual sitemaps for each page based on the pages present in the Next.js application.
• Sitemap Index Creation: Automatically generate a sitemap index file in the root directory of the Next.js application, which will reference all individual sitemaps.
• Automatic Update: Ensure that whenever a new page is added or removed from the Next.js application, the sitemap and sitemap index are automatically updated during the build process.

2. Technical Implementation:

• Next.js Plugin or Middleware: Develop a Next.js plugin or middleware responsible for handling the sitemap generation and update process. This plugin will be invoked during the build process to generate sitemaps and update the sitemap index.
• Dynamic Routing: Utilize Next.js dynamic routing capabilities to iterate through all pages in the application and generate corresponding sitemaps.
• XML Sitemap Generation: Generate XML-formatted sitemap files for each page containing relevant information such as URL, last modified date, change frequency, etc.
• Sitemap Index Generation: Generate a sitemap index file (sitemap.xml) that references all individual sitemaps created for each page.

3. Do not Remove

• Make sure this functionality is Required for Generating Sitemaps for other External sources, This Feature Required

Code Example

/** @type {import('next').NextConfig} */
const nextConfig = {
 // your configs....
 sitemap: {
  siteUrl: 'https://example.com',
  changefreq: 'daily',
  priority: 0.7,
  sitemapSize: 5000,
  generateRobotsTxt: true,
  exclude: ['/protected-page', '/awesome/secret-page'],
  alternateRefs: [
    {
      href: 'https://es.example.com',
      hreflang: 'es',
    },
    {
      href: 'https://fr.example.com',
      hreflang: 'fr',
    },
  ],
  // Default transformation function
  transform: async (config, path) => {
    return {
      loc: path, // => this will be exported as http(s)://<config.siteUrl>/<path>
      changefreq: config.changefreq,
      priority: config.priority,
      lastmod: config.autoLastmod ? new Date().toISOString() : undefined,
      alternateRefs: config.alternateRefs ?? [],
    }
  },
  additionalPaths: async (config) => [
    await config.transform(config, '/additional-page'),
  ],
  robotsTxtOptions: {
    policies: [
      {
        userAgent: '*',
        allow: '/',
      },
      {
        userAgent: 'test-bot',
        allow: ['/path', '/path-2'],
      },
      {
        userAgent: 'black-listed-bot',
        disallow: ['/sub-path-1', '/path-2'],
      },
    ],
    additionalSitemaps: [
      'https://example.com/my-custom-sitemap-1.xml',
      'https://example.com/my-custom-sitemap-2.xml',
      'https://example.com/my-custom-sitemap-3.xml',
    ],
  },
 }
};

module.exports = nextConfig;