[SEO] - How to use sitemap and robots with next-intl

[davidbourrel]

Hi !

I'm watching this video for SEO in Next.js 14 and i wonder how to use sitemap (43:30 in the video) and robots (50:40) with next-intl.

The sitemap.xml and the robots.txt have to be in the [locale] folder ?

How to set sitemap.xml and robots.txt proprely with the dynamic "[locale]" folder ? do you have any example ?

Thank you so much !

[amannn]

Next.js is currently missing support for hreflang in sitemap.xml AFAIK. For the time being I think you have to use a route handler and implement the sitemap by yourself.

What would you need you need from next-intl to implement robots.txt?

This being said, it would be cool to have a guide for this at some point. If someone has a working implementation, it would be cool if you could share it here.

[amannn]

@levipadre since you've asked a similar question in #264, do you by chance have a working implementation?

[levipadre]

@davidbourrel I'm using headers to get local information for robots.txt as it's in my root. Both are coming from a CMS's restapi:

async function getPageData() {
    const headersList = headers();
    const host = headersList.get('host') as string;
    const protocol = headersList.get('x-forwarded-proto') || 'http';

    const res = await fetch(`${protocol}://${host}/api/pages/robotstxt`);
    
    if (!res.ok) {
        throw new Error('Failed to fetch page data');
    }
   
    return res.json();
}
  
export default async function robots(): Promise<MetadataRoute.Robots> {
    const headersList = headers();
    const host = (headersList.get('host') || headersList.get('x-forwarded-host')) as string;
    const protocol = headersList.get('x-forwarded-proto') || 'http';

    const page = await getPageData();
    const rules = parseRulesString(page.properties.rules);
  
    return {
        rules: rules,
        sitemap: `${protocol}://${host}/sitemap.xml`
    };
}

My sitemap.xml is within [locale] and I use this code:

import { MetadataRoute } from 'next';
import { headers } from 'next/headers';
import { extractTLD } from '@/lib/extractTLD';

export default async function sitemap(): Promise<MetadataRoute.Sitemap> {
    const headersList = headers()
    const host = headersList.get('host') as string;
    const protocol = headersList.get('x-forwarded-proto') || 'http';
    const locale = extractTLD((headersList.get('host') || headersList.get('x-forwarded-host')) as string);

    const res = await fetch(`${protocol}://${host}/api/sitemap-xml`);
    const sitemap = await res.json();

    const urls = sitemap.map((url: {
        loc: string;
        lastMod: string;
        changeFreq: string;
        priority: number;
    }) => ({
        url: url.loc,
        lastModified: url.lastMod,
        changeFrequency: url.changeFreq,
        priority: url.priority,
    }))

    return urls;
}

[davidbourrel]

Hi ! Really thank you for your response ! I'll try something like this asap

[davidbourrel]

What do you use in your extractTLD file ?

[mashupmind]

To implement a sitemap.xml and robots.txt with dynamic localization (e.g., using [locale] in Next.js), you can follow these steps:

Sitemap.xml:
Create a Sitemap Generator Function:

Create a function to generate your sitemap dynamically based on the localized pages of your Next.js application. This function will generate the sitemap.xml content.
jsx
Copy code
// utils/sitemap.js
import { getAllLocales } from './locales'; // Function to get all supported locales

const generateSitemap = async () => {
const locales = await getAllLocales(); // Implement this function to get supported locales
const pages = []; // Add your localized page URLs here

const sitemap = <?xml version="1.0" encoding="UTF-8"?> <urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"> ${locales .map((locale) => pages.map((page) =>

${https://yourdomain.com/${locale}/${page}}
weekly
0.8

).join('')) .join('')} </urlset> ;

return sitemap;
};

export default generateSitemap;
Create an API Route for Sitemap:

Create an API route that will serve the dynamically generated sitemap.
jsx
Copy code
// pages/api/sitemap.xml
import generateSitemap from '../../utils/sitemap';

export default async function handler(req, res) {
const sitemap = await generateSitemap();
res.setHeader('Content-Type', 'text/xml');
res.write(sitemap);
res.end();
}
Submit Sitemap to Search Engines:

Once you've implemented the sitemap, submit it to search engines like Google Search Console.
Robots.txt:
Create a Robots.txt Generator Function:

Similarly, create a function to generate your robots.txt dynamically based on your localized pages.
jsx
Copy code
// utils/robots.js
import { getAllLocales } from './locales';

const generateRobotsTxt = async () => {
const locales = await getAllLocales();
const disallow = '/admin'; // Add any paths you want to disallow here

const robotsTxt = User-agent: * Disallow: ${disallow} ${locales.map((locale) =>Sitemap: https://yourdomain.com/${locale}/sitemap.xml`).join('\n')}
`;

return robotsTxt;
};

export default generateRobotsTxt;
Create an API Route for Robots.txt:

Create an API route to serve the dynamically generated robots.txt.
jsx
Copy code
// pages/api/robots.txt
import generateRobotsTxt from '../../utils/robots';

export default async function handler(req, res) {
const robotsTxt = await generateRobotsTxt();
res.setHeader('Content-Type', 'text/plain');
res.write(robotsTxt);
res.end();
}
Include in the Next.js App:

Make sure to include the generated robots.txt and sitemap.xml URLs in your Next.js app where appropriate, such as linking them in the header.
Remember to replace 'https://yourdomain.com' with your actual domain. Also, adjust the paths and logic according to your specific project structure and needs.

Certainly! For SEO, you may want to include meta tags in the head of your pages to provide additional information for search engines. Here's how you can add two common meta tags, title and description, to the head of your pages:

Modify the Head of Your Pages:
Update the Head in Your Layout Component:

jsx
Copy code
// components/Layout.js
import Head from 'next/head';

const Layout = ({ children }) => {
return (

<title>Your Site Title</title>

{/* Add any other SEO-related meta tags as needed */}

{children}

);
};

export default Layout;
Make sure to replace "Your Site Title" and "Your site description goes here." with your actual site title and description.

Include Layout in Your Pages:

jsx
Copy code
// pages/_app.js
import Layout from '../components/Layout';

function MyApp({ Component, pageProps }) {
return (

<Component {...pageProps} />

);
}

export default MyApp;
This assumes you have a Layout component that wraps your pages.

Additional Notes:
Dynamic SEO Tags:
If your SEO tags need to be dynamic based on the content of each page, you can pass them as props to your Layout component or set them within each individual page.

Locale-Specific SEO:
If you want to include locale-specific SEO information, you can customize the title and description tags based on the current locale.

Remember to adapt these examples to fit the specifics of your project. SEO best practices may vary, so consider other meta tags or elements that might be relevant to your content and audience.