Update preview deployments docs to inform about x-robots-tag (cloudflare#25498)

Author: rita3ko

src/content/docs/pages/configuration/preview-deployments.mdx

@@ -67,3 +67,19 @@ Branch name aliases are lowercased and non-alphanumeric characters are replaced
 To view branch aliases within your Pages project, select **View build** for any preview deployment. **Deployment details** will display all aliases associated with that deployment.
 
 You can attach a Preview alias to a custom domain by [adding a custom domain to a branch](https://developers.cloudflare.com/pages/how-to/custom-branch-aliases/).
+
+## Preview indexing by search engines 
+
+To maintain a healthy SEO profile, it's vital to prevent search engines from finding duplicate content across the web. Because preview deployments are designed to be an exact replica of your production environment, they inherently create this exact situation. Cloudflare Pages by default ensures your search rankings are not harmed by these temporary previews.
+
+### X-Robots-Tag: noindex on Preview Deployments
+By default, every preview deployment generated by Cloudflare Pages includes the X-Robots-Tag: noindex HTTP response header. This header acts as a clear directive to search engine crawlers, instructing them to disregard the page and not include it in their search results.
+
+You can easily confirm that your preview deployments are correctly configured to block indexing. Run the following curl command in your terminal, replacing the placeholder with your unique preview URL:
+
+``` 
+curl -I https://<your-preview-url>.pages.dev
+```
+
+Inspect the output for the x-robots-tag: noindex line to verify that your preview site is not being indexed.
+