Docs for custom generated files (#994)

* add info on custom `llms.txt` files

* add llms-full.txt

* add sitemap and robots.txt

* Update llmstxt.mdx

Author: ethanpalm

ai/llmstxt.mdx

@@ -6,22 +6,20 @@ icon: "file-code"
 
 import { PreviewButton } from "/snippets/previewbutton.jsx"
 
-The [llms.txt file](https://llmstxt.org) is an industry standard that helps LLMs index content more efficiently, similar to how a sitemap helps search engines.
+The [llms.txt file](https://llmstxt.org) is an industry standard that helps LLMs index content more efficiently, similar to how a sitemap helps search engines. AI tools can use this file to understand your documentation structure and find content relevant to user queries.
 
-Mintlify automatically hosts an `llms.txt` file at the root of your project that lists all available pages in your documentation. AI tools can use this file to understand your documentation structure and find relevant content to user prompts.
+Mintlify automatically hosts an `llms.txt` file at the root of your project that lists all available pages in your documentation. This file is always up to date and requires zero maintenance. You can optionally add a custom `llms.txt` file to the root of your project.
 
 View your `llms.txt` by appending `/llms.txt` to your documentation site's URL.
 
-<PreviewButton href="https://mintlify.com/docs/llms.txt">Open the llms.txt for this site</PreviewButton>
-
-Your site's `llms.txt` is always up to date and requires zero maintenance.
+<PreviewButton href="https://mintlify.com/docs/llms.txt">Open the llms.txt for this site.</PreviewButton>
 
 ## llms.txt structure
 
 An `llms.txt` file is a plain Markdown file that contains:
 
-- **Site title** as an H1 heading. This is the only required section of an `llms.txt`.
-- **Structured content sections** with descriptive links to key pages.
+- **Site title** as an H1 heading.
+- **Structured content sections** with links and a description of each page in your documentation.
 
 ```mdx Example llms.txt
 # Example product docs
@@ -34,14 +32,18 @@ An `llms.txt` file is a plain Markdown file that contains:
 - [API](https://example.com/docs/api): Endpoint list and usage
 ```
 
-This structured approach allows LLMs to quickly process your documentation hierarchy and locate relevant content for user queries, improving the accuracy and speed of AI-assisted documentation searches.
+This structured approach allows LLMs to efficiently process your documentation at a high level and locate relevant content for user queries, improving the accuracy and speed of AI-assisted documentation searches.
 
 ## llms-full.txt
 
-The `llms-full.txt` file combines your entire documentation site into a single file as context for AI tools and is indexed by LLM traffic. Users can paste a single URL as context for AI tools for more relevant and accurate responses.
+The `llms-full.txt` file combines your entire documentation site into a single file as context for AI tools and is indexed by LLM traffic.
 
 Mintlify automatically hosts an `llms-full.txt` file at the root of your project. View your `llms-full.txt` by appending `/llms-full.txt` to your documentation site's URL.
 
-<PreviewButton href="https://mintlify.com/docs/llms-full.txt">Open the llms-full.txt for this site</PreviewButton>
+<PreviewButton href="https://mintlify.com/docs/llms-full.txt">Open the llms-full.txt for this site.</PreviewButton>
+
+## Custom files
+
+To add a custom `llms.txt` or `llms-full.txt` file, create an `llms.txt` or `llms-full.txt` file at the root of your project. Adding a custom file will override the automatically generated file of the same name. If you delete a custom file, the default file will be used again.
 
-Your site's `llms-full.txt` is always up to date and requires zero maintenance.
+Your custom `llms.txt` or `llms-full.txt` file must have a site title as an H1 heading. Other content is optional. See [Format](https://llmstxt.org/#format) in the `llms.txt` specification for more information on optional sections and best practices.

settings/seo.mdx

@@ -138,9 +138,9 @@ You can preview how your meta tags will appear on different platforms using [met
 }
 ```
 
-## Sitemaps
+## Sitemaps and robots.txt files
 
-Mintlify automatically generates a `sitemap.xml` file and a `robots.txt` file. Simply navigate to `/sitemap.xml` to see your sitemap.
+Mintlify automatically generates a `sitemap.xml` file and a `robots.txt` file. You can view your sitemap by appending `/sitemap.xml` to your documentation site's URL.
 
 Only pages included in your `docs.json` are included by default. To include hidden links, add `seo.indexing` to your `docs.json`:
 
@@ -150,6 +150,10 @@ Only pages included in your `docs.json` are included by default. To include hidd
 }
 ```
 
+### Custom sitemaps and robots.txt files
+
+To add a custom `sitemap.xml` or `robots.txt` file, create a `sitemap.xml` or `robots.txt` file at the root of your project. Adding a custom file will override the automatically generated file of the same name. If you delete a custom file, the default file will be used again.
+
 ## Disabling indexing
 
 If you want to stop a page from being indexed by search engines, you can include the following in the [frontmatter](pages) of your page: