llms.txt being indexed by search engines and appearing in results #1567
Description
Tried searching for a query related to svelte today: "svelte remote function arguments" on several search engines (Google, Kagi, duckduckgo) and keep getting the llms.txt in the search results. Some of them show the title to be just "svelte.dev" while others (Google) have the proper file name listed.
Are these files even supposed to appear to the user given that they are intended to be read by non-humans?
The llms result was the first thing I clicked on in the results and I thought the site was broken until I saw the filename.
I am wondering if this is something that should be handled by the website owner in this case or if it is the search providers responsibility to purge from the results? I thought the files might be listed in the sites sitemap.xml but that does not appear to exist, at least not at svelte.dev/sitemap.xml
Is there anything to be done here to unlist these files from the search engines, I don't think these llms.txt links should appear in a normal search result.
So far I have managed to stumble upon the following links:
https://svelte.dev/docs/kit/remote-functions/llms.txt
https://svelte.dev/docs/svelte/getting-started/llms.txt