Theme is missing robots.txt

[tzach]

All docs sites are missing robots.txt, which hurt the site SEO
Example: https://www.scylladb.com/robots.txt

[dgarcia360]

The main purpose of robots.txt is to prevent certain groups of pages from being crawled. If it’s not defined, crawlers assume they are allowed to crawl and potentially index all public content.

Question: Are there specific groups of pages we want to stop crawlers from accessing?

If we want to prioritize adding robots.txt support, we could include a default configuration in the theme, while allowing individual projects to override it.

[tzach]

You are right, its probably not very urgent.
I have seen some recommendation for it, but not an explicit Google recondition to use it.
https://developers.google.com/search/docs/crawling-indexing/robots/intro