How to add robots.txt to your quarto website?

[kurianbenoy]

I was curious to know how is robots.txt be added for a quarto website? I noticed quarto website has a robots.txt file enabled. Can you share how it was added to quarto-website, I can prevent web crawling?

https://quarto.org/robots.txt

[cscheid]

Adding a robots.txt file to your repository should just work.

[kurianbenoy]

Thanks you @cscheid . Where can I find it quarto-website repositories?

I couldn't find it checking it in repositories:

https://github.com/quarto-dev/quarto-web
https://github.com/quarto-dev/quarto-cli

[cscheid]

That's not the only way one can do it (quarto-web does it differently). But if you do it in yours, it will work.

[danielvartan]

@cscheid do you know a way to add a disallow command to crawlers using _quarto.yml?

[mcanouil]

This is not a Quarto option/question.
You have to use robots.txt which is an heavily documented file.

Also, you have to list that file as your website resources, see https://quarto.org/docs/websites/website-tools.html#site-resources.