Explore generating `/llms.txt` for our docs

[vytas7]

Generate /llms.txt for our docs site.

Is it possible to host that file on RtD root?
Is there an easy way to convert the generated Sphinx output (with all code and autodoc already included) to Markdown?

[vytas7]

Maybe this could be helpful? https://pypi.org/project/sphinx-markdown-builder/

[CaselIT]

interesting idea, but I'm not sure how useful it is for "current" ai tools

[vytas7]

It is a strange situation indeed, everyone is adding it including Claude themselves, but no major AI provider has confirmed to actually be scraping them... 🤯

[thefl0ur]

I took some time to explore the scope of this task.

In case of using mentioned package sphinx-markdown-builder, extra steps will definitely be required.
While it generates perfect docs in Markdown, there are a couple of pitfalls, from merging the results of markdown and html builds to generating /llms-full.txt with additional script.

Since this isn't a hot topic, I suggest looking into sphinx-llms-txt, which is currently pending (jdillard/sphinx-llms-txt#51) due block by this PR: liran-funaro/sphinx-markdown-builder#37.

Additionally, it seems that configuring the read the docs server will be needed for redirects like /llms.txt > /en/stable/llms.txt.

[CaselIT]

do we actually need to have this? I'm not sure how popular it is

[vytas7]

do we actually need to have this? I'm not sure how popular it is

@CaselIT This is an uncharted territory, opinions vary wildly. I renamed the issue to highlight its exploratory nature.