Feature Request: "RAG-Ready" Documentation

[Pavdig]

Hello team!

The Problem:
I've been trying to load the SillyTavern documentation into Open WebUI's knowledge base (RAG), but the current structure makes it difficult for LLMs to parse context correctly due to:

1. Nested directories losing context when flattened.
2. Duplicate filenames (e.g., multiple index.md files).
3. YAML frontmatter and Docusaurus tags (!!!) adding token noise.

The Suggestion:
It would be great to have a "clean" version of the SillyTavern docs for RAG users.

Proof of Concept:
I whacked together a workflow that flattens filenames (to e.g., SillyTavern_Installation_index.md) and strips the metadata using sed: Pavdig/SillyTavern-Docs-RAG.

Maybe something similar could be made, but officially so that users could simply download the ZIP?

[Cohee1207]

Curiously enough, I thought about the same thing after noticing the Tailwind situation. I think a workflow that generates a said file and places it alongside the docs website (e.g. https://docs.sillytavern.app/llms.txt) could work for us. You think it should be multiple files?

[Pavdig]

With my short period of using RAGs on Open WebUI and MCP servers, I believe keeping the files separated (but with correct names) is the best approach for RAG/Vector Databases for a few reasons:

Semantic Chunking:

• Vector databases chop text into chunks. With separate files, "hard breaks" between topics are natural. With a single monolithic file, you run the risk of a chunk bleeding from the end of one topic (e.g. "Installation Windows") into the start of another (e.g. "Installation Docker") without the AI realizing the context switched.

Context via Filenames:

• With a correct filename (e.g. SillyTavern_extensions_alltalk.md), the file itself carries its own "index" or context, so the AI knows exactly what feature the text belongs to without needing a master index file.

Citations & Debugging:

• When the AI retrieves information, it can cite the specific source file (e.g. SillyTavern_Extensions_TTS.md). If everything is in one giant file, the source is always just "Manual.md", which makes it hard to verify where the info came from.

A single monolithic file is really only suitable for "Long Context" models (dumping the whole manual into the prompt), but for an actual search/retrieval system, granular files seem to work better for me.

I lately tested cheatengine-mcp-bridge where the author used three doc files with the CE LUA documentation above 200KB in size. It works fine with Gemini 3 Pro, but after splitting it into smaller segments, it worked better, with fewer errors by executing functions, and faster.

But this is all just with... IMHO.

Do what you think is best for everyone. Everyone else has their own ways anyway. ;)

[Pavdig]

Just a quick update on this topic.

I kept tinkering lately with my repo Pavdig/SillyTavern-Docs-RAG and managed to get the automation working properly now.

I also added a python script that bundles everything into single files (llms.txt and llms.json) alongside the standard markdown.

Feel free to use the script and workflows if you want it for the main ST docs, but otherwise, the repo is working fine for anyone who needs it.

[Cohee1207]

I pinned this issue just to have it handy for lurkers. I can also fork your repo under the official ST org and link it in the index.md so that it gets more recognition.

[Pavdig]

Sounds good to me.

My intention was always for this to be a Proof of Concept to demonstrate the workflow, rather than maintaining a separate building block long-term.

So, please go ahead. Fork it, take it or do whatever you like with it. 🤘