WIP Autorag

src/content/changelog/autorag/2025-04-07-autorag-open-beta.mdx

@@ -0,0 +1,21 @@
+---
+title: Create fully-managed RAG pipelines for your AI applications with AutoRAG
+description: AutoRAG lets you create fully-managed, retrieval-augmented generation (RAG) pipelines that continuously updates and scales on Cloudflare.
+date: 2025-04-07T6:00:00Z
+hidden: true
+---
+
+[AutoRAG](/autorag) is now in open beta, making it easy for you to build fully-managed retrieval-augmented generation (RAG) pipelines without managing infrasturcture. Just upload your docs to [R2](/r2/get-started/), and AutoRAG handles the rest: embeddings, indexing, retrieval, and response generation via API.
+
+![AutoRAG open beta demo](~/assets/images/changelog/autorag/autorag-open-beta.gif)
+
+With AutoRAG, you can:
+
+- **Customize your pipeline:** Choose from [Workers AI](/workers-ai) models, configure chunking strategies, edit system prompts, and more.
+- **Instant setup:** AutoRAG provisions everything you need from [Vectorize](/vectorize), [AI gateway](/ai-gateway), to pipeline logic for you, so you can go from zero to a working RAG pipeline in seconds.
+- **Keep your index fresh:** AutoRAG continuously syncs your index with your data source to ensure responses stay accurate and up to date.
+- **Ask questions:** Query your data and receive grounded responses via a [Workers binding](/autorag/usage/workers-binding/) or [API](/autorag/usage/rest-api/).
+
+Whether you're building internal tools, AI-powered search, or a support assistant, AutoRAG gets you from idea to deployment in minutes.
+
+Get started in the [Cloudflare dashboard](https://dash.cloudflare.com/?to=/:account/ai/autorag) or check out the [guide](/autorag/get-started/) for instructions on how to build your RAG pipeline today.

src/content/docs/autorag/concepts/how-autorag-works.mdx

@@ -0,0 +1,44 @@
+---
+pcx_content_type: concept
+title: How AutoRAG works
+sidebar:
+  order: 2
+---
+
+AutoRAG sets up and manages your RAG pipeline for you. It connects the tools needed for indexing, retrieval, and generation, and keeps everything up to date by syncing with your data with the index regularly. Once set up, AutoRAG indexes your content in the background and responds to queries in real time.
+
+AutoRAG consists of two core processes:
+
+- **Indexing:** An asynchronous background process that monitors your data source for changes and converts your data into vectors for search.
+- **Querying:** A synchronous process triggered by user queries. It retrieves the most relevant content and generates context-aware responses.
+
+## How indexing works
+
+Indexing begins automatically when you create an AutoRAG instance and connect a data source.
+
+Here is what happens during indexing:
+
+1. **Data ingestion:** AutoRAG reads from your connected data source.
+2. **Markdown conversion:** AutoRAG uses [Workers AI’s Markdown Conversion](/workers-ai/markdown-conversion/) to convert [supported data types](/autorag/configuration/data-source/) into structured Markdown. This ensures consistency across diverse file types. For images, Workers AI is used to perform object detection followed by vision-to-language transformation to convert images into Markdown text.
+3. **Chunking:** The extracted text is [chunked](/autorag/configuration/chunking/) into smaller pieces to improve retrieval granularity.
+4. **Embedding:** Each chunk is embedded using Workers AI’s embedding model to transform the content into vectors.
+5. **Vector storage:** The resulting vectors, along with metadata like file name, are stored in a the [Vectorize](/vectorize/) database created on your Cloudflare account.
+
+After the initial data set is indexed, AutoRAG will regularly check for updates in your data source (e.g. additions, updates, or deletes) and index changes to ensure your vector database is up to date.
+
+![Indexing](~/assets/images/autorag/indexing.png)
+
+## How querying works
+
+Once indexing is complete, AutoRAG is ready to respond to end-user queries in real time.
+
+Here is how the querying pipeline works:
+
+1. **Receive query from AutoRAG API:** The query workflow begins when you send a request to either the AutoRAG’s [AI Search](/autorag/usage/rest-api/#ai-search) or [Search](/autorag/usage/rest-api/#search) endpoints.
+2. **Query rewriting (optional):** AutoRAG provides the option to [rewrite the input query](/autorag/configuration/query-rewriting/) using one of Workers AI’s LLMs to improve retrieval quality by transforming the original query into a more effective search query.
+3. **Embedding the query:** The rewritten (or original) query is transformed into a vector via the same embedding model used to embed your data so that it can be compared against your vectorized data to find the most relevant matches.
+4. **Querying Vectorize index:** The query vector is [queried](/vectorize/best-practices/query-vectors/) against stored vectors in the associated Vectorize database for your AutoRAG.
+5. **Content retrieval:** Vectorize returns the metadata of the most relevant chunks, and the original content is retrieved from the R2 bucket. If you are using the Search endpoint, the content is returned at this point.
+6. **Response generation:** If you are using the AI Search endpoint, then a text-generation model from Workers AI is used to generate a response using the retrieved content and the original user’s query, combined via a [system prompt](/autorag/configuration/system-prompt/). The context-aware response from the model is returned.
+
+![Querying](~/assets/images/autorag/querying.png)

src/content/docs/autorag/concepts/index.mdx

@@ -0,0 +1,12 @@
+---
+pcx_content_type: navigation
+title: Concepts
+sidebar:
+  order: 3
+  group:
+    hideIndex: true
+---
+
+import { DirectoryListing } from "~/components";
+
+<DirectoryListing />

src/content/docs/autorag/concepts/what-is-rag.mdx

@@ -0,0 +1,41 @@
+---
+pcx_content_type: concept
+title: What is RAG
+sidebar:
+  order: 1
+---
+
+Retrieval-Augmented Generation (RAG) is a way to use your own data with a large language model (LLM). Instead of relying only on what the model was trained on, RAG searches for relevant information from your data source and uses it to help answer questions.
+
+## How RAG works
+
+Here’s a simplified overview of the RAG pipeline:
+
+1. **Indexing:** Your content (e.g. docs, wikis, product information) is split into smaller chunks and converted into vectors using an embedding model. These vectors are stored in a vector database.
+2. **Retrieval:** When a user asks a question, it’s also embedded into a vector and used to find the most relevant chunks from the vector database.
+3. **Generation:** The retrieved content and the user’s original question are combined into a single prompt. An LLM uses that prompt to generate a response.
+
+The resulting response should be accurate, relevant, and based on your own data.
+
+![What is RAG](~/assets/images/autorag/RAG.png)
+
+:::note[How does AutoRAG work]
+To learn more details about how AutoRAG uses RAG under the hood, reference [How AutoRAG works](/autorag/concepts/how-autorag-works/).
+:::
+
+## Why use RAG?
+
+RAG lets you bring your own data into LLM generation without retraining or fine-tuning a model. It improves both accuracy and trust by retrieving relevant content at query time and using that as the basis for a response.
+
+Benefits of using RAG:
+
+- **Accurate and current answers:** Responses are based on your latest content, not outdated training data.
+- **Control over information sources:** You define the knowledge base so answers come from content you trust.
+- **Fewer hallucinations:** Responses are grounded in real, retrieved data, reducing made-up or misleading answers.
+- **No model training required:** You can get high-quality results without building or fine-tuning your own LLM which can be time consuming and costly.
+
+RAG is ideal for building AI-powered apps like:
+
+- AI assistants for internal knowledge
+- Support chatbots connected to your latest content
+- Enterprise search across documentation and files

src/content/docs/autorag/configuration/cache.mdx

@@ -0,0 +1,50 @@
+---
+pcx_content_type: concept
+title: Similarity cache
+sidebar:
+  order: 6
+---
+
+Similarity-based caching in AutoRAG lets you serve responses from Cloudflare’s cache for queries that are similar to previous requests, rather than creating new, unique responses for every request. This speeds up response times and cuts costs by reusing answers for questions that are close in meaning.
+
+## How It Works
+
+Unlike with basic caching, which creates a new response with every request, this is what happens when a request is received using similarity-based caching:
+
+1. AutoRAG checks if a _similar_ prompt (based on your chosen threshold) has been answered before.
+2. If a match is found, it returns the cached response instantly.
+3. If no match is found, it generates a new response and caches it.
+
+To see if a response came from the cache, check the `cf-aig-cache-status` header: `HIT` for cached and `MISS` for new.
+
+## What to consider when using similarity cache
+
+Consider these behaviors when using similarity caching:
+
+- **Volatile Cache**: If two similar requests hit at the same time, the first might not cache in time for the second to use it, resulting in a `MISS`.
+- **30-Day Cache**: Cached responses last 30 days, then expire automatically. No custom durations for now.
+- **Data Dependency**: Cached responses are tied to specific document chunks. If those chunks change or get deleted, the cache clears to keep answers fresh.
+
+## How similarity matching works
+
+AutoRAG’s similarity cache uses **MinHash and Locality-Sensitive Hashing (LSH)** to find and reuse responses for prompts that are worded similarly.
+
+Here’s how it works when a new prompt comes in:
+
+1. The prompt is split into small overlapping chunks of words (called shingles), like “what’s the” or “the weather.”
+2. These shingles are turned into a “fingerprint” using MinHash. The more overlap two prompts have, the more similar their fingerprints will be.
+3. Fingerprints are placed into LSH buckets, which help AutoRAG quickly find similar prompts without comparing every single one.
+4. If a past prompt in the same bucket is similar enough (based on your configured threshold), AutoRAG reuses its cached response.
+
+## Choosing a threshold
+
+The similarity threshold decides how close two prompts need to be to reuse a cached response. Here are the available thresholds:
+
+| Threshold        | Description                 | Example Match                                                                   |
+| ---------------- | --------------------------- | ------------------------------------------------------------------------------- |
+| Exact            | Near-identical matches only | "What’s the weather like today?" matches with "What is the weather like today?" |
+| Strong (default) | High semantic similarity    | "What’s the weather like today?" matches with "How’s the weather today?"        |
+| Broad            | Moderate match, more hits   | "What’s the weather like today?" matches with "Tell me today’s weather"         |
+| Loose            | Low similarity, max reuse   | "What’s the weather like today?" matches with "Give me the forecast"            |
+
+Test these values to see which works best with your application.

src/content/docs/autorag/configuration/chunking.mdx

@@ -0,0 +1,50 @@
+---
+pcx_content_type: concept
+title: Chunking
+sidebar:
+  order: 6
+---
+
+Chunking is the process of splitting large data into smaller segments before embedding them for search. AutoRAG uses **recursive chunking**, which breaks your content at natural boundaries (like paragraphs or sentences), and then further splits it if the chunks are too large.
+
+## What is recurisve chunking
+
+Recursive chunking tries to keep chunks meaningful by:
+
+- **Splitting at natural boundaries:** like paragraphs, then sentences.
+- **Checking the size:** if a chunk is too long (based on token count), it’s split again into smaller parts.
+
+This way, chunks are easy to embed and retrieve, without cutting off thoughts mid-sentence.
+
+## Chunking controls
+
+AutoRAG exposes two parameters to help you control chunking behavior:
+
+- **Chunk size**: The number of tokens per chunk.
+  - Minimum: `64`
+  - Maximum: `512`
+- **Chunk overlap**: The percentage of overlapping tokens between adjacent chunks.
+  - Minimum: `0%`
+  - Maximum: `30%`
+
+These settings apply during the indexing step, before your data is embedded and stored in Vectorize.
+
+## Choosing chunk size and overlap
+
+Chunking affects both how your content is retrieved and how much context is passed into the generation model. Try out this external [chunk visualizer tool](https://huggingface.co/spaces/m-ric/chunk_visualizer) to help understand how different chunk settings could look.
+
+For chunk size, consider how:
+
+- **Smaller chunks** create more precise vector matches, but may split relevant ideas across multiple chunks.
+- **Larger chunks** retain more context, but may dilute relevance and reduce retrieval precision.
+
+For chunk overlap, consider how:
+
+- **More overlap** helps preserve continuity across boundaries, especially in flowing or narrative content.
+- **Less overlap** reduces indexing time and cost, but can miss context if key terms are split between chunks.
+
+### Additional considerations:
+
+- **Vector index size:** Smaller chunk sizes produce more chunks and more total vectors. Refer to the [Vectorize limits](/vectorize/platform/limits/) to ensure your configuration stays within the maximum allowed vectors per index.
+- **Generation model context window:** Generation models have a limited context window that must fit all retrieved chunks (`topK` × `chunk size`), the user query, and the model’s output. Be careful with large chunks or high topK values to avoid context overflows.
+- **Cost and performance:** Larger chunks and higher topK settings result in more tokens passed to the model, which can increase latency and cost. You can monitor this usage in [AI Gateway](/ai-gateway/).

src/content/docs/autorag/configuration/data-source.mdx

@@ -0,0 +1,27 @@
+---
+title: Data source
+pcx_content_type: how-to
+sidebar:
+  order: 2
+---
+
+import { Render } from "~/components";
+
+AutoRAG currently supports Cloudflare R2 as the data source for storing your knowledge base. To get started, [configure an R2 bucket](/r2/get-started/) containing your data.
+
+AutoRAG will automatically scan and process supported files stored in that bucket. Files that are unsupported or exceed the size limit will be skipped during indexing and logged as errors.
+
+## File limits
+
+AutoRAG has different file size limits depending on the file type:
+
+- Up to **4 MB** for files that are already in plain text or Markdown.
+- Up to **1 MB** for files that need to be converted into Markdown (like PDFs or other rich formats).
+
+Files that exceed these limits will not be indexed and will show up in the error logs.
+
+## File types
+
+AutoRAG is powered by and accepts the same file types as [Markdown Conversion](/workers-ai/markdown-conversion/). The following table lists the supported formats:
+
+<Render file="markdown-conversion-support" product="workers-ai" />

src/content/docs/autorag/configuration/index.mdx

@@ -0,0 +1,35 @@
+---
+pcx_content_type: navigation
+title: Configuration
+sidebar:
+  order: 5
+---
+
+import { MetaInfo, Type } from "~/components";
+
+When creating an AutoRAG instance, you can customize how your RAG pipeline ingests, processes, and responds to data using a set of configuration options. Some settings can be updated after the instance is created, while others are fixed at creation time.
+
+The table below lists all available configuration options:
+
+| Configuration                                                                | Editable after creation | Description                                                                                |
+| ---------------------------------------------------------------------------- | ----------------------- | ------------------------------------------------------------------------------------------ |
+| [Data source](/autorag/configuration/data-source/)                           | no                      | The source where your knowledge base is stored (for example, R2 bucket)                            |
+| [Chunk size](/autorag/configuration/chunking/)                               | yes                     | Number of tokens per chunk                                                                 |
+| [Chunk overlap](/autorag/configuration/chunking/)                            | yes                     | Number of overlapping tokens between chunks                                                |
+| [Embedding model](/autorag/configuration/models/)                            | no                      | Model used to generate vector embeddings                                                   |
+| [Query rewrite](/autorag/configuration/query-rewriting/)                     | yes                     | Enable or disable query rewriting before retrieval                                         |
+| [Query rewrite model](/autorag/configuration/models/)                        | yes                     | Model used for query rewriting                                                             |
+| [Query rewrite system prompt](/autorag/configuration/system-prompt/)         | yes                     | Custom system prompt to guide query rewriting behavior                                     |
+| [Match threshold](/autorag/configuration/retrieval-configuration/)           | yes                     | Minimum similarity score required for a vector match                                       |
+| [Maximum number of results](/autorag/configuration/retrieval-configuration/) | yes                     | Maximum number of vector matches returned (`top_k`)                                        |
+| [Generation model](/autorag/configuration/models/)                           | yes                     | Model used to generate the final response                                                  |
+| [Generation system prompt](/autorag/configuration/system-prompt/)            | yes                     | Custom system prompt to guide response generation                                          |
+| [Similarity caching](/autorag/configuration/cache/)                          | yes                     | Enable or disable caching of responses for similar (not just exact) prompts                |
+| [Similarity caching threshold](/autorag/configuration/cache/)                | yes                     | Controls how similar a new prompt must be to a previous one to reuse its cached response   |
+| [AI Gateway](/ai-gateway)                                                    | yes                     | AI Gateway for monitoring and controlling model usage                                      |
+| AutoRAG name                                                                 | no                      | Name of your AutoRAG instance                                                              |
+| Service API token                                                            | yes                     | API token granted to AutoRAG to give it permission to configure resources on your account. |
+
+:::note[API token]
+The Service API token is different from the AutoRAG API token that you can make to interact with your AutoRAG. The Service API token is only used by AutoRAG to get permissions to configure resources on your account.
+:::