progress

Author: aninibread

src/content/docs/autorag/concepts/how-autorag-works.mdx

@@ -2,11 +2,7 @@
 pcx_content_type: navigation
 title: Configuration
 sidebar:
-  order: 4
-  group:
-    hideIndex: true
+  order: 5
 ---
 
-import { DirectoryListing } from "~/components";
-
-<DirectoryListing />
+something about all the configurations

src/content/docs/autorag/configuration/index.mdx

@@ -0,0 +1,17 @@
+---
+pcx_content_type: concept
+title: Indexing
+sidebar:
+  order: 4
+---
+
+import {
+	Badge,
+	Description,
+	Render,
+	TabItem,
+	Tabs,
+	WranglerConfig,
+	MetaInfo,
+	Type,
+} from "~/components";

src/content/docs/autorag/configuration/indexing.mdx

@@ -0,0 +1,17 @@
+---
+pcx_content_type: concept
+title: System prompt
+sidebar:
+  order: 4
+---
+
+import {
+	Badge,
+	Description,
+	Render,
+	TabItem,
+	Tabs,
+	WranglerConfig,
+	MetaInfo,
+	Type,
+} from "~/components";

src/content/docs/autorag/configuration/system-prompt.mdx

@@ -11,9 +11,9 @@ head:
 
 AutoRAG allows developers to create fully managed retrieval-augmented generation (RAG) pipelines to power AI applications with accurate and up-to-date information without needing to manage infrastructure.
 
-## 1. Upload data, or use existing data in R2
+## 1. Upload data or use existing data in R2
 
-AutoRAG integrates with R2 for data import. Create an R2 bucket if you don’t have one and upload your data. 
+AutoRAG integrates with R2 for data import. Create an R2 bucket if you don’t have one and upload your data.
 
 :::note
 Before you create your first bucket, you must purchase R2 from the Cloudflare dashboard.
@@ -29,24 +29,18 @@ To create and upload objects to your bucket from the Cloudflare Dashboard:
 
 To create a new AutoRAG:
 
-1. Log in to the [Cloudflare Dashboard](https://dash.cloudflare.com) and select **AI** then **AutoRAG**.
-2. Select **Create AutoRAG** and complete the setup process:
-    - Select a **R2 bucket**.
-    - Select an **embedding model**.
-    - Select a **LLM model**.
-    - Select or create an **AI Gateway**.
-    - Enter a **name** for the AutoRAG.
-    - Create a new **API token** or reuse an existing API token.
-3. Select **Create**
+1. Log in to the [Cloudflare Dashboard](https://dash.cloudflare.com) and select **AI** > **AutoRAG**.
+2. Select **Create AutoRAG**, configure the AutoRAG, and complete the setup process.
+3. Select **Create**.
 
-## 3. Monitor indexing 
+## 3. Monitor indexing
 
-Once created, AutoRAG will create a Vectorize database in your account and begin indexing the data. 
+Once created, AutoRAG will create a Vectorize index in your account and begin indexing the data.
 
 To monitor the indexing progress:
 
 1. From the **AutoRAG** page in the dashboard, locate and select your AutoRAG.
-2. Navigate to the **Overview** page to view the current indexing progress.
+2. Navigate to the **Overview** page to view the current indexing status.
 
 ## 4. Try it out
 
@@ -57,10 +51,9 @@ Once indexing is complete, you can run your first query:
 3. Select **Search with AI** or **Search**.
 4. Enter a **query** to test out its response.
 
-
 ## Use AutoRAG options
 
 Cloudflare provides multiple ways for developers to use AutoRAG in their applications:
 
-- REST API
-- Workers Binding
+- [Workers Binding](/autorag/use-autorag/workers-binding/)
+- [REST API](/autorag/use-autorag/rest-api/)

src/content/docs/autorag/configuration/workers-binding.mdx

@@ -0,0 +1,46 @@
+---
+pcx_content_type: concept
+title: How AutoRAG works
+sidebar:
+  order: 3
+---
+
+AutoRAG simplifies the process of building and managing a Retrieval-Augmented Generation (RAG) pipeline using Cloudflare's serverless platform. Instead of manually stitching together components like Workers AI, Vectorize, and writing custom logic for indexing, retireval, and generation, AutoRAG handles it all for you. It also continuously indexes your data to ensure responses stay accurate and up-to-date.
+
+AutoRAG consists of two core processes:
+
+- **Indexing:** An asynchronous background process that monitors your data source for changes and transforms your data into vector representation for search.
+- **Querying:** A synchronous process triggered by user queries. It retrieves the most relevant content and generates context-aware responses using a large language model (LLM).
+
+## Indexing
+
+Indexing begins automatically when you create an AutoRAG instance and connect a data source. It runs asynchronously in the background and checks for updates periodically, so new or updated data are automatically indexed in the vector index.
+
+Here is what happens during indexing:
+
+1. **Data ingestion:** AutoRAG reads from your connected data source. Files that are unsupported or exceed size limits are flagged and reported as indexing errors.
+2. **Markdown conversion:** AutoRAG uses a Worker powered by [Workers AI’s Markdown Conversion](/workers-ai/markdown-conversion/) to convert all data into structured Markdown. This ensures consistency across diverse file types. For images, Workers AI is used to perform object detection followed by vision-to-language transformation to convert images into Markdown text.
+3. **Chunking:** The extracted text is chunked to improve retrieval granularity.
+4. **Embedding:** Each chunk is embedded using Workers AI’s embedding model to transform the content into vectors.
+5. **Vector storage:** The resulting vectors, along with metadata like source location and file name, are stored in a Vectorize index created on your account.
+
+[INSERT IMAGE]
+
+## Querying
+
+Once indexing is complete, AutoRAG is ready to respond to end-user queries in real time.
+
+Here’s how the querying pipeline works:
+
+1. **Receive query from AutoRAG API:** The query workflow begins when you send a request to the AutoRAG API.
+2. **Query rewriting (optional):** The input query can be rewritten by one of Workers AI’s LLMs to improve semantic retrieval, if enabled.
+3. **Embedding the query:** The rewritten (or original) query is turned into a vector via the same embedding model in Workers AI.
+4. **Querying Vectorize index:** The query vector is [searched](/vectorize/best-practices/query-vectors/) against the Vectorize index associated with your AutoRAG instance.
+5. **Content retrieval:** Vectorize returns the most relevant chunks and their metadata. And the original content is retrieved from the R2 bucket. These are passed to a text-generation model.
+6. **Response generation:** A text-generation model from Workers AI is used to a response using the retrieved content and the original user’s query using the generation model you select.
+
+[INSERT IMAGE]
+
+## Get Started
+
+Learn how to [get started](/autorag/get-started/) with AutoRAG.

src/content/docs/autorag/get-started.mdx

@@ -15,14 +15,88 @@ import {
 	LinkTitleCard,
 	Plan,
 	RelatedProduct,
+	LinkButton,
 } from "~/components";
 
 <Plan type="all" />
+
+AutoRAG allows developers to create fully-managed retrieval-augmented generation (RAG) pipelines powered by Cloudflare allowing developers to integrate context-aware AI into their applications without managing infrastructure.
+
+<div>
+	<LinkButton href="/autorag/get-started">Get started</LinkButton>
+	<LinkButton target="_blank" variant="secondary" icon="external" href="">
+		Watch an AutoRAG demo
+	</LinkButton>
+</div>
+
 ---
 
 ## Features
+
 ---
 
 ## Related products
+
+<RelatedProduct header="Workers AI" href="/workers-ai/" product="workers-ai">
+
+Run machine learning models, powered by serverless GPUs, on Cloudflare’s global network.
+
+</RelatedProduct>
+
+<RelatedProduct header="AI Gateway" href="/ai-gateway/" product="ai-gateway">
+
+Observe and control your AI applications with caching, rate limiting, request retries, model fallback, and more.
+
+</RelatedProduct>
+
+<RelatedProduct header="Vectorize" href="/vectorize/" product="vectorize">
+
+Build full-stack AI applications with Vectorize, Cloudflare’s vector database.
+
+</RelatedProduct>
+
+<RelatedProduct header="Workers" href="/workers/" product="workers">
+
+Build serverless applications and deploy instantly across the globe for exceptional performance, reliability, and scale.
+
+</RelatedProduct>
+
+<RelatedProduct header="R2" href="/r2/" product="r2">
+
+Store large amounts of unstructured data without the costly egress bandwidth fees associated with typical cloud storage services.
+
+</RelatedProduct>
+
 ---
-## More resources
+
+## More resources
+
+<CardGrid>
+
+<LinkTitleCard
+	title="Get started"
+	href="/workers-ai/get-started/workers-wrangler/"
+	icon="open-book"
+>
+	Build and deploy your first Workers AI application.
+</LinkTitleCard>
+
+<LinkTitleCard
+	title="Developer Discord"
+	href="https://discord.cloudflare.com"
+	icon="discord"
+>
+	Connect with the Workers community on Discord to ask questions, share what you
+	are building, and discuss the platform with other developers.
+</LinkTitleCard>
+
+<LinkTitleCard
+	title="@CloudflareDev"
+	href="https://x.com/cloudflaredev"
+	icon="x.com"
+>
+	Follow @CloudflareDev on Twitter to learn about product announcements, and
+	what is new in Cloudflare Workers.
+</LinkTitleCard>
+
+</CardGrid>

src/content/docs/autorag/how-autorag-works.mdx

@@ -1,12 +1,12 @@
 ---
 pcx_content_type: navigation
-title: Key concepts
+title: Use AutoRAG
 sidebar:
-  order: 2
+  order: 4
   group:
     hideIndex: true
 ---
 
 import { DirectoryListing } from "~/components";
 
-<DirectoryListing />
+<DirectoryListing />