From c61cc5c459eac1daf57b01b8fd1b96f288f93552 Mon Sep 17 00:00:00 2001 From: Anni Wang Date: Thu, 21 Aug 2025 20:35:59 -0400 Subject: [PATCH 01/11] nlweb docs --- src/content/docs/autorag/how-to/nlweb.mdx | 61 +++++++++++++++++++++++ 1 file changed, 61 insertions(+) create mode 100644 src/content/docs/autorag/how-to/nlweb.mdx diff --git a/src/content/docs/autorag/how-to/nlweb.mdx b/src/content/docs/autorag/how-to/nlweb.mdx new file mode 100644 index 000000000000000..fadc042fef3398f --- /dev/null +++ b/src/content/docs/autorag/how-to/nlweb.mdx @@ -0,0 +1,61 @@ +--- +pcx_content_type: concept +title: NLWeb +sidebar: + order: 1 +--- + +Enable conversational search on your website with NLWeb and Cloudflare AutoRAG. This template crawls your site, indexes the content, and deploys NLWeb-standard endpoints to serve both people and AI agents. + +:::note +This is a public preview ideal for experimentation. If you're interested in running this in production workflows please contact us at nlweb@cloudflare.com +::: + +## What is NLWeb + +[NLWeb]() is an open project developed by Microsoft that defines a standard protocol for natural language queries on websites. Its goal is to make every website as accessible and interactive as a conversational AI app, so both people and AI agents can reliably query site content. It does this by exposing two key endpoints: + +- `/ask`: Conversational endpoint for user queries +- `/mcp`: Structured Model Context Protocol (MCP) endpoint for AI agents + +## How to Use It + +You can deploy NLWeb on your website directly through the AutoRAG dashboard: + +1. Navigate to Compute & AI > AutoRAG in the Cloudflare Dashboard +2. Select Create, then choose the NLWeb Website option +3. Select your domain from your Cloudflare account +4. Click Deploy + +Once complete, AutoRAG will crawl and index your site, then deploy an NLWeb Worker for you. + +## What This Template Includes + +Choosing the NLWeb Website option extends a normal AutoRAG by tailoring it for content‑heavy websites and giving you everything needed to adopt NLWeb as the standard for conversational search on your site. Specifically, the template provides: + +- **Website as a data source:** Uses [Website](/autorag/configuration/data-source/website/) as data source option to crawl and ingest pages with the Rendered Sites option. +- **Defaults for content-heavy websites:** Applies tuned embedding and retrieval configurations ideal for publishing and content‑rich websites. +- **NLWeb Worker deployment:** Automatically spins up a Cloudflare Worker from the [NLWeb Worker template](https://github.com/cloudflare/templates). + +## What the Worker Includes + +Your deployed Worker provides two endpoints: + +- `/ask` — NLWeb’s standard conversational endpoint + - Powers the conversational UI at the root (`/`) + - Powers the embeddable preview widget (`/snippet.html`) +- `/mcp` — NLWeb’s MCP server endpoint for trusted AI agents + +These endpoints give both people and agents structured access to your content. + +## Using It on Your Website + +To integrate NLWeb search directly into your site: + +1. Find your deployed Worker in the Cloudflare Dashboard. You can do so by going to your AutoRAG, selecting Connect, then the NLWeb tab, and clicking Go to Worker +2. Add a [custom domain](/workers/configuration/routing/custom-domains/) to your Worker (for example, ask.example.com) +3. Use the /ask endpoint on your custom domain to power the search (for example, https://ask.example.com/ask) + +This lets you serve conversational AI search directly from your own domain, with full control over how people and agents access your content. + + From d9cfebee3b0638f1435046340457820017dfadc1 Mon Sep 17 00:00:00 2001 From: Anni Wang Date: Thu, 21 Aug 2025 20:38:32 -0400 Subject: [PATCH 02/11] add link --- src/content/docs/autorag/how-to/nlweb.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/content/docs/autorag/how-to/nlweb.mdx b/src/content/docs/autorag/how-to/nlweb.mdx index fadc042fef3398f..9a7f40ddeb706bb 100644 --- a/src/content/docs/autorag/how-to/nlweb.mdx +++ b/src/content/docs/autorag/how-to/nlweb.mdx @@ -13,7 +13,7 @@ This is a public preview ideal for experimentation. If you're interested in runn ## What is NLWeb -[NLWeb]() is an open project developed by Microsoft that defines a standard protocol for natural language queries on websites. Its goal is to make every website as accessible and interactive as a conversational AI app, so both people and AI agents can reliably query site content. It does this by exposing two key endpoints: +[NLWeb](https://github.com/nlweb-ai/NLWeb) is an open project developed by Microsoft that defines a standard protocol for natural language queries on websites. Its goal is to make every website as accessible and interactive as a conversational AI app, so both people and AI agents can reliably query site content. It does this by exposing two key endpoints: - `/ask`: Conversational endpoint for user queries - `/mcp`: Structured Model Context Protocol (MCP) endpoint for AI agents From dac2d0fd35b19865caab7cc41b2e60a1e27534ca Mon Sep 17 00:00:00 2001 From: Anni Wang Date: Thu, 21 Aug 2025 20:44:55 -0400 Subject: [PATCH 03/11] fix link --- src/content/docs/autorag/how-to/nlweb.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/content/docs/autorag/how-to/nlweb.mdx b/src/content/docs/autorag/how-to/nlweb.mdx index 9a7f40ddeb706bb..6d9c03a3e94beea 100644 --- a/src/content/docs/autorag/how-to/nlweb.mdx +++ b/src/content/docs/autorag/how-to/nlweb.mdx @@ -54,7 +54,7 @@ To integrate NLWeb search directly into your site: 1. Find your deployed Worker in the Cloudflare Dashboard. You can do so by going to your AutoRAG, selecting Connect, then the NLWeb tab, and clicking Go to Worker 2. Add a [custom domain](/workers/configuration/routing/custom-domains/) to your Worker (for example, ask.example.com) -3. Use the /ask endpoint on your custom domain to power the search (for example, https://ask.example.com/ask) +3. Use the /ask endpoint on your custom domain to power the search (for example, ask.example.com/ask) This lets you serve conversational AI search directly from your own domain, with full control over how people and agents access your content. From abf5e2f48b2b300f6ba592b874233b1e80c6418c Mon Sep 17 00:00:00 2001 From: Anni Wang Date: Thu, 21 Aug 2025 22:41:13 -0400 Subject: [PATCH 04/11] fix --- src/content/docs/autorag/how-to/nlweb.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/content/docs/autorag/how-to/nlweb.mdx b/src/content/docs/autorag/how-to/nlweb.mdx index 6d9c03a3e94beea..00864273a77e434 100644 --- a/src/content/docs/autorag/how-to/nlweb.mdx +++ b/src/content/docs/autorag/how-to/nlweb.mdx @@ -56,6 +56,6 @@ To integrate NLWeb search directly into your site: 2. Add a [custom domain](/workers/configuration/routing/custom-domains/) to your Worker (for example, ask.example.com) 3. Use the /ask endpoint on your custom domain to power the search (for example, ask.example.com/ask) -This lets you serve conversational AI search directly from your own domain, with full control over how people and agents access your content. +This lets you serve conversational AI search directly from your own domain, with control over how people and agents access your content. From 6494066e90fc4a9cc2cf5dfc09900c32c3a807d4 Mon Sep 17 00:00:00 2001 From: Anni Wang <54481763+aninibread@users.noreply.github.com> Date: Fri, 22 Aug 2025 09:22:57 -0400 Subject: [PATCH 05/11] Update src/content/docs/autorag/how-to/nlweb.mdx Co-authored-by: Jun Lee --- src/content/docs/autorag/how-to/nlweb.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/content/docs/autorag/how-to/nlweb.mdx b/src/content/docs/autorag/how-to/nlweb.mdx index 00864273a77e434..9072f52ef6813b3 100644 --- a/src/content/docs/autorag/how-to/nlweb.mdx +++ b/src/content/docs/autorag/how-to/nlweb.mdx @@ -8,7 +8,7 @@ sidebar: Enable conversational search on your website with NLWeb and Cloudflare AutoRAG. This template crawls your site, indexes the content, and deploys NLWeb-standard endpoints to serve both people and AI agents. :::note -This is a public preview ideal for experimentation. If you're interested in running this in production workflows please contact us at nlweb@cloudflare.com +This is a public preview ideal for experimentation. If you're interested in running this in production workflows, please contact us at nlweb@cloudflare.com. ::: ## What is NLWeb From 8c4f2546ab59458ec0a7a0695d0c04b55a3e784f Mon Sep 17 00:00:00 2001 From: Anni Wang <54481763+aninibread@users.noreply.github.com> Date: Fri, 22 Aug 2025 09:23:09 -0400 Subject: [PATCH 06/11] Update src/content/docs/autorag/how-to/nlweb.mdx Co-authored-by: Jun Lee --- src/content/docs/autorag/how-to/nlweb.mdx | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/src/content/docs/autorag/how-to/nlweb.mdx b/src/content/docs/autorag/how-to/nlweb.mdx index 9072f52ef6813b3..a7056ef6c57547d 100644 --- a/src/content/docs/autorag/how-to/nlweb.mdx +++ b/src/content/docs/autorag/how-to/nlweb.mdx @@ -22,10 +22,11 @@ This is a public preview ideal for experimentation. If you're interested in runn You can deploy NLWeb on your website directly through the AutoRAG dashboard: -1. Navigate to Compute & AI > AutoRAG in the Cloudflare Dashboard -2. Select Create, then choose the NLWeb Website option -3. Select your domain from your Cloudflare account -4. Click Deploy +1. Log in to your [Cloudflare dashboard](https://dash.cloudflare.com/). +2. Go to **Compute & AI** > **AutoRAG**. +3. Select **Create AutoRAG**, then choose the **NLWeb Website** option. +4. Select your domain from your Cloudflare account. +5. Click **Deploy**. Once complete, AutoRAG will crawl and index your site, then deploy an NLWeb Worker for you. From 9d92dbb7ee8bd03bee8df41926788bd551740d9d Mon Sep 17 00:00:00 2001 From: Anni Wang <54481763+aninibread@users.noreply.github.com> Date: Fri, 22 Aug 2025 09:23:16 -0400 Subject: [PATCH 07/11] Update src/content/docs/autorag/how-to/nlweb.mdx Co-authored-by: Jun Lee --- src/content/docs/autorag/how-to/nlweb.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/content/docs/autorag/how-to/nlweb.mdx b/src/content/docs/autorag/how-to/nlweb.mdx index a7056ef6c57547d..b70d357fb66ecae 100644 --- a/src/content/docs/autorag/how-to/nlweb.mdx +++ b/src/content/docs/autorag/how-to/nlweb.mdx @@ -18,7 +18,7 @@ This is a public preview ideal for experimentation. If you're interested in runn - `/ask`: Conversational endpoint for user queries - `/mcp`: Structured Model Context Protocol (MCP) endpoint for AI agents -## How to Use It +## How to use it You can deploy NLWeb on your website directly through the AutoRAG dashboard: From 7cc8dc0cbecfca4322ad366a1ccedef6f5a0a151 Mon Sep 17 00:00:00 2001 From: Anni Wang <54481763+aninibread@users.noreply.github.com> Date: Fri, 22 Aug 2025 09:23:40 -0400 Subject: [PATCH 08/11] Apply suggestions from code review Co-authored-by: Jun Lee --- src/content/docs/autorag/how-to/nlweb.mdx | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/src/content/docs/autorag/how-to/nlweb.mdx b/src/content/docs/autorag/how-to/nlweb.mdx index b70d357fb66ecae..e0774fe4a26c2e4 100644 --- a/src/content/docs/autorag/how-to/nlweb.mdx +++ b/src/content/docs/autorag/how-to/nlweb.mdx @@ -30,15 +30,15 @@ You can deploy NLWeb on your website directly through the AutoRAG dashboard: Once complete, AutoRAG will crawl and index your site, then deploy an NLWeb Worker for you. -## What This Template Includes +## What this template includes -Choosing the NLWeb Website option extends a normal AutoRAG by tailoring it for content‑heavy websites and giving you everything needed to adopt NLWeb as the standard for conversational search on your site. Specifically, the template provides: +Choosing the NLWeb Website option extends a normal AutoRAG by tailoring it for content‑heavy websites and giving you everything that is required to adopt NLWeb as the standard for conversational search on your site. Specifically, the template provides: - **Website as a data source:** Uses [Website](/autorag/configuration/data-source/website/) as data source option to crawl and ingest pages with the Rendered Sites option. - **Defaults for content-heavy websites:** Applies tuned embedding and retrieval configurations ideal for publishing and content‑rich websites. - **NLWeb Worker deployment:** Automatically spins up a Cloudflare Worker from the [NLWeb Worker template](https://github.com/cloudflare/templates). -## What the Worker Includes +## What the Worker includes Your deployed Worker provides two endpoints: @@ -53,9 +53,12 @@ These endpoints give both people and agents structured access to your content. To integrate NLWeb search directly into your site: -1. Find your deployed Worker in the Cloudflare Dashboard. You can do so by going to your AutoRAG, selecting Connect, then the NLWeb tab, and clicking Go to Worker +1. Find your deployed Worker in the [Cloudflare dashboard](https://dash.cloudflare.com/): + - Go to **Compute & AI** > **AutoRAG**. + - Select **Connect**, then go to the **NLWeb** tab. + - Select **Go to Worker**. 2. Add a [custom domain](/workers/configuration/routing/custom-domains/) to your Worker (for example, ask.example.com) -3. Use the /ask endpoint on your custom domain to power the search (for example, ask.example.com/ask) +3. Use the `/ask` endpoint on your custom domain to power the search (for example, ask.example.com/ask) This lets you serve conversational AI search directly from your own domain, with control over how people and agents access your content. From 59dd4cdf758942ac833d612d15231af6c8a7e017 Mon Sep 17 00:00:00 2001 From: Anni Wang Date: Fri, 22 Aug 2025 09:43:15 -0400 Subject: [PATCH 09/11] future change flow --- src/content/docs/autorag/how-to/nlweb.mdx | 20 +++++++++++++++++++- 1 file changed, 19 insertions(+), 1 deletion(-) diff --git a/src/content/docs/autorag/how-to/nlweb.mdx b/src/content/docs/autorag/how-to/nlweb.mdx index e0774fe4a26c2e4..6cbe92e2a73d4be 100644 --- a/src/content/docs/autorag/how-to/nlweb.mdx +++ b/src/content/docs/autorag/how-to/nlweb.mdx @@ -26,7 +26,7 @@ You can deploy NLWeb on your website directly through the AutoRAG dashboard: 2. Go to **Compute & AI** > **AutoRAG**. 3. Select **Create AutoRAG**, then choose the **NLWeb Website** option. 4. Select your domain from your Cloudflare account. -5. Click **Deploy**. +5. Click **Start indexing**. Once complete, AutoRAG will crawl and index your site, then deploy an NLWeb Worker for you. @@ -63,3 +63,21 @@ To integrate NLWeb search directly into your site: This lets you serve conversational AI search directly from your own domain, with control over how people and agents access your content. +## Modifying or updating the Worker + +You may want to customize your Worker, for example, to adjust the UI for the embeddable snippet. In those cases, we recommend calling the `/ask` endpoint for queries and building your own UI on top of it, however, you may also choose to modify the Worker's code for the embeddable UI. + +If the NLWeb standard is updated, you can update your Worker to stay compatible and recieve the latest updates. + +The simplest way to apply changes or updates is to redeploy the Worker template: + +[![Deploy to Cloudflare](https://deploy.workers.cloudflare.com/button)](https://deploy.workers.cloudflare.com/?url=https://github.com/cloudflare/templates/tree/main/nlweb-template) + +To do so: + +1. Select the **Deploy to Cloudflare** button from above to deploy the Worker template to your Cloudflare account. +2. Enter the name of your AutoRAG in the `RAG_ID` environment variable field. +3. Click **Deploy**. +4. Select the **GitHub/GitLab** icon on the Workers Dashboard. +4. Clone the repository that is created for your Worker. +5. Make your modifications, then commit and push changes to the repository to update your Worker. \ No newline at end of file From 3aeb0af3a066a2a63495e827f367108ed2c2ea70 Mon Sep 17 00:00:00 2001 From: Anni Wang Date: Fri, 22 Aug 2025 09:45:42 -0400 Subject: [PATCH 10/11] update --- src/content/docs/autorag/how-to/nlweb.mdx | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/src/content/docs/autorag/how-to/nlweb.mdx b/src/content/docs/autorag/how-to/nlweb.mdx index 6cbe92e2a73d4be..63eb66837c332c7 100644 --- a/src/content/docs/autorag/how-to/nlweb.mdx +++ b/src/content/docs/autorag/how-to/nlweb.mdx @@ -80,4 +80,6 @@ To do so: 3. Click **Deploy**. 4. Select the **GitHub/GitLab** icon on the Workers Dashboard. 4. Clone the repository that is created for your Worker. -5. Make your modifications, then commit and push changes to the repository to update your Worker. \ No newline at end of file +5. Make your modifications, then commit and push changes to the repository to update your Worker. + +Now you can use this Worker as the new NLWeb endpoint for your website. \ No newline at end of file From fe962ee47882736ff07cbf100127c424eea73baf Mon Sep 17 00:00:00 2001 From: Anni Wang Date: Fri, 22 Aug 2025 12:05:35 -0400 Subject: [PATCH 11/11] embeddable snippet --- src/content/docs/autorag/how-to/nlweb.mdx | 25 ++++++++++++++++++++++- 1 file changed, 24 insertions(+), 1 deletion(-) diff --git a/src/content/docs/autorag/how-to/nlweb.mdx b/src/content/docs/autorag/how-to/nlweb.mdx index 63eb66837c332c7..2979eec040dc2fa 100644 --- a/src/content/docs/autorag/how-to/nlweb.mdx +++ b/src/content/docs/autorag/how-to/nlweb.mdx @@ -51,7 +51,7 @@ These endpoints give both people and agents structured access to your content. ## Using It on Your Website -To integrate NLWeb search directly into your site: +To integrate NLWeb search directly into your site you can: 1. Find your deployed Worker in the [Cloudflare dashboard](https://dash.cloudflare.com/): - Go to **Compute & AI** > **AutoRAG**. @@ -60,6 +60,29 @@ To integrate NLWeb search directly into your site: 2. Add a [custom domain](/workers/configuration/routing/custom-domains/) to your Worker (for example, ask.example.com) 3. Use the `/ask` endpoint on your custom domain to power the search (for example, ask.example.com/ask) +You can also use the embeddable snippet to add a search UI directly into your website. For example: + +```html + + + + + +
+ + + +``` + This lets you serve conversational AI search directly from your own domain, with control over how people and agents access your content.