Remove pipelines from llm-app and update website (#9418)

Co-authored-by: Olivier Ruas <[email protected]>
GitOrigin-RevId: 1a2b161e1de92d649b8fd938e8cbd54186a74d1c

Author: 2 people

docs/2.developers/4.user-guide/20.connect/99.connectors/100.slack_send_alerts.md

@@ -60,4 +60,4 @@ class: 'mx-auto'
 
 Note, that the values of the column `messages` in the above example do not have spaces. It is a restriction of `pw.debug.table_from_markdown` which uses spaces to separate columns. Any regular string works with the other connectors.
 
-If you want to see more examples with `pw.io.slack.send_alerts` you can check the [`drive_alert`](https://github.com/pathwaycom/llm-app/tree/main/examples/pipelines/drive_alert) example in the llm-app.
+If you want to see more examples with `pw.io.slack.send_alerts` you can check the [`drive_alert`](https://github.com/pathwaycom/llm-app/tree/main/templates/drive_alert) example in the llm-app.

docs/2.developers/4.user-guide/30.data-transformation/.indexes-in-pathway/article.py

@@ -430,7 +430,7 @@ class PointSchema(pw.Schema):
 #     run(data_dir=args.data_dir, host=args.host, port=args.port)
 # ```
 # %% [markdown]
-# A similar approach was taken in our [alerting example](https://github.com/pathwaycom/llm-app/tree/main/examples/pipelines/drive_alert).
+# A similar approach was taken in our [alerting example](https://github.com/pathwaycom/llm-app/tree/main/templates/drive_alert).
 # It is an LLM app that can send you alerts on slack when the response to your query has changed significantly.
 # %% [markdown]
 # ## Summary

docs/2.developers/4.user-guide/50.llm-xpack/.llm-examples.md

@@ -16,7 +16,7 @@ If you want to see how it works, this page gathers practical examples using Path
     <tbody>
         <tr>
             <td class="text-center">
-                <a href="https://github.com/pathwaycom/llm-app/tree/main/examples/pipelines/demo-question-answering">Pathway RAG app with always up-to-date knowledge</a>
+                <a href="https://github.com/pathwaycom/llm-app/tree/main/templates/question_answering_rag">Pathway RAG app with always up-to-date knowledge</a>
             </td>
             <td class="text-center">
                 This example shows how to create a RAG application using Pathway that provides always up-to-date knowledge to your LLM without the need for a separate ETL.
@@ -25,23 +25,23 @@ If you want to see how it works, this page gathers practical examples using Path
         </tr>
         <tr>
             <td class="text-center">
-                <a href="https://github.com/pathwaycom/llm-app/tree/main/examples/pipelines/adaptive-rag">Adaptive RAG</a>
+                <a href="https://github.com/pathwaycom/llm-app/tree/main/templates/adaptive_rag">Adaptive RAG</a>
             </td>
             <td class="text-center">
                 Example of the Adaptive RAG, a technique to dynamically adapt the number of documents in a RAG prompt using feedback from the LLM.
             </td>
         </tr>
         <tr>
             <td class="text-center">
-                <a href="https://github.com/pathwaycom/llm-app/tree/main/examples/pipelines/private-rag">Fully Private RAG with Pathway</a>
+                <a href="https://github.com/pathwaycom/llm-app/tree/main/templates/private_rag">Fully Private RAG with Pathway</a>
             </td>
             <td class="text-center">
                 This example shows how to set up a private RAG pipeline with adaptive retrieval using Pathway, Mistral, and Ollama.
             </td>
         </tr>
         <tr>
             <td class="text-center">
-                <a href="https://github.com/pathwaycom/llm-app/tree/main/examples/pipelines/gpt_4o_multimodal_rag">Multimodal RAG with Pathway</a>
+                <a href="https://github.com/pathwaycom/llm-app/tree/main/templates/multimodal_rag">Multimodal RAG with Pathway</a>
             </td>
             <td class="text-center">
                 This example demonstrates how you can launch a Multimodal RAG with Pathway. It relies on a document processing pipeline that utilizes GPT-4o in the parsing stage. Pathway extracts information from unstructured financial documents in your folders, updating results as documents change or new ones arrive. You can make your AI application run in permanent connection with your drive, in sync with your documents which include visually formatted elements: tables, charts, etc.
@@ -63,15 +63,15 @@ If you want to see how it works, this page gathers practical examples using Path
     <tbody>
         <tr>
             <td class="text-center">
-                <a href="https://github.com/pathwaycom/llm-app/tree/main/examples/pipelines/demo-document-indexing">Realtime Document Indexing with Pathway</a>
+                <a href="https://github.com/pathwaycom/llm-app/tree/main/templates/document_indexing">Realtime Document Indexing with Pathway</a>
             </td>
             <td class="text-center">
                 Basic example of a real-time document indexing pipeline powered by Pathway. You can index documents from different data sources, such as SharePoint or Google Drive. You can then query the index to retrieve documents, get statistics about the index, and retrieve file metadata.
             </td>
         </tr>
         <tr>
             <td class="text-center">
-                <a href="https://github.com/pathwaycom/llm-app/tree/main/examples/pipelines/drive_alert">Drive Alert Pipeline</a>
+                <a href="https://github.com/pathwaycom/llm-app/tree/main/templates/drive_alert">Drive Alert Pipeline</a>
             </td>
             <td class="text-center">
                 This example is very similar to the "Alert" example, the only difference is the data source (Google Drive).
@@ -80,7 +80,7 @@ If you want to see how it works, this page gathers practical examples using Path
         </tr>
         <tr>
             <td class="text-center">
-                <a href="https://github.com/pathwaycom/llm-app/tree/main/examples/pipelines/unstructured_to_sql_on_the_fly">Unstructured to SQL</a>
+                <a href="https://github.com/pathwaycom/llm-app/tree/main/templates/unstructured_to_sql_on_the_fly">Unstructured to SQL</a>
             </td>
             <td class="text-center">
                 The example extracts and structures the data out of unstructured data (PDFs and queries) on the fly.

docs/2.developers/4.user-guide/50.llm-xpack/10.overview.md

@@ -84,7 +84,7 @@ With these tools it is easy to create in Pathway a pipeline serving as a [`Docum
 
 To make interaction with DocumentStore easier you can also use [`DocumentStoreServer`](/developers/api-docs/pathway-xpacks-llm/servers#pathway.xpacks.llm.servers.DocumentStoreServer) that handles API calls.
 
-You can learn more about Document Store in Pathway in a [dedicated tutorial](/developers/user-guide/llm-xpack/docs-indexing) and check out a QA app example in [the llm-app repository](https://github.com/pathwaycom/llm-app/blob/main/examples/pipelines/demo-question-answering/app.py). 
+You can learn more about Document Store in Pathway in a [dedicated tutorial](/developers/user-guide/llm-xpack/docs-indexing) and check out a QA app example in [the llm-app repository](https://github.com/pathwaycom/llm-app/blob/main/templates/question_answering_rag/app.py). 
 
 ### Integrating with LlamaIndex and LangChain
 

docs/2.developers/7.templates/20.run-a-template.md

@@ -59,10 +59,10 @@ Whether you need a real-time ETL, document indexing, or context-based Q&A, you'l
     ::
 ::
 
-Then you need to go the repository of the chosen template, let's take the `demo-question-answering` as an example.
+Then you need to go the repository of the chosen template, let's take the `question_answering_rag` as an example.
 
 ```
-cd llm-app/examples/pipelines/demo-question-answering
+cd llm-app/templates/question_answering_rag
 ```
 
 ## Configuring Pathway Templates

docs/2.developers/7.templates/30.configure-yaml.md

@@ -80,7 +80,7 @@ embedder: !pw.xpacks.llm.embedders.OpenAIEmbedder
 ```
 
 ### Environment Variables
-You can also use `$` to refer to environment variables. For that purpose, you need to use identifiers that consist only of upper case letters and `_`. Also, if a variable is defined in the YAML file and present in the environment variables, the definition from the YAML file takes precedence. As an example, you can set the `port` in the [demo-question-answering pipeline](https://github.com/pathwaycom/llm-app/tree/main/examples/pipelines/demo-question-answering) to be taken from the `$PATHWAY_PORT` environment variable.
+You can also use `$` to refer to environment variables. For that purpose, you need to use identifiers that consist only of upper case letters and `_`. Also, if a variable is defined in the YAML file and present in the environment variables, the definition from the YAML file takes precedence. As an example, you can set the `port` in the [question_answering_rag pipeline](https://github.com/pathwaycom/llm-app/tree/main/templates/question_answering_rag) to be taken from the `$PATHWAY_PORT` environment variable.
 
 ```yaml
 port: $PATHWAY_PORT
@@ -93,11 +93,11 @@ If the values of the environment variables are a valid integer, float or boolean
 
 <!-- TODO: consider writing about pw.load_yaml, but it's not in the api docs currently. -->
 
-## Example: Demo-Question-Answering
+## Example: Question-Answering RAG
 
-To see YAMLs in practice let's look at the [demo-question-answering pipeline](https://github.com/pathwaycom/llm-app/tree/main/examples/pipelines/demo-question-answering). Note, that it differs from [adaptive RAG](https://github.com/pathwaycom/llm-app/tree/main/examples/pipelines/adaptive-rag), [multimodal RAG](https://github.com/pathwaycom/llm-app/tree/main/examples/pipelines/gpt_4o_multimodal_rag) and [private RAG](https://github.com/pathwaycom/llm-app/tree/main/examples/pipelines/private-rag) by the YAML configuration file - their Python code is the same.
+To see YAMLs in practice let's look at the [Question-Answering RAG](https://github.com/pathwaycom/llm-app/tree/main/templates/question_answering_rag). Note, that it differs from [adaptive RAG](https://github.com/pathwaycom/llm-app/tree/main/templates/adaptive_rag), [multimodal RAG](https://github.com/pathwaycom/llm-app/tree/main/multimodal_rag) and [private RAG](https://github.com/pathwaycom/llm-app/tree/main/templates/private_rag) by the YAML configuration file - their Python code is the same.
 
-Here is the content of `app.yaml` from demo-question-answering:
+Here is the content of `app.yaml` from question_answering_rag:
 ```yaml
 $sources:
   - !pw.io.fs.read
@@ -227,7 +227,7 @@ question_answerer: !pw.xpacks.llm.question_answering.BaseRAGQuestionAnswerer
   indexer: $document_store
 ```
 
- If you want to change the provider of LLM models, you can change values of `llm` and `embedder`. By changing `llm` to be `LiteLLMChat`, that uses local `api_base`, and `embedder` to be `SentenceTransformerEmbedder` you obtain a local RAG that does not call external services (this pipeline is now very similar to [private RAG](https://github.com/pathwaycom/llm-app/tree/main/examples/pipelines/private-rag) from the llm-app). 
+ If you want to change the provider of LLM models, you can change values of `llm` and `embedder`. By changing `llm` to be `LiteLLMChat`, that uses local `api_base`, and `embedder` to be `SentenceTransformerEmbedder` you obtain a local RAG that does not call external services (this pipeline is now very similar to [private RAG](https://github.com/pathwaycom/llm-app/tree/main/templates/private_rag) from the llm-app). 
 
 ```yaml
 $sources:

docs/2.developers/7.templates/35.custom-components.md

@@ -53,7 +53,7 @@ def augment_metadata(sources: list[pw.Table]) -> list[pw.Table]:
     return [t.with_columns(_metadata=add_isin_and_currency(pw.this._metadata)) for t in sources]
 ```
 
-Then, in the `app.yaml` from [demo-question-answering](https://github.com/pathwaycom/llm-app/tree/main/examples/pipelines/demo-question-answering) apply this function on `$sources` to obtain the changed tables, which are then given to the `$document_store`.
+Then, in the `app.yaml` from [question_answering_rag](https://github.com/pathwaycom/llm-app/tree/main/templates/question_answering_rag) apply this function on `$sources` to obtain the changed tables, which are then given to the `$document_store`.
 
 
 ```yaml [app.yaml]

docs/2.developers/7.templates/39.yaml-snippets/20.rag-configuration-examples.md

@@ -133,7 +133,7 @@ $llm: !pw.xpacks.llm.llms.LiteLLMChat
   temperature: 0
   api_base: "http://localhost:11434"  
 ```
-You can learn more about this template by visiting its associated public [GitHub project](https://github.com/pathwaycom/llm-app/tree/main/examples/pipelines/private-rag).
+You can learn more about this template by visiting its associated public [GitHub project](https://github.com/pathwaycom/llm-app/tree/main/templates/private_rag).
 
 ::
 ::openable-list

docs/2.developers/7.templates/40.rag-customization/20.custom-prompt.md

@@ -9,8 +9,8 @@ heading: false
 
 In the RAG templates, you can customize the LLM Q&A prompt directly in the YAML configuration file. You just need to set the `prompt_template` argument for the `BaseRAGQuestionAnswerer`.
 
-Using [`demo-question-answering`](https://github.com/pathwaycom/llm-app/tree/main/examples/pipelines/demo-question-answering) as an example, let's see how to customize the prompt. 
-By default, the `BaseRAGQuestionAnswerer` in the [`app.yaml`](https://github.com/pathwaycom/llm-app/blob/main/examples/pipelines/demo-question-answering/app.yaml) is initialized with:
+Using [`question_answering_rag`](https://github.com/pathwaycom/llm-app/tree/main/templates/question_answering_rag) as an example, let's see how to customize the prompt. 
+By default, the `BaseRAGQuestionAnswerer` in the [`app.yaml`](https://github.com/pathwaycom/llm-app/blob/main/templates/question_answering_rag/app.yaml) is initialized with:
 
 ```yaml
 question_answerer: !pw.xpacks.llm.question_answering.BaseRAGQuestionAnswerer

docs/2.developers/7.templates/rag/.multimodal-rag/article.py

@@ -210,7 +210,7 @@
 # default file you can use to test
 # to use your own data via the Colab UI, click on the 'Files' tab in the left sidebar, go to data folder (that was created prior to this) then drag and drop your files there.
 
-# !wget -q -P ./data/ https://github.com/pathwaycom/llm-app/raw/main/examples/pipelines/gpt_4o_multimodal_rag/data/20230203_alphabet_10K.pdf
+# !wget -q -P ./data/ https://github.com/pathwaycom/llm-app/raw/main/templates/multimodal_rag/data/20230203_alphabet_10K.pdf
 
 # + [markdown] id="D7HFGv7ZFl_g"
 # #### **Read Documents**
@@ -296,7 +296,7 @@
 # + [markdown] id="S3zsr-NGop8B"
 # ## **Conclusion**
 # This is how you can easily implement a Multimodal RAG Pipeline using GPT-4o and Pathway. You used the [BaseRAGQuestionAnswerer](https://pathway.com/developers/api-docs/pathway-xpacks-llm/question_answering#pathway.xpacks.llm.question_answering.BaseRAGQuestionAnswerer) class from [pathway.xpacks](https://pathway.com/developers/user-guide/llm-xpack/overview), which integrates the foundational components for our RAG application, including data ingestion, LLM integration, database creation and querying, and serving the application on an endpoint. For more advanced RAG options, you can explore [rerankers](https://pathway.com/developers/api-docs/pathway-xpacks-llm/rerankers#pathway.xpacks.llm.rerankers.CrossEncoderReranker) and the [adaptive RAG example](https://pathway.com/developers/showcases/adaptive-rag).
-# For implementing this example using open source LLMs, here’s a [private RAG app template](https://pathway.com/developers/showcases/private-rag-ollama-mistral) that you can use as a starting point. It will help you run the entire application locally making it ideal for use-cases with sensitive data and explainable AI needs. You can do this within Docker as well by following the steps in [Pathway’s LLM App templates](https://github.com/pathwaycom/llm-app/tree/main/examples/pipelines/gpt_4o_multimodal_rag) repository.
+# For implementing this example using open source LLMs, here’s a [private RAG app template](https://pathway.com/developers/showcases/private-rag-ollama-mistral) that you can use as a starting point. It will help you run the entire application locally making it ideal for use-cases with sensitive data and explainable AI needs. You can do this within Docker as well by following the steps in [Pathway’s LLM App templates](https://github.com/pathwaycom/llm-app/tree/main/templates/multimodal_rag) repository.
 #
 #
 # To explore more app templates and advanced use cases, visit [Pathway App Templates](https://pathway.com/developers/showcases) or Pathway’s [official blog](https://pathway.com/blog).