Updates

Author: theresa-i

learn-pr/wwl-data-ai/introduction-language-models-databricks/includes/02-what-is-generative-ai.md

@@ -25,7 +25,7 @@ The underlying technology involves training on diverse text corpora, allowing th
 
 In the realm of visual arts, generative AI is making significant strides with the development of **Generative Adversarial Networks** (**GANs**).
 
-GANs consist of two neural networks—a **generator** and a **discriminator—that work in tandem to create realistic images. The generator creates images, while the discriminator evaluates them, leading to the production of increasingly authentic visuals over time. This technology is used to create stunning artwork, realistic human faces, and even design new products.
+GANs consist of two neural networks—a **generator** and a **discriminator**—that work in tandem to create realistic images. The generator creates images, while the discriminator evaluates them, leading to the production of increasingly authentic visuals over time. This technology is used to create stunning artwork, realistic human faces, and even design new products.
 
 The ability to generate high-quality images also finds applications in industries like fashion, where AI designs clothing, and in entertainment, where it creates special effects and virtual characters.
 

learn-pr/wwl-data-ai/introduction-language-models-databricks/includes/04-key-components-llms.md

@@ -80,17 +80,21 @@ Transformers consist of layers of **encoders** and **decoders** that work togeth
 Let's use this diagram as an example of how LLM processing works.
 
 The **LLM** is trained on a large volume of natural language text.
-**Step1: Input** Training documents and a prompt "When my dog was..." enter the system.
-**Step 2: Encoder (The analyzer)** Breaks text into **tokens** and analyzes its meaning. The **encoder** block processes token sequences using **self-attention** to determine the relationships between tokens or words.
-**Step 3: Embeddings are created** The output from the encoder is a collection of **vectors** (multi-valued numeric arrays) in which each element of the vector represents a semantic attribute of the tokens. These vectors are referred to as **embeddings**. They're numerical representations that capture meaning:
 
-- **dog [10,3,2]** - animal, pet, subject
-- **cat [10,3,1]** - animal, pet, different species
-- **puppy [5,2,1]** - young animal, related to dog
-- **skateboard [-3,3,2]** - object, unrelated to animals
+- **Step 1: Input** - Training documents and a prompt "When my dog was..." enter the system.
 
-**Step 4: Decoder (The writer)** block works on a new sequence of text tokens and uses the embeddings generated by the encoder to generate an appropriate natural language output. It compares the options and chooses the most appropriate response.
-**Step 5: Output generated** Given an input sequence like `When my dog was`, the model can use the self-attention mechanism to analyze the input tokens and the semantic attributes encoded in the embeddings to predict an appropriate completion of the sentence, such as `a puppy`.
+- **Step 2: Encoder (The analyzer)** - Breaks text into **tokens** and analyzes its meaning. The **encoder** block processes token sequences using **self-attention** to determine the relationships between tokens or words.
+
+- **Step 3: Embeddings are created** - The output from the encoder is a collection of **vectors** (multi-valued numeric arrays) in which each element of the vector represents a semantic attribute of the tokens. These vectors are referred to as **embeddings**. They're numerical representations that capture meaning:
+
+  - **dog [10,3,2]** - animal, pet, subject
+  - **cat [10,3,1]** - animal, pet, different species
+  - **puppy [5,2,1]** - young animal, related to dog
+  - **skateboard [-3,3,2]** - object, unrelated to animals
+
+- **Step 4: Decoder (The writer)** - The decoder block works on a new sequence of text tokens and uses the embeddings generated by the encoder to generate an appropriate natural language output. It compares the options and chooses the most appropriate response.
+
+- **Step 5: Output generated** - Given an input sequence like `When my dog was`, the model can use the self-attention mechanism to analyze the input tokens and the semantic attributes encoded in the embeddings to predict an appropriate completion of the sentence, such as `a puppy`.
 
 This architecture is highly parallelizable, making it efficient for training on large datasets. The size of the LLM, often defined by the number of parameters, determines its capacity to store linguistic knowledge and perform complex tasks. Think of parameters as millions or billions of tiny memory cells that store language rules and patterns. More memory cells mean the model can remember more about language and handle harder tasks. Large models, such as GPT-3 and GPT-4, contain billions of parameters, allowing them to store vast language knowledge.
 

learn-pr/wwl-data-ai/introduction-language-models-databricks/includes/05-use-llms.md

@@ -8,7 +8,7 @@ These tasks include:
 - **Zero-shot classification**: Categorize text into predefined labels without prior examples.
 - **Few-shot learning**: Adapt to new tasks with minimal training data.
 
-These versatile applications demonstrate the transformative potential of LLMs in processing and understanding human language.
+These applications demonstrate the transformative potential of LLMs in processing and understanding human language.
 
 Let's explore each of these tasks in more detail.
 

learn-pr/wwl-data-ai/retrieval-augmented-generation-azure-databricks/includes/1-introduction.md

@@ -2,18 +2,18 @@ Retrieval Augmented Generation (RAG) is a technique in natural language processi
 
 Language models have become incredibly popular because they can generate impressive, well-structured answers to user questions. When people interact with these models through chat interfaces, it feels like a natural and intuitive way to get information.
 
-However, there's a major challenge: ensuring the AI's responses are accurate and factual. This challenge is called **groundedness** - which simply means whether the AI's answer is based on real, reliable information rather than made-up or incorrect details. Without proper groundedness, language models might confidently stating things that aren't true. Another challenge is that traditional models use only information they were trained on that can be outdated or incomplete.
+However, there's a challenge: ensuring the AI's responses are accurate and factual. This challenge is called **groundedness** - which simply means whether the AI's answer is based on real, reliable information rather than made-up or incorrect details. Without proper groundedness, language models might confidently state things that aren't true. Another challenge is that traditional models use only information they were trained on that can be outdated or incomplete.
 
 When you want a language model to have access to specific knowledge, you have three main options:
 
 :::image type="content" source="../media/learn-knowledge.png" alt-text="Diagram of three approaches for language models to learn knowledge.":::
 
-1. **Model pretraining**: Build a language model from the ground up, which requires massive datasets with billions or trillions of text pieces or tokens. This is extremely expensive and time-consuming.
+- **Model pretraining**: Build a language model from the ground up, which requires massive datasets with billions or trillions of text pieces or tokens. This is extremely expensive and time-consuming.
 
-2. **Model fine-tuning**: Take an existing language model and train it further on your specific data or industry, which requires thousands of specialized examples. This is moderately expensive and complex.
+- **Model fine-tuning**: Take an existing language model and train it further on your specific data or industry, which requires thousands of specialized examples. This is moderately expensive and complex.
 
-3. **Passing contextual information**: Connect a language model to external databases or documents so it can look up information in real-time. This is a strategy known as Retrieval Augmented Generation (RAG). This requires setting up a knowledge base but is much simpler than the other options.
+- **Passing contextual information**: Connect a language model to external databases or documents so it can look up information in real-time. This is a strategy known as Retrieval Augmented Generation (RAG). This requires setting up a knowledge base but is much simpler than the other options.
 
-RAG is most practical when you need an AI with access to current, verifiable information. It's easier to implement and uses less computing power than retraining entire models.
+RAG is most practical when you need an AI system with access to current, verifiable information. It's easier to implement and uses less computing power than retraining entire models.
 
-In this module, you'll learn when and how to use RAG to make language models more reliable and accurate. You'll also discover how vector search technology helps the AI quickly find the most relevant information to include in its responses.
+In this module, you'll learn when and how to use RAG to make language models more reliable and accurate. You'll also discover how vector search technology helps AI quickly find the most relevant information to include in its responses.

learn-pr/wwl-data-ai/retrieval-augmented-generation-azure-databricks/includes/2-workflow.md

@@ -8,7 +8,7 @@
 
 3. **Add Context to Prompt**: The relevant information retrieved from your documents is combined with the user's original question to create an enhanced prompt that provides the LLM with the specific context it needs.
 
-4. **LLM Generates Response**: The base language model processes both the original question and the retrieved context from your documents to generate an accurate, grounded response based on your specific data.**
+4. **LLM Generates Response**: The base language model processes both the original question and the retrieved context from your documents to generate an accurate, grounded response based on your specific data.
 
 This process bridges the gap between a general-purpose LLM and your specific, private, or recent information, allowing you to get accurate answers based on your own documents without having to retrain the entire base model.
 
@@ -18,9 +18,9 @@ Let's look at when you can use RAG, and then review the main components and conc
 
 You can use RAG for chatbots, search enhancement, and content creation and summarization.
 
-**Chatbots**: RAG helps chatbots provide more accurate answers by accessing your current company information. When integrated with customer support systems, RAG-powered chatbots can automate support and quickly resolve customer questions using up-to-date data.
+**Chatbots**: RAG helps chatbots provide more accurate answers by accessing current information. When integrated with customer support systems, RAG-powered chatbots can automate support and quickly resolve customer questions using up-to-date data.
 
-**Search enhancement**: Instead of returning just links and snippets, RAG-powered search engines provide complete, conversational answers. Users get comprehensive responses that synthesize information from multiple sources, making it much easier to find what they need.
+**Search enhancement**: Instead of returning just links and snippets, RAG-powered search engines provide complete, conversational answers. Users get comprehensive responses that synthesize information from multiple sources, making it easy to find what they need.
 
 **Content creation and summarization**: Produce high-quality, fact-based content using your own data sources. RAG enables you to generate informed articles, create summaries from lengthy documents, and develop reports that synthesize information from multiple sources.
 
@@ -45,7 +45,7 @@ Document embedding, as shown in the diagram, is part of a preparation phase. Thi
 
 :::image type="content" source="../media/document-embedding.png" alt-text="Diagram of embeddings model converting documents to vectors.":::
 
-Query embedding, shown in the diagram, happens each time a user asks a question. First, the user's question is converted into an embedding using the same embedding model. This real-time conversion prepares the query for comparison against your preprocessed document embeddings. Only after the query is embedded can the system begin searching for relevant documents.
+Query embedding, shown in the diagram, happens each time a user asks a question. First, the user's question is converted into an embedding using the same embedding model that was used to process the documents. This real-time conversion prepares the query for comparison against your preprocessed document embeddings. Only after the query is embedded can the system begin searching for relevant documents.
 
 :::image type="content" source="../media/query-embedding.png" alt-text="Diagram of embeddings model.":::
 
@@ -63,21 +63,21 @@ The vector store enables semantic search, which means it finds relevant content
 
 ### Augment your prompt with retrieved content
 
-After finding the most relevant documents, the RAG system combines this information with the user's original question to create an "enhanced prompt" that gives the LLM everything it needs to provide an accurate answer.
+After finding the most relevant documents, the RAG system combines this information with the user's original question to create an "augmented prompt" that gives the LLM everything it needs to provide an accurate answer.
 
 The augmentation process looks like this:
 
 1. Start with the user's question: "What's our vacation policy?"
 2. Add retrieved context: Include relevant excerpts from your HR documents
-3. Create enhanced prompt: "Based on these HR policy documents: [retrieved content], what's our vacation policy?"
+3. Create augmented prompt: "Based on these HR policy documents: [retrieved content], what's our vacation policy?"
 The LLM now has both the user's question **and** the specific information needed to answer it accurately. This is called "in-context learning" because the LLM learns from the context provided in the prompt rather than from its original training data.
 
-In the final step, the enhanced prompt is sent to the Large Language Model (LLM), which generates a response based on both the question and the retrieved information. The LLM can include citations of the original sources, allowing users to verify where the information came from.
+In the final step, the augmented prompt is sent to the Large Language Model (LLM), which generates a response based on both the question and the retrieved information. The LLM can include citations of the original sources, allowing users to verify where the information came from.
 
 The key benefit of the RAG workflows is that it gives you accurate, source-backed answers without having to retrain the entire language model on your specific documents.
 
 ## RAG architecture overview
-The complete RAG workflow combines all the components we've discussed into a unified system that transforms general-purpose LLMs into knowledgeable assistants for your specific domain.
+The complete RAG workflow combines all the components we've reviewed into a unified system that transforms general-purpose LLMs into knowledgeable assistants for your specific domain.
 
 The key mechanism is **in-context learning** - instead of retraining the LLM, you provide relevant information as context in each prompt, allowing the LLM to generate informed responses without permanent modification.
 

learn-pr/wwl-data-ai/retrieval-augmented-generation-azure-databricks/includes/4-vector-search.md

@@ -12,7 +12,7 @@ This mathematical representation is what makes vector search possible - instead
 
 ## How vector search works
 
-Vector search builds on the text embedding process. Here's how it works step-by-step:
+Vector search builds on the text embedding process. Here's how it works step by step:
 
 1. **Query conversion**: When a user asks a question, your system converts their query into an embedding using the same model you used for your document data chunks.
 2. **Similarity calculation**: The system compares the query embedding to all document chunk embeddings using mathematical similarity measures.
@@ -30,29 +30,23 @@ This semantic understanding comes from the embedding model, which learned these
 
 ## Choosing your vector search approach
 
-You have two main options for implementing vector search in Azure Databricks: vector databases and vector libraries. Let's explore each of these options, so you can determine which approach best fits your needs.
+There are two main options for implementing vector search in Azure Databricks: vector databases and vector libraries. Let's explore each of these options.
 
-### Vector databases: For dynamic, large-scale data
-
-**What is a vector database?**
+### What is a vector database?
 
 A vector database is a specialized database optimized to store and retrieve embeddings - those vectors with hundreds or thousands of numbers that represent meaning. Like traditional databases, vector databases use indices (organized structures that speed up searches) to quickly find relevant data, but these vector indices are designed to find mathematically similar vectors rather than exact matches. In RAG applications, vector databases primarily store text embeddings - vectors that represent the semantic meaning of your document chunks - along with metadata about each chunk (like source document, page numbers, or categories).
 
-:::image type="content" source="../media/matched-query.png" alt-text="Vector space visualization showing document vectors as blue dots and a query vector as an orange dot, with the relevant document vector positioned close to the query vector.":::
+:::image type="content" source="../media/matched-query.png" alt-text="Vector space visualization showing both document and query vectors as dots. A relevant document vector is positioned close to the query vector.":::
 
 This visualization shows how vectors work in practice. Each dot represents a vector - the blue dots are document chunk embeddings stored in the vector database, and the orange dot is a query vector. The image labels "Relevant document" and "Query" indicate vectors that are close together in the mathematical space, showing similarity. When you search, the database finds document vectors nearest to your query vector.
 
 This image illustrates why vector databases are so powerful for RAG: documents with similar meanings are positioned close together in the vector space, making it easy to find relevant content by measuring mathematical distance between vectors.
 
-**Azure Databricks option: Mosaic AI Vector Search**
-
 In Azure Databricks, you can use Mosaic AI Vector Search as a vector database to store the vector representations of your data and metadata. Mosaic AI Vector Search integrates with your Lakehouse data and provides search capabilities for your embeddings, with support for filtering results based on metadata you've stored with your document chunks.
 
 Vector databases like Mosaic AI Vector Search can be used when you have large amounts of data and need to store your embeddings persistently for long-term use. Vector databases are well-suited for dynamic data because they support real-time operations - you can add new document chunks, update existing ones, or delete outdated content without rebuilding the entire search system. This makes them ideal for scenarios like knowledge bases that grow over time, document repositories with frequent updates, or applications where multiple users are adding content. They typically provide search capabilities and support filtering on metadata, making them applicable for scenarios where multiple applications need to access the same vector data.
 
-### Vector libraries: For smaller, static datasets
-
-**What are vector libraries?**
+### What are vector libraries?
 
 Vector libraries are tools that create vector indices to enable fast similarity search without requiring a separate database system. Think of these indices like a specialized filing system that organizes your embeddings so they can be searched quickly - similar to how a database index speeds up queries, but designed for mathematical similarity searches.