add rag guide

HomelessDinosaur · HomelessDinosaur · commit 23fbfa8c1c69 · 2024-12-24T11:27:11.000+11:00
diff --git a/docs/guides/python/llama-rag.mdx b/docs/guides/python/llama-rag.mdx
@@ -0,0 +1,290 @@
+---
+description: 'Transform your LLMs with Retrieval Augmented Generation'
+tags:
+  - API
+  - AI & Machine Learning
+languages:
+  - python
+published_at: 2024-11-16
+updated_at: 2024-11-16
+---
+
+# Using Retrieval Augmented Generation to enhance your LLMs
+
+This guide shows how to use Retrieval Augmented Generation (RAG) to enhance a large language model (LLM). RAG is the process of enabling an LLM to reference context outside of its intial training data before generating its response. It can be extremely expensive in both time and computing power to train a model that is useful for your own domain-specific purposes. Therefore, using RAG is a cost-effective alternative to extending the capabilities of an existing LLM.
+
+## Prerequisites
+
+- [uv](https://docs.astral.sh/uv/#getting-started) - for Python dependency management
+- The [Nitric CLI](/get-started/installation)
+- _(optional)_ An [AWS](https://aws.amazon.com) account
+
+## Getting started
+
+We'll start by creating a new project using Nitric's python starter template.
+
+<Note>
+  If you want to take a look at the finished code, it can be found
+  [here](https://github.com/nitrictech/examples/tree/main/v1/llama-rag).
+</Note>
+
+```bash
+nitric new llama-rag py-starter
+cd llama-rag
+```
+
+Next, let's install our base dependencies, then add the `llama-index` libraries. We'll be using [llama index](https://docs.llamaindex.ai/en/stable/) as it makes creating RAGs extremely simple and has support for running our own local Llama 3.2 models.
+
+```bash
+# Install the base dependencies
+uv sync
+# Add Llama index dependencies
+uv add llama-index llama-index-embeddings-huggingface llama-index-llama-cpp
+```
+
+We'll organize our project structure like so:
+
+```text
++--common/
+|  +-- __init__.py
+|  +-- model_parameters.py
++--model/
+|  +-- Llama-3.2-1B-Instruct-Q4_K_M.gguf
++--services/
+|  +-- api.py
++--.gitignore
++--.python-version
++-- build_query_engine.py
++-- pyproject.toml
++-- python.dockerfile
++-- python.dockerignore
++-- nitric.yaml
++-- README.md
+```
+
+## Setting up our LLM
+
+Before we even start writing code for our LLM we'll want to download the model into our project. For this project we'll be using Llama 3.2 with a Q4_K_M quant.
+
+```bash
+mkdir model
+cd model
+curl -OL https://huggingface.co/bartowski/Llama-3.2-1B-Instruct-GGUF/resolve/main/Llama-3.2-1B-Instruct-Q4_K_M.gguf
+cd ..
+```
+
+Now that we have our model we can load it into our code. We'll also define our [embed model](https://docs.llamaindex.ai/en/stable/module_guides/models/embeddings/) - for vectorising our documentation - using a recommend [embed model](https://huggingface.co/BAAI/bge-large-en-v1.5) from Hugging Face. At this point we can also create a prompt template for prompts with our query engine. It will just sanitise some of the hallucinations so that if the model does not know an answer it won't pretend like it does.
+
+```python title:common/model_paramters.py
+from llama_index.core import ChatPromptTemplate
+from llama_index.embeddings.huggingface import HuggingFaceEmbedding
+from llama_index.llms.llama_cpp import LlamaCPP
+
+
+# Load the locally stored Llama model
+llm = LlamaCPP(
+  model_url=None,
+  model_path="./model/Llama-3.2-1B-Instruct-Q4_K_M.gguf",
+  temperature=0.7,
+  verbose=False,
+)
+
+# Load the embed model from hugging face
+embed_model = HuggingFaceEmbedding(model_name="BAAI/bge-large-en-v1.5", trust_remote_code=True)
+
+# Set the location that we will persist our embeds
+persist_dir = "query_engine_vectors"
+
+# Create the prompt query templates to sanitise hallucinations
+text_qa_template = ChatPromptTemplate.from_messages([
+  (
+    "system",
+    "If the context is not useful, respond with 'I'm not sure'.",
+  ),
+  (
+    "user",
+    (
+      "Context information is below.\n"
+      "---------------------\n"
+      "{context_str}\n"
+      "---------------------\n"
+      "Given the context information and not prior knowledge "
+      "answer the question: {query_str}\n."
+    )
+  ),
+])
+```
+
+## Building a Query Engine
+
+The next step is where we embed our context into the LLM. For this example we can embed the Nitric documentation to allow searchability using the LLM. It's open-source on [GitHub](https://github.com/nitrictech/docs), so we can clone it into our project.
+
+```bash
+git clone https://github.com/nitrictech/docs.git nitric-docs
+```
+
+We can then create our embedding and store it locally.
+
+```python title:build_query_engine.py
+from common.model_parameters import llm, embed_model, persist_dir
+
+from llama_index.core import SimpleDirectoryReader, VectorStoreIndex, Settings
+
+
+# Set global settings for llama index
+Settings.llm = llm
+Settings.embed_model = embed_model
+
+# Load data from the documents directory
+loader = SimpleDirectoryReader(
+    # The location of the documents you want to embed
+    input_dir = "./nitric-docs/",
+    # Set the extension to what format your documents are in
+    required_exts=[".mdx"],
+    # Search through documents recursively
+    recursive=True
+)
+docs = loader.load_data()
+
+# Embed the docs into the Llama model
+index = VectorStoreIndex.from_documents(docs, show_progress=True)
+
+# Save the query engine index to the local machine
+index.storage_context.persist(persist_dir)
+```
+
+You can then run this using the following command. This should output the embeds into your `persist_dir`.
+
+```bash
+uv run build_query_engine.py
+```
+
+## Creating an API for querying our model
+
+With our LLM ready for querying, we can create an API to handle prompts.
+
+```python
+import os
+
+from common.model_parameters import embed_model, llm, text_qa_template, persist_dir
+
+from nitric.resources import api
+from nitric.context import HttpContext
+from nitric.application import Nitric
+from llama_index.core import StorageContext, load_index_from_storage, Settings
+
+# Set global settings for llama index
+Settings.llm = llm
+Settings.embed_model = embed_model
+
+main_api = api("main")
+
+@main_api.post("/prompt")
+async def query_model(ctx: HttpContext):
+  # Pull the data from the request body
+  query = str(ctx.req.data)
+
+  print(f"Querying model: \"{query}\"")
+
+  # Get the model from the stored local context
+  if os.path.exists(persist_dir):
+    storage_context = StorageContext.from_defaults(persist_dir=persist_dir)
+
+    index = load_index_from_storage(storage_context)
+
+    # Get the query engine from the index, and use the prompt template for santisation.
+    query_engine = index.as_query_engine(streaming=False, similarity_top_k=4, text_qa_template=text_qa_template)
+  else:
+    print("model does not exist")
+    ctx.res.success= False
+    return ctx
+
+  # Query the model
+  response = query_engine.query(query)
+
+  ctx.res.body = f"{response}"
+
+  print(f"Response: \n{response}")
+
+  return ctx
+
+Nitric.run()
+```
+
+## Test it locally
+
+Now that you have an API defined, we can test it locally. You can do this using `nitric start` and make a request to the API either through the [Nitric Dashboard](/get-started/foundations/projects/local-development#local-dashboard) or another HTTP client like cURL.
+
+```bash
+curl -X POST http://localhost:4001/prompt -d "What is Nitric?"
+```
+
+This should produce an output similar to:
+
+```text
+Nitric is a cloud-agnostic framework designed to aid developers in building full cloud applications, including infrastructure. It is a declarative cloud framework with common resources like APIs, websockets, databases, queues, topics, buckets, and more. The framework provides tools for locally simulating a cloud environment, to allow an application to be tested locally, and it makes it possible to interact with resources at runtime. It is a lightweight and flexible framework that allows developers to structure their projects according to their preferences and needs. Nitric is not a replacement for IaC tools like Terraform but rather introduces a method of bringing developer self-service for infrastructure directly into the developer application. Nitric can be augmented through use of tools like Pulumi or Terraform and even be fully customized using such tools. The framework supports multiple programming languages, and its default deployment engines are built with Pulumi. Nitric provides tools for defining services in your project's `nitric.yaml` file, and each service can be run independently, allowing your app to scale and manage different workloads efficiently. Services are the heart of Nitric apps, they're the entrypoints to your code. They can serve as APIs, websockets, schedule handlers, subscribers and a lot more.
+```
+
+## Get ready for deployment
+
+Now that its tested locally, we can get our project ready for containerization. The default python dockerfile uses `python3.11-bookworm-slim` as its basic container image, which doesn't have the right dependencies to load the Llama model. So, all we need to do is update the Dockerfile to use python3.11-bookworm (the non-slim version) instead.
+
+Update line 2:
+
+```dockerfile title:python.dockerfile
+# !diff -
+FROM ghcr.io/astral-sh/uv:python3.11-bookworm-slim AS builder
+# !diff +
+FROM ghcr.io/astral-sh/uv:python3.11-bookworm AS builder
+```
+
+And line 18:
+
+```dockerfile title:python.dockerfile
+# !diff -
+FROM python:3.11-slim-bookworm
+# !diff +
+FROM python:3.11-bookworm
+```
+
+When you're ready to deploy the project, we can create a new Nitric stack file that will target AWS:
+
+```bash
+nitric stack new dev aws
+```
+
+Update the stack file `nitric.dev.yaml` with the appropriate AWS region and memory allocation to handle the model:
+
+```yaml title:nitric.dev.yaml
+provider: nitric/aws@1.14.0
+region: us-east-1
+config:
+  # How services will be deployed by default, if you have other services not running models
+  # you can add them here too so they don't use the same configuration
+  default:
+    lambda:
+      # Set the memory to 6GB to handle the model, this automatically sets additional CPU allocation
+      memory: 6144
+      # Set a timeout of 30 seconds (this is the most API Gateway will wait for a response)
+      timeout: 30
+      # We add more storage to the lambda function, so it can store the model
+      ephemeral-storage: 1024
+```
+
+We can then deploy using the following command:
+
+```bash
+nitric up
+```
+
+Testing on AWS will be the same as we did locally, we'll just use cURL to make a request to the API URL that was outputted at the end of the deployment.
+
+```bash
+curl -x POST {your AWS endpoint URL here}/prompt -d "What is Nitric?"
+```
+
+Once you're finished querying the model, you can destroy the deployment using `nitric down`.
+
+## Summary
+
+In this project we've successfully augmented an LLM using Retrieval Augmented Generation (RAG) techniques with Llama Index and Nitric. You can modify this project to use any LLM, change the prompt template to be more specific in responses, or change the context for your own personal requirements. We could extend this project to maintain context between requests using WebSockets to have more of a chat-like experience with the model.
