Managed database setup

Author: Laure-di

tutorials/how-to-implement-rag/index.mdx

@@ -1,10 +1,10 @@
 ---
 meta:
    title: How to implement RAG with managed inference
-   description:
+   description: Learn how to implement Retrieval-Augmented Generation (RAG) using Scaleway's managed inference, PostgreSQL, pgvector, and object storage.
 content:
     h1: How to implement RAG with managed inference
-tags: inference managed postgresql pgvector object storage
+tags: inference, managed, postgresql, pgvector, object storage
 categories:
     - inference
 ---
@@ -14,47 +14,78 @@ Scaleway's robust infrastructure makes it easier than ever to implement RAG, as
 By utilizing our managed inference services, managed databases, and object storage, you can effortlessly build and deploy a customized model tailored to your specific needs.
 
 <Macro id="requirements" />
+
 - A Scaleway account logged into the [console](https://console.scaleway.com)
 - [Owner](/identity-and-access-management/iam/concepts/#owner) status or [IAM permissions](/identity-and-access-management/iam/concepts/#permission) allowing you to perform actions in the intended Organization
 - [Inference Deployment](/ai-data/managed-inference/how-to/create-deployment/): Set up an inference deployment using [sentence-transformers/sentence-t5-xxl](/ai-data/managed-inference/reference-content/sentence-t5-xxl/) on an L4 instance to efficiently process embeddings.
 - [Inference Deployment](/ai-data/managed-inference/how-to/create-deployment/) with the model of your choice.
 - [Object Storage Bucket](/storage/object/how-to/create-a-bucket/) to store all the data you want to inject into your LLM model.
 - [Managed Database](/managed-databases/postgresql-and-mysql/how-to/create-a-database/) to securely store all your embeddings.
 
-## Configure your developement environnement
+## Configure your development environment
+
 1. Install necessary packages: run the following command to install the required packages:
+
    ```sh
    pip install langchain psycopg2 python-dotenv scaleway
    ```
-2. Configure your environnement variables: create a .env file and add the following variables. These will store your API keys, database connection details, and other configuration values.
+2. Configure your environment variables: create a .env file and add the following variables. These will store your API keys, database connection details, and other configuration values.
+
    ```sh
-  # .env file
-
-  # Scaleway API credentials
-  SCW_ACCESS_KEY=your_scaleway_access_key
-  SCW_SECRET_KEY=your_scaleway_secret_key
-  SCW_API_KEY=your_scaleway_api_key
-
-  # Scaleway project and region
-  SCW_DEFAULT_PROJECT_ID=your_scaleway_project_id
-  SCW_DEFAULT_REGION=your_scaleway_region
-
-  # Scaleway managed database (PostgreSQL) credentials
-  SCW_DB_NAME=your_scaleway_managed_db_name
-  SCW_DB_USER=your_scaleway_managed_db_username
-  SCW_DB_PASSWORD=your_scaleway_managed_db_password
-  SCW_DB_HOST=your_scaleway_managed_db_host  # The IP address of your database instance
-  SCW_DB_PORT=your_scaleway_managed_db_port  # The port number for your database instance
-
-  # Scaleway S3 bucket configuration
-  SCW_BUCKET_NAME=your_scaleway_bucket_name
-  SCW_BUCKET_ENDPOINT=your_scaleway_bucket_endpoint  # S3 endpoint, e.g., https://s3.fr-par.scw.cloud
-
-  # Scaleway Inference API configuration (Embeddings)
-  SCW_INFERENCE_EMBEDDINGS_ENDPOINT=your_scaleway_inference_embeddings_endpoint  # Endpoint for sentence-transformers/sentence-t5-xxl deployment
-  SCW_INFERENCE_API_KEY_EMBEDDINGS=your_scaleway_api_key_for_embeddings
-
-  # Scaleway Inference API configuration (LLM deployment)
-  SCW_INFERENCE_DEPLOYMENT_ENDPOINT=your_scaleway_inference_endpoint  # Endpoint for your LLM deployment
-  SCW_INFERENCE_API_KEY=your_scaleway_api_key_for_inference_deployment
-  ```
+   # .env file
+
+   # Scaleway API credentials
+   SCW_ACCESS_KEY=your_scaleway_access_key
+   SCW_SECRET_KEY=your_scaleway_secret_key
+   SCW_API_KEY=your_scaleway_api_key
+
+   # Scaleway project and region
+   SCW_DEFAULT_PROJECT_ID=your_scaleway_project_id
+   SCW_DEFAULT_REGION=your_scaleway_region
+
+   # Scaleway managed database (PostgreSQL) credentials
+   SCW_DB_NAME=your_scaleway_managed_db_name
+   SCW_DB_USER=your_scaleway_managed_db_username
+   SCW_DB_PASSWORD=your_scaleway_managed_db_password
+   SCW_DB_HOST=your_scaleway_managed_db_host  # The IP address of your database instance
+   SCW_DB_PORT=your_scaleway_managed_db_port  # The port number for your database instance
+
+   # Scaleway S3 bucket configuration
+   SCW_BUCKET_NAME=your_scaleway_bucket_name
+   SCW_BUCKET_ENDPOINT=your_scaleway_bucket_endpoint  # S3 endpoint, e.g., https://s3.fr-par.scw.cloud
+
+   # Scaleway Inference API configuration (Embeddings)
+   SCW_INFERENCE_EMBEDDINGS_ENDPOINT=your_scaleway_inference_embeddings_endpoint  # Endpoint for sentence-transformers/sentence-t5-xxl deployment
+   SCW_INFERENCE_API_KEY_EMBEDDINGS=your_scaleway_api_key_for_embeddings
+
+   # Scaleway Inference API configuration (LLM deployment)
+   SCW_INFERENCE_DEPLOYMENT_ENDPOINT=your_scaleway_inference_endpoint  # Endpoint for your LLM deployment
+   SCW_INFERENCE_API_KEY=your_scaleway_api_key_for_inference_deployment
+   ```
+
+### Set Up Managed Database
+
+1. Connect to your PostgreSQL instance and install the pg_vector extension.
+
+    ```python
+    conn = psycopg2.connect(
+    database="your_database_name",
+    user="your_db_user",
+    password=os.getenv("SCW_DB_PASSWORD"),
+    host="your_db_host",
+    port="your_db_port"
+    )
+
+    cur = conn.cursor()
+
+    # Install pg_vector extension
+    cur.execute("CREATE EXTENSION IF NOT EXISTS vector;")
+    conn.commit()
+    ```
+2. To avoid reprocessing documents that have already been loaded and vectorized, create a table in your PostgreSQL database to track them. This ensures that new documents added to your object storage bucket are processed only once, preventing duplicate downloads and redundant vectorization.
+
+    ```python
+    cur.execute("CREATE TABLE IF NOT EXISTS object_loaded (id SERIAL PRIMARY KEY, object_key TEXT)")
+    conn.commit()
+   ```
+