RAG and Tools Python experiment example

README.md

@@ -88,11 +88,21 @@ You can read more about these in our [Get started with sample projects documenta
 ### 📊 Dataset & Experiment Examples
 
 #### Python Datasets 🐍
+
 - **[Dataset Experiments](/python/dataset-experiments/)** - Managing test data and running controlled experiments
 
+#### Python Experiments
+
+- **[Experiments with RAG and tools](/python/experiments/rag-and-tools/)** - Using RAG and tools in an experiment
+
 #### TypeScript Datasets 📜
+
 - **[Dataset Experiments](/typescript/datasets-experiments/)** - Dataset management and experimentation in TypeScript
 
+#### TypeScript Experiments
+
+- **[Experiments with RAG and tools](/typescript/experiments/rag-and-tools/)** - Using RAG and tools in an experiment
+
 #### 📚 Additional Resources
 
 - [Galileo SDK Documentation](https://v2docs.galileo.ai/sdk-api/overview) - SDK documentation

python/experiments/rag-and-tools/.env.example

@@ -0,0 +1,10 @@
+# Galileo Environment Variables
+GALILEO_API_KEY=                        # Your Galileo API key.
+GALILEO_PROJECT=                        # Your Galileo project name.
+GALILEO_LOG_STREAM=                     # The name of the log stream you want to use for logging.
+
+# Provide the console url below if you are using a custom deployment, and not using app.galileo.ai
+# GALILEO_CONSOLE_URL=                  # Optional if you are using a hosted version of Galileo
+
+# OpenAI Environment Variables
+OPENAI_API_KEY=                         # Your OpenAI API key.

python/experiments/rag-and-tools/README.md

@@ -0,0 +1,60 @@
+# Experiments with RAG and tools
+
+This is an example project demonstrating how to use Galileo experiments with applications thar use RAG and tools.
+
+This code is used in the [Run an experiment against a RAG app](http://v2docs.galileo.ai/how-to-guides/experiments/rag-app/rag-app) how-to guide in the Galileo documentation.
+
+## Get Started
+
+To get started with this project, you'll need to have Python 3.9 or later installed. You can then install the required dependencies in a virtual environment:
+
+```bash
+pip install -r requirements.txt
+```
+
+## Configure environment variables
+
+You will need to configure environment variables to use this project. Copy the `.env.example` file to `.env`, then update the environment variables in the `.env` file with your OpenAI and Galileo values:
+
+```ini
+# Galileo environment variables
+GALILEO_API_KEY=
+GALILEO_PROJECT=
+GALILEO_LOG_STREAM=
+
+# OpenAI environment variables
+OPENAI_API_KEY=
+```
+
+## Usage
+
+This sample contains both an application that generates a fake horoscope using a tool and a mock RAG function, as well as an experiment to test the same code.
+
+To run the application, run:
+
+```bash
+python app.py
+```
+
+Traces will be captured and logged to Galileo.
+
+To run the experiment, run:
+
+```bash
+python experiment.py
+```
+
+A link to the results of the experiment will be written to the console.
+
+## Project Structure
+
+The project structure is as follows:
+
+```folder
+rag-and-tools/
+├── env.example        # List of environment variables
+├── requirements.txt   # Python project requirements
+├── app.py             # The main python application
+├── experiment.py      # Code to run the main application as an experiment
+└── README.md          # Project documentation
+```

python/experiments/rag-and-tools/app.py

@@ -0,0 +1,170 @@
+import json
+
+from galileo import log, galileo_context
+
+from galileo.openai import OpenAI
+
+from dotenv import load_dotenv
+
+# Load environment variables from .env file
+load_dotenv(override=True)
+
+
+# A mock RAG retriever function
+@log(span_type="retriever")
+def retrieve_horoscope_data(sign):
+    """
+    Mock function to simulate retrieving horoscope data for a given sign.
+    This is decorated with logging for tracing a retriever span.
+    """
+    horoscopes = {
+        "Aquarius": [
+            "Next Tuesday you will befriend a baby otter.",
+            "Next Tuesday you will find a dollar on the ground.",
+        ],
+        "Taurus": [
+            "Next Tuesday you will find a four-leaf clover.",
+            "Next Tuesday you will have a great conversation with a stranger.",
+        ],
+        "Gemini": [
+            "Next Tuesday you will learn to juggle.",
+            "Next Tuesday you will discover a new favorite book.",
+        ],
+    }
+    return horoscopes.get(sign, ["No horoscope available."])
+
+
+@log(span_type="tool")
+def get_horoscope(sign):
+    """
+    Tool function to get a horoscope for a given astrological sign.
+    """
+    return "\n".join(retrieve_horoscope_data(sign))
+
+
+# Define a list of callable tools for the model
+tools = [
+    {
+        "type": "function",
+        "function": {
+            "name": "get_horoscope",
+            "description": "Get today's horoscope for an astrological sign.",
+            "parameters": {
+                "type": "object",
+                "properties": {
+                    "sign": {
+                        "type": "string",
+                        "description": "An astrological sign like Taurus or Aquarius",
+                    },
+                },
+                "required": ["sign"],
+            },
+        },
+    },
+]
+
+# Map tool names to their implementations
+available_tools = {"get_horoscope": get_horoscope}
+
+
+# Create the OpenAI client
+client = OpenAI()
+
+
+def call_llm(messages):
+    """
+    Call the LLM with the provided messages and tools.
+    """
+    return client.chat.completions.create(
+        model="gpt-5",
+        tools=tools,
+        messages=messages,
+    )
+
+
+def get_users_horoscope(sign: str) -> str:
+    """
+    Get the user's horoscope
+    """
+    # Create a running message history list we will add to over time
+    message_history = [
+        {
+            "role": "system",
+            "content": """
+            You are a helpful assistant that provides horoscopes.
+            Provide a flowery response based off any information retrieved.
+            Include typical horoscope phrases, and characteristics of
+            the sign in question.
+            """,
+        },
+        {"role": "user", "content": f"What is my horoscope? I am {sign}."},
+    ]
+
+    # Prompt the model with tools defined
+    response = call_llm(message_history)
+
+    # Add the tool call to the message history
+    message_history.append(
+        {
+            "role": "assistant",
+            "tool_calls": [
+                {
+                    "id": response.choices[0].message.tool_calls[0].id,
+                    "type": "function",
+                    "function": {
+                        "name": response.choices[0].message.tool_calls[0].function.name,
+                        "arguments": response.choices[0].message.tool_calls[0].function.arguments,
+                    },
+                }
+            ],
+        }
+    )
+
+    # Call any tools the model requested
+    completion_tool_calls = response.choices[0].message.tool_calls
+
+    for call in completion_tool_calls if completion_tool_calls else []:
+        # Get the tool to call and its arguments
+        tool_to_call = available_tools[call.function.name]
+        args = json.loads(call.function.arguments)
+
+        # Call the tool
+        result = tool_to_call(**args)
+
+        # Add the tool result to the message history
+        message_history.append(
+            {
+                "role": "tool",
+                "content": result,
+                "tool_call_id": call.id,
+                "name": call.function.name,
+            }
+        )
+
+    # Now we call the model again, with the tool results included
+    response = call_llm(message_history)
+
+    # Return the final response from the model
+    return response.choices[0].message.content
+
+
+def main():
+    """
+    Get the user's horoscope
+    """
+    # Start a session and trace
+    galileo_logger = galileo_context.get_logger_instance()
+    galileo_logger.start_session("RAG with Tools Example")
+    galileo_logger.start_trace(input="What is my horoscope? I am Aquarius.", name="Calling LLM with Tool")
+
+    response = get_users_horoscope("Aquarius")
+
+    # Conclude the trace and flush
+    galileo_logger.conclude(response)
+    galileo_logger.flush()
+
+    print(response)
+
+
+if __name__ == "__main__":
+    main()

python/experiments/rag-and-tools/experiment.py

@@ -0,0 +1,42 @@
+import os
+
+from galileo import GalileoScorers
+from galileo.experiments import run_experiment
+
+from app import get_users_horoscope
+
+
+def main():
+    """
+    Run the horoscope experiment
+    """
+    # Define a dataset of astrological signs to use
+    # in the experiment
+    dataset = [
+        {"input": "Aquarius"},
+        {"input": "Taurus"},
+        {"input": "Gemini"},
+        {"input": "Leo"},
+    ]
+
+    # Run the experiment
+    results = run_experiment(
+        "horoscope-experiment-2",
+        dataset=dataset,
+        function=get_users_horoscope,
+        metrics=[
+            GalileoScorers.tool_error_rate,
+            GalileoScorers.tool_selection_quality,
+            GalileoScorers.chunk_attribution_utilization,
+            GalileoScorers.context_adherence,
+        ],
+        project=os.environ["GALILEO_PROJECT"],
+    )
+
+    # Print a link to the experiment results
+    print("Experiment Results:")
+    print(results["link"])
+
+
+if __name__ == "__main__":
+    main()

python/experiments/rag-and-tools/requirements.txt

@@ -0,0 +1,2 @@
+galileo[openai]
+python-dotenv

typescript/experiments/rag-and-tools/.env.example

@@ -0,0 +1,10 @@
+# Galileo Environment Variables
+GALILEO_API_KEY=                        # Your Galileo API key.
+GALILEO_PROJECT=                        # Your Galileo project name.
+GALILEO_LOG_STREAM=                     # The name of the log stream you want to use for logging.
+
+# Provide the console url below if you are using a custom deployment, and not using app.galileo.ai
+# GALILEO_CONSOLE_URL=                  # Optional if you are using a hosted version of Galileo
+
+# OpenAI Environment Variables
+OPENAI_API_KEY=                         # Your OpenAI API key.

typescript/experiments/rag-and-tools/README.md

@@ -0,0 +1,61 @@
+# Experiments with RAG and tools
+
+This is an example project demonstrating how to use Galileo experiments with applications thar use RAG and tools.
+
+This code is used in the [Run an experiment against a RAG app](http://v2docs.galileo.ai/how-to-guides/experiments/rag-app/rag-app) how-to guide in the Galileo documentation.
+
+## Get Started
+
+Install the dependencies:
+
+```bash
+npm i
+```
+
+## Configure environment variables
+
+You will need to configure environment variables to use this project. Copy the `.env.example` file to `.env`, then update the environment variables in the `.env` file with your OpenAI and Galileo values:
+
+```ini
+# Galileo environment variables
+GALILEO_API_KEY=
+GALILEO_PROJECT=
+GALILEO_LOG_STREAM=
+
+# OpenAI environment variables
+OPENAI_API_KEY=
+```
+
+## Usage
+
+This sample contains both an application that generates a fake horoscope using a tool and a mock RAG function, as well as an experiment to test the same code.
+
+To run the application, run:
+
+```bash
+npx tsx app.ts
+```
+
+Traces will be captured and logged to Galileo.
+
+To run the experiment, run:
+
+```bash
+npx tsx experiment.ts
+```
+
+A link to the results of the experiment will be written to the console.
+
+## Project Structure
+
+The project structure is as follows:
+
+```folder
+rag-and-tools/
+├── env.example        # List of environment variables
+├── package.json       # NPM project requirements
+├── package-lock.json  # NPM project requirements
+├── app.ts             # The main python application
+├── experiment.ts      # Code to run the main application as an experiment
+└── README.md          # Project documentation
+```