Simplify test environment creation and add option to run tests against local chatbot-rag-app

Author: SrdjanLL

example-apps/chatbot-rag-app/test/tracetest/README.md

@@ -5,9 +5,9 @@ We plan to add tests for additional model configurations in the future.*
 
 ## Introduction to Trace Testing
 
-Trace testing is a modern approach to testing distributed systems by leveraging the distributed traces that flow through your applications. In a complex system like the Chatbot RAG application, traditional testing approaches may fall short because they can't effectively monitor the interactions between microservices, databases, and external APIs.
+Trace testing is a modern approach to testing distributed systems by leveraging the distributed traces that flow through your applications.
 
-Tracetest is an open-source tool that enables you to create, run, and maintain integration tests using distributed traces with support for OpenTelemetry and observability backends such as Elastic APM. It allows you to:
+Tracetest is an open-source tool that enables you to create, run, and maintain tests using distributed traces with support for OpenTelemetry and observability backends such as Elastic APM. It allows you to:
 
 - Validate the flow of requests through your entire system
 - Assert on specific spans within a trace
@@ -17,27 +17,18 @@ For more information about Tracetest, visit the [official documentation](https:/
 
 ## Setup
 
-Chatbot RAG application setup uses Docker to create a testing environment that includes:
+Our setup uses Docker to create a testing environment that includes:
 
 1. A Tracetest server for executing and managing tests
-2. An Elasticsearch cluster for storing traces, logs, and application data
-3. An OpenTelemetry collector for processing and routing telemetry data
-4. The chatbot RAG application itself
-
-The setup leverages several Docker Compose files to combine the test environment with the local Elastic Stack (from [docker/docker-compose-elastic.yml](../../../../docker/docker-compose-elastic.yml)) and Chatbot RAG application (from [example-apps/chatbot-rag-app/docker-compose.yml](../../docker-compose.yml)). In order to spin up the up-to-date versions of all moving parts, we leverage overrides maintained within this directory. We use:
-
-- `docker-compose.test.yml` - for Tracetest configuration
-- `docker-compose.test.override.yml` - Test-specific Tracetest configuration
-- `elastic-stack.override.yml` - for test-specific configuration for Elasticsearch and OpenTelemetry Collector
-- `chatbot-rag.override.yml` - for configuration of the chatbot application in test mode.
-
-All services are connected through a shared Docker network to enable communication between components.
+2. An OpenTelemetry collector for processing and routing telemetry data
+3. Tracetest test running that uses Tracetest CLI to communicate with Tracetest server and execute tests based on the specified configuration.
+4. Optionally, you can also setup the Elastic Stack and the chatbot-rag-app itself if you don't have them running in your environment.
 
 ## Environment Configuration
 
-Before running tests, you need to prepare a `.env.test` file with the necessary environment variables. This file configures the behavior of the chatbot application during testing (same configuration as described in [the applications's directory](../../README.md)).
+The test environment expects the Elastic Stack and chatbot-rag-app to run locally. If you already have these services running, you're ready to execute the tests and skip to the next section.
 
-Create a `.env.test` file in the `test/tracetest` directory with the following content to reproduce the environment we're testing with:
+If you are starting without from scratch, you will need to set up the necessary environment variables for the chatbot-rag-app. For this you can follow setup the instructions in [the applications's directory](../../README.md#make-your-env-file) or copy our example to `.env` file in `example-apps/chatbot-rag-app`:
 
 ```bash
 # Location of the application routes
@@ -46,7 +37,7 @@ FLASK_APP=api/app.py
 PYTHONUNBUFFERED=1
 
 # How you connect to Elasticsearch: change details to your instance
-ELASTICSEARCH_URL=http://elasticsearch:9200
+ELASTICSEARCH_URL=http://localhost:9200
 ELASTICSEARCH_USER=elastic
 ELASTICSEARCH_PASSWORD=elastic
 
@@ -66,7 +57,7 @@ OTEL_SDK_DISABLED=false
 OTEL_SERVICE_NAME=chatbot-rag-app
 
 # OpenTelemetry configuration
-OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318
+OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
 OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
 OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true
 
@@ -94,6 +85,12 @@ To run the trace-based tests for the chatbot RAG application, follow these steps
     ./run-tests.sh
    ```
 
+   If you already have Elastic stack and chatbot-rag-app running locally, you use the `--skip-stack-setup` and only spin up test resources.
+
+   ```bash
+    ./run-tests.sh --skip-stack-setup
+   ```
+
    To automatically clean up resources after the tests complete (or if they fail), you can use the --with-cleanup flag:
 
    ```bash
@@ -102,12 +99,13 @@ To run the trace-based tests for the chatbot RAG application, follow these steps
 
 The script performs the following operations:
 
-- Creates a shared Docker network for all services
 - Sets up the Tracetest server
-- Starts the Elastic stack (Elasticsearch and OpenTelemetry Collector)
-- Builds and starts the chatbot RAG application
+- If not using `--skip-stack-setup`:
+  - Starts the Elastic stack (Elasticsearch and Kibana)
+  - Builds and starts the chatbot-rag-app
+- Configures the OpenTelemetry Collector to send data to Tracetest
 - Executes the trace tests defined in `resources/openai-chatbot-test.yaml`
-- If `--with-cleanup` is provided, automatically cleans up all resources when the script exits (normally or due to an error)
+- If `--with-cleanup` is provided, automatically cleans up all resources when the script exits
 
 The example test sends a question about working from home policy to the LLM via API and validates several aspects of the application:
 

example-apps/chatbot-rag-app/test/tracetest/chatbot-rag.override.yml

@@ -21,7 +21,7 @@ services:
         environment:
             TRACETEST_DEV: ${TRACETEST_DEV}
         extra_hosts:
-            - host.docker.internal:host-gateway
+            - localhost:host-gateway
         healthcheck:
             test:
                 - CMD
@@ -37,6 +37,11 @@ services:
               target: 11633
               published: 11633
               protocol: tcp
+            # Exposing the port for the tracetest otlp receiver that won't overlap with otel-collector.
+            - mode: ingress
+              target: 4317
+              published: 11634
+              protocol: tcp
         volumes:
             - type: bind
               source: ./resources/tracetest.yaml

example-apps/chatbot-rag-app/test/tracetest/docker-compose.test.override.yml

@@ -1,24 +1,5 @@
-networks:
-  shared-test-network:
-    external: true # Reference the network created above
-
-services:
-  elasticsearch:
-    networks:
-      - shared-test-network
-  elasticsearch_settings: 
-    networks:
-      - shared-test-network
-  kibana:
-    networks:
-      - shared-test-network
-  otel-collector:
-    networks:
-      - shared-test-network
 configs:
-  # This is the minimal yaml configuration needed to listen on all interfaces
-  # for OTLP logs, metrics and traces, exporting to Elasticsearch.
-  # Overriding for test purposes in order to add tracetest exporter
+  # Overriding the config from docker/docker-compose-elastic.yml for test purposes in order to add tracetest exporter
   edot-collector-config:
     content: |
       receivers:
@@ -52,7 +33,7 @@ configs:
           flush:
             interval: 1s  # improve responsiveness in example apps (default 30s)
         otlp/tracetest:
-          endpoint: http://tracetest:4317
+          endpoint: http://localhost:11634
           tls:
             insecure: true
       
@@ -76,4 +57,9 @@ configs:
           logs:
             receivers: [otlp]
             processors: []
-            exporters: [elasticapm, elasticsearch]
+            exporters: [elasticapm, elasticsearch]
+
+services:
+  otel-collector:
+    extra_hosts:
+      - localhost:host-gateway

example-apps/chatbot-rag-app/test/tracetest/docker-compose.test.yml

@@ -7,7 +7,7 @@ spec:
     type: http
     httpRequest:
       method: POST
-      url: http://api-frontend:4000/api/chat
+      url: http://localhost:4000/api/chat
       body: "{\"question\": \"What is the work from home policy?\"}"
       headers:
       - key: Content-Type

example-apps/chatbot-rag-app/test/tracetest/elastic-stack.override.yml renamed to example-apps/chatbot-rag-app/test/tracetest/otel-collector.override.yml

@@ -2,63 +2,124 @@
 set -e
 
 CLEANUP=false
+SKIP_STACK_SETUP=false
+
+show_help() {
+    cat << EOF
+Usage: ./run-tests.sh [OPTIONS]
+
+Run trace tests for the chatbot RAG application using Tracetest.
+
+Options:
+  --skip-stack-setup   Assume Elastic stack and chatbot RAG are already running
+  --with-cleanup       Clean up resources after test execution
+  --help               Display this help message and exit
+
+Examples:
+  # Set up complete environment and run tests
+  ./run-tests.sh
+
+  # Run tests with existing, local Elastic stack and chatbot RAG
+  ./run-tests.sh --skip-stack-setup
+
+  # Set up everything, run tests, and clean up afterward
+  ./run-tests.sh --with-cleanup
+
+  # Run tests with existing services and clean up test components afterward
+  ./run-tests.sh --skip-stack-setup --with-cleanup
+EOF
+}
+
 for arg in "$@"; do
   case $arg in
     --with-cleanup)
     CLEANUP=true
     shift
     ;;
+    --skip-stack-setup)
+    SKIP_STACK_SETUP=true
+    shift
+    ;;
+    --help)
+    show_help
+    exit 0
+    ;;
   esac
 done
 
-NETWORK_NAME="shared-test-network"
-
-ELASTIC_COMPOSE="../../../../docker/docker-compose-elastic.yml"
-ELASTIC_STACK_OVERRIDES="elastic-stack.override.yml"
 
+ELASTIC_STACK_COMPOSE="../../../../docker/docker-compose-elastic.yml"
+# Docker compose override that configures the otel-collector to send traces to Tracetest
+OTEL_COLLECTOR_OVERRIDE="otel-collector.override.yml"
 TRACETEST_COMPOSE="docker-compose.test.yml"
-TRACETEST_COMPOSE_OVERRIDES="docker-compose.test.override.yml"
-
 CHATBOT_RAG_COMPOSE="../../docker-compose.yml"
-CHATBOT_RAG_COMPOSE_OVERRIDES="chatbot-rag.override.yml"
 
 cleanup() {
     echo "Cleaning up resources..."
-    docker compose -f $CHATBOT_RAG_COMPOSE -f $CHATBOT_RAG_COMPOSE_OVERRIDES down
-    docker compose -f $ELASTIC_COMPOSE -f $ELASTIC_STACK_OVERRIDES down
-    docker compose -f $TRACETEST_COMPOSE -f $TRACETEST_COMPOSE_OVERRIDES down
-    docker network rm $NETWORK_NAME || true
+    
+    if [ "$SKIP_STACK_SETUP" = false ]; then
+        # Clean up everything if not running in local mode
+        echo "Cleaning up chatbot-rag-app..."
+        docker compose -f $CHATBOT_RAG_COMPOSE down
+        echo "Cleaning up Elastic stack..."
+        docker compose -f $ELASTIC_STACK_COMPOSE down
+    else
+        # If running locally restore otel-collector to pre-test state
+        echo "Stopping and removing otel-collector..."
+        docker compose \
+          -f $ELASTIC_STACK_COMPOSE \
+          -f $OTEL_COLLECTOR_OVERRIDE \
+          stop otel-collector
+        docker compose \
+          -f $ELASTIC_STACK_COMPOSE \
+          -f $OTEL_COLLECTOR_OVERRIDE \
+          rm -f otel-collector
+        echo "Restarting otel-collector with original configuration..."
+        docker compose \
+          -f $ELASTIC_STACK_COMPOSE \
+          up -d --wait otel-collector
+    fi
+    
+    # Always clean up Tracetest
+    echo "Cleaning up Tracetest..."
+    docker compose -f $TRACETEST_COMPOSE down
+
+    echo "Cleanup complete."
 }
 
 # Set up cleanup trap if --with-cleanup was provided
 if [ "$CLEANUP" = true ]; then
     trap cleanup EXIT
 fi
 
-# Create the shared network if it doesn't exist
-echo "Creating shared Docker network: $NETWORK_NAME"
-docker network create $NETWORK_NAME || echo "Shared network already exists"
+if [ "$SKIP_STACK_SETUP" = false ]; then
+    echo "Setting up Elastic stack"
+    docker compose \
+      -f $ELASTIC_STACK_COMPOSE \
+      up -d --wait elasticsearch kibana elasticsearch_settings
+fi
+
+echo "Setting up Tracetest server"
+docker compose -f $TRACETEST_COMPOSE up --force-recreate -d --wait
 
-echo "Setting up Tracetest tracetest server"
-docker compose -f $TRACETEST_COMPOSE -f $TRACETEST_COMPOSE_OVERRIDES up --force-recreate -d --wait
 
-echo "Setting up Elastic stack"
+echo "Adding Tracetest exporter to otel-collector"
 docker compose \
-  -f $ELASTIC_COMPOSE \
-  -f $ELASTIC_STACK_OVERRIDES \
-  up -d --wait
+  -f $ELASTIC_STACK_COMPOSE \
+  -f $OTEL_COLLECTOR_OVERRIDE \
+  up -d --wait --force-recreate otel-collector
 
-echo "Building and starting test environment"
-# Rebuilding the chatbot-rag app in order to test locally made changes
-docker compose -f $CHATBOT_RAG_COMPOSE -f $CHATBOT_RAG_COMPOSE_OVERRIDES build api-frontend
-docker compose -f $CHATBOT_RAG_COMPOSE -f $CHATBOT_RAG_COMPOSE_OVERRIDES up -d --wait
+if [ "$SKIP_STACK_SETUP" = false ]; then
+    echo "Building and starting chatbot-rag-app"
+    docker compose -f $CHATBOT_RAG_COMPOSE up --build -d --wait
+fi
 
 echo "Executing trace tests"
 
 docker run --rm \
   -v $(pwd)/resources:/resources \
-  --network=$NETWORK_NAME \
+  --add-host=localhost:host-gateway \
   --entrypoint tracetest \
   kubeshop/tracetest:v1.7.1 \
-  -s http://tracetest:11633 \
+  -s http://localhost:11633 \
   run test --file /resources/openai-chatbot-test.yaml