Skip to content

README.md

Latest commit

 

History

History
403 lines (268 loc) · 12.7 KB

README.md

File metadata and controls

403 lines (268 loc) · 12.7 KB

Testing the plugin

Prerequisites

  • Node.js and npm installed on your system.
  • Docker installed on your system.
  • Helm installed on your system.
  • A Kubernetes cluster with KEDA installed.
    • ⚠️ Note: If you want to enable support for Prometheus metrics exposed by KEDA, please refer to the optional section below before installing KEDA.
    • If you need to install KEDA directly in your cluster without any additional configurations, please follow the official installation guide.

Optionals (to enable support for Prometheus metrics exposed by KEDA)

  • Prometheus installed in your Kubernetes cluster.
    • If you need to install Prometheus in your cluster, please follow the official installation guide of kube-prometheus-stack Helm Chart on ArtifactHUB.
  • A Kubernetes cluster with KEDA installed via special flags enabling Prometheus metrics export using the below command: 
    •  helm install keda kedacore/keda \
 --namespace keda \
 --create-namespace \
 --set prometheus.operator.enabled=true \
 --set prometheus.operator.serviceMonitor.enabled=true \
 --set prometheus.operator.serviceMonitor.interval="10s" \
 --set prometheus.operator.serviceMonitor.additionalLabels.release="prometheus"

Steps to Test

  1. Clone the plugins repository:

    git clone https://github.com/headlamp-k8s/plugins.git

  2. Switch to the keda branch:

    git checkout keda

  3. Navigate to the keda plugin directory:

    cd keda

  4. Install the required dependencies:

    npm install

  5. Start the plugin in development mode:

    npm run start

  6. Launch Headlamp. You should now see "KEDA" in the sidebar.

Generate KEDA Resources for Testing

1. Build the Message Consumer Image

Before deploying the examples, build the message consumer Docker image using the provided Dockerfile:

# Navigate to the 'test-files' directory
cd test-files

# Build & Push the image to ttl.sh for temporary hosting
# Images on ttl.sh expire after 24 hours by default
# To specify a different expiration time, add a suffix like ":1h" for 1 hour
IMAGE_NAME=$(uuidgen)
docker build -t ttl.sh/${IMAGE_NAME}:1h .
docker push ttl.sh/${IMAGE_NAME}:1h

Take a note of your image name as you'll need to update it in the deployment YAML files.

2. Update the YAML files with your image

Update the image reference in these files:

For example:

Replace

image: some-registry/keda-message-consumer:latest

with

image: ttl.sh/36733c85-1c5e-470b-bb86-f9b93b6d8d82:1h # Replace with your actual image name

3. Deploy Message Brokers

Install RabbitMQ via Helm

Since the Helm stable repository was migrated to the Bitnami Repository, add the Bitnami repo and use it during the installation:

helm repo add bitnami https://charts.bitnami.com/bitnami
Helm 3

RabbitMQ Helm Chart version 7.0.0 or later

helm install rabbitmq --set auth.username=user --set auth.password=PASSWORD bitnami/rabbitmq --wait

Notes:

  • If you are running the RabbitMQ image on KinD, you will run into permission issues unless you set volumePermissions.enabled=true. Use the following command if you are using KinD:

    helm install rabbitmq --set auth.username=user --set auth.password=PASSWORD --set volumePermissions.enabled=true bitnami/rabbitmq --wait

  • With RabbitMQ Helm Chart version 6.x.x or earlier, username and password should be specified with rabbitmq.username and rabbitmq.password parameters https://hub.helm.sh/charts/bitnami/rabbitmq

Helm 2

RabbitMQ Helm Chart version 7.0.0 or later

helm install --name rabbitmq --set auth.username=user --set auth.password=PASSWORD bitnami/rabbitmq --wait

Install Redis via Helm

Install Redis using the Bitnami Redis Helm chart:

helm install redis --set auth.password=PASSWORD bitnami/redis --wait

Notes:

  • If you are running the Redis image on KinD, you may need to set volumePermissions.enabled=true:

    helm install redis --set auth.password=PASSWORD --set volumePermissions.enabled=true bitnami/redis --wait

Wait for Services to Deploy

⚠️ Be sure to wait until both deployments have completed before continuing. ⚠️

kubectl get pods

NAME         READY   STATUS    RESTARTS   AGE
rabbitmq-0   1/1     Running   0          3m3s
redis-master-0   1/1     Running   0          2m45s
redis-replica-0   1/1     Running   0          2m45s
redis-replica-1   1/1     Running   0          2m30s

4. Deploy Authentication Resources

Deploy the authentication resources for both RabbitMQ and Redis:

kubectl apply -f authentication.yaml

This creates:

  • A Secret with connection strings for both RabbitMQ and Redis
  • TriggerAuthentication resources for both message brokers

The authentication resources provide the necessary credentials for KEDA to connect to both RabbitMQ and Redis and monitor their respective queues/lists.

5. Testing KEDA ScaledObject

Deploy a Message Consumer

kubectl apply -f consumer-deployment.yaml

Validate the consumer has been deployed

kubectl get deployments

You should see message-consumer deployment with 1 running pod but after deploying the ScaledObject, it will display 0 running pods as we haven't yet published any messages to RabbitMQ or Redis and hence KEDA will scale down the pods to 0 since we have not mentioned any minReplicaCount in the spec.

NAME               READY   UP-TO-DATE   AVAILABLE   AGE
message-consumer   1/1     1            1           2m32s

This consumer is designed to process messages from both RabbitMQ queues and Redis lists. It consumes one message per instance, sleeps for 1 second to simulate work, and then acknowledges completion of the message.

Deploy ScaledObject

kubectl apply -f scaledObject.yaml

This ScaledObject configures autoscaling for the message-consumer deployment with dual triggers:

  • RabbitMQ Trigger: Monitors the hello queue, scaling based on a target of 5 messages per replica
  • Redis Trigger: Monitors the tasks list, scaling based on a target of 5 messages per replica

The ScaledObject is configured to:

  • Scale to a minimum of 0 replicas when there are no messages in either queue/list
  • Scale up to a maximum of 30 replicas when either queue/list is heavily loaded
  • Use a polling interval of 5 seconds (faster than the default 30 seconds)
  • Apply a cooldown period of 30 seconds before scaling down

These settings can be adjusted in the YAML Specification as needed.

Trigger the Publisher Job to Generate Messages

Now that all components are set up, trigger the publisher job to send messages to both RabbitMQ and Redis:

kubectl apply -f publisher-job.yaml

This job will publish:

  • 300 messages to the RabbitMQ hello queue
  • 500 messages to the Redis tasks list

This will trigger scaling of the consumer deployment via the ScaledObject's dual triggers.

Observe Autoscaling in Action

Monitor the scaling activities:

# Watch the deployments scale up
kubectl get deployments -w

# Check the pods being created
kubectl get pods

# View KEDA ScaledObject status
kubectl get scaledobject

# Check the specific ScaledObject details
kubectl describe scaledobject message-consumer-scaledobject

You should see the number of replicas for the message-consumer deployment increase as messages are added to either the RabbitMQ queue or Redis list, and decrease as messages are processed from both sources.

Monitor Message Processing

You can monitor the message processing by checking the logs:

# View consumer logs
kubectl logs -l app=message-consumer -f

# Check RabbitMQ queue status
kubectl exec rabbitmq-0 -- rabbitmqctl list_queues

# Check Redis list length
kubectl exec redis-master-0 -- redis-cli -a PASSWORD llen tasks

Clean Up Resources

When you're done testing, you can clean up the deployed resources:

kubectl delete -f publisher-job.yaml
kubectl delete -f scaledObject.yaml
kubectl delete -f consumer-deployment.yaml

6. Testing KEDA ScaledJob

Deploy ScaledJob

kubectl apply -f scaledJob.yaml

This ScaledJob creates Kubernetes Jobs to process messages from both RabbitMQ and Redis in batch mode. It features dual triggers similar to the ScaledObject:

  • RabbitMQ Trigger: Monitors the hello queue with a target of 20 messages per job
  • Redis Trigger: Monitors the tasks list with a target of 20 messages per job

The ScaledJob is configured to:

  • Scale to a minimum of 0 jobs when there are no messages in either queue/list
  • Scale up to a maximum of 10 jobs when either queue/list is heavily loaded
  • Process 20 messages per job with a 2-minute timeout
  • Scale from 0 when there are 5+ messages in either queue/list (activation threshold)
  • Use a polling interval of 10 seconds

Unlike the ScaledObject which scales a Deployment, ScaledJob creates separate Job instances that process a batch of messages and then terminate. Each job runs in batch mode with the -batch flag, consuming up to 20 messages before completing.

Trigger the Publisher Job to Generate Messages

Now that all components are set up, trigger the publisher job to send messages to both message brokers:

kubectl apply -f publisher-job.yaml

This job will publish:

  • 300 messages to the RabbitMQ hello queue
  • 500 messages to the Redis tasks list

This will trigger the creation of batch processing jobs via the ScaledJob's dual triggers.

Observe Autoscaling in Action

Monitor the scaling activities:

# View the jobs being created
kubectl get jobs

# Check the pods being created
kubectl get pods

# View KEDA ScaledJob status
kubectl get scaledjob

# Check the specific ScaledJob details
kubectl describe scaledjob message-consumer-scaledjob

You will see jobs created to process batches of messages from both RabbitMQ and Redis based on the ScaledJob configuration.

Monitor Batch Processing

Monitor the batch processing:

# View job logs to see batch processing
kubectl logs -l job-name=message-consumer-scaledjob-xxxxx -f

# Check message broker status
kubectl exec rabbitmq-0 -- rabbitmqctl list_queues
kubectl exec redis-master-0 -- redis-cli -a PASSWORD llen tasks

Clean Up

When you're done testing, you can clean up the deployed resources:

kubectl delete -f publisher-job.yaml
kubectl delete -f scaledJob.yaml

7. Clean Up All The Resources You Deployed

When you're done testing, you can clean up all the deployed resources:

# Clean up KEDA resources
kubectl delete -f publisher-job.yaml # ignore if you've already done this
kubectl delete -f scaledJob.yaml # ignore if you've already done this
kubectl delete -f scaledObject.yaml # ignore if you've already done this
kubectl delete -f consumer-deployment.yaml # ignore if you've already done this
kubectl delete -f authentication.yaml

# Uninstall message brokers
helm uninstall rabbitmq
helm uninstall redis

Architecture Overview

This testing setup demonstrates KEDA's ability to handle multiple message brokers simultaneously:

Message Flow

  1. Publisher Job sends messages to both RabbitMQ (queue) and Redis (list)
  2. KEDA Scalers monitor both message brokers independently
  3. Consumer Applications process messages from both sources
  4. Scaling decisions are made based on the combined load from both brokers

Scaling Behavior

  • ScaledObject: Maintains long-running pods that continuously process messages
  • ScaledJob: Creates short-lived jobs that process batches of messages and terminate
  • Dual Triggers: Both configurations monitor RabbitMQ and Redis simultaneously
  • Independent Scaling: Each trigger can independently cause scaling events

This setup showcases KEDA's flexibility in handling complex, multi-broker scenarios commonly found in production environments.