This repository contains examples for configuring monitoring of RedPanda using Prometheus and Grafana.
The grafana-dashboards folder contains sample dashboards that can be imported directly into a Grafana or Grafana Cloud instance.
The following dashboards are provided as examples:
- Redpanda Ops Dashboard - Provides an overview of KPIs for a RedPanda cluster with health indicators. This is suitable for ops or SRE to montior on a daily or continuous basis.
- Kafka Topic Metrics - Provides throughput, read/write rates and on-disk sizes of each/all topics.
- Kafka Consumer Offsets - Metrics and KPIs that provide details of topic consumers and how far they are lagging behind the end of the log.
- Redpanda Dashboard - The default dashboard that would be generated by
the
rpk generate grafana-dashboardcommand. Note: This is considered deprecated in favour of the above dashboards. - Kafka Consumer Metrics - Allows for monitoring of Java Kafka consumers, using the Prometheus JMX Exporter and the Kafka Sample Configuration.
The demo folder includes a full dockerized sandbox, that will spin up a three-node RedPanda cluster, an instance of Redpanda Console, a Prometheus instance and a Grafana instance with Prometheus and Grafana configured with the dasboards in the grafana-dashboards folder.
To get started it is as simple as running the docker compose file:
$ cd demo
$ docker-compose up -d
Creating network "demo_default" with the default driver
Creating redpanda-0 ... done
Creating redpanda-1 ... done
Creating redpanda-2 ... done
Creating grafana ... done
Creating console ... done
Creating prometheus ... done
Note: If you wish to include the Owlshop demo, you may include the separate Owlshop docker compose file, replacing the above command as follows:
$ cd demo
$ docker compose -f docker-compose.yml -f docker-compose-owlshop -d
Creating network "demo_default" with the default driver
[+] Running 8/8
⠿ Container redpanda-1 Started 0.5s
⠿ Container redpanda-2 Started 0.5s
⠿ Container grafana Started 0.7s
⠿ Container redpanda-0 Started 0.7s
⠿ Container prometheus Started 0.6s
⠿ Container demo-owl-shop-1 Started 1.1s
⠿ Container demo-console-1 Started 1.2s
⠿ Container connect Started
You can check the status of the RedPanda cluster using the following command:
$ docker exec -it redpanda-0 rpk cluster status
CLUSTER
=======
redpanda.initializing
BROKERS
=======
ID HOST PORT
0* redpanda-0 29092
1 redpanda-1 29093
2 redpanda-2 29094
Once the bootstrap is complete, you should see all three nodes running and the cluster's UUID displayed.
$docker exec -it redpanda-0 rpk cluster status
CLUSTER
=======
redpanda.initializing
BROKERS
=======
ID HOST PORT
0* redpanda-0 29092
1 redpanda-1 29093
2 redpanda-2 29094
You should now be able to open the following URIs in your browser for each service:
- RedPanda Console: http://localhost:8080/
- Prometheus: http://localhost:9090
- Grafana: http://localhost:3000
Once you log into Grafana, click on the Dashboards icon on the left and select Browse. From there, you should be able to see the imported dashboards described above.
To generate a workload in RedPanda we can use the following commands. We have provided one version that uses rpk
installed locally on your machine, and one where you use rpk running inside on of the docker containers.
You can install rpk locally on a Mac using the instructions here.
rpk topic create test_topic -r 3 -p 4 --brokers localhost:9092
docker exec redpanda-0 rpk topic create test_topic -r 3 -p 4
In this section we're going to stream some data from the Wikimedia changelog using Curl.
Set up Producer:
curl --retry 9999 -s -H 'Accept: application/json' https://stream.wikimedia.org/v2/stream/recentchange | rpk --brokers localhost:9092 topic produce test_topic -o ''
Set up one or more consumers (you can run this command in multiple windows, although as the topic we created only has 4 partitions, only the first 4 will consume any data).
docker exec -it redpanda-0 rpk topic consume test_topic --group singleGroup
Set up one or more consumers (you can run this command in multiple windows, although as the topic we created only has 4 partitions, only the first 4 will consume any data).
docker exec -t redpanda-0 rpk topic consume test_topic --group singleGroup
rpk topic consume test_topic --group singleGroup --brokers localhost:9092