Skip to content

KC-1029 updating MRC demo with observer promotion #857

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 40 commits into from
Nov 25, 2020
Merged

KC-1029 updating MRC demo with observer promotion #857

Show file tree
Hide file tree
Changes from 23 commits
Commits
Show all changes
40 commits
Select commit Hold shift + click to select a range
05de5b5
REP-1233 fix for slow start up due to cpus restriction that was causi…
thomaskwscott Sep 30, 2020
39888a8
Merge branch '6.0.0-post' of github.com:confluentinc/examples into 6.…
thomaskwscott Nov 2, 2020
52eb629
Demo script for observer promotion
thomaskwscott Nov 2, 2020
fd09332
KC-1027 multiregion observer promotion demo and docs
thomaskwscott Nov 4, 2020
82b0590
Merge remote-tracking branch 'origin/6.0.0-post'
thomaskwscott Nov 4, 2020
0c59f52
KC-1029 removed notes committed by mistake
thomaskwscott Nov 4, 2020
c284f9a
KC-1029 revert env file
thomaskwscott Nov 5, 2020
0632027
KC-1029 fixes from docs build
thomaskwscott Nov 5, 2020
6392d1a
KC-1029 fixes from docs build
thomaskwscott Nov 5, 2020
3da43f8
KC-1029 fixes from docs build
thomaskwscott Nov 5, 2020
c0a13c7
KC-1029 fixes from docs build
thomaskwscott Nov 5, 2020
df0df2f
KC-1029 fixes from docs build
thomaskwscott Nov 5, 2020
812a555
KC-1029 fixes from docs build
thomaskwscott Nov 5, 2020
120ac89
KC-1029 fixes from docs build
thomaskwscott Nov 5, 2020
059c2c5
Update multiregion/docs/multiregion.rst
thomaskwscott Nov 13, 2020
9e24ade
Update multiregion/docs/multiregion.rst
thomaskwscott Nov 13, 2020
b0a0deb
KC-1029 fixes per pr review
thomaskwscott Nov 13, 2020
f16b076
Merge branch 'master' of github.com:thomaskwscott/examples
thomaskwscott Nov 13, 2020
57c957e
Update multiregion/docs/multiregion.rst
thomaskwscott Nov 13, 2020
89a74e4
Update multiregion/docs/multiregion.rst
thomaskwscott Nov 13, 2020
bcc1018
Update multiregion/docs/multiregion.rst
thomaskwscott Nov 13, 2020
6e23948
KC-1029 fixes per pr review
thomaskwscott Nov 19, 2020
d2d4fa2
Merge branch 'master' of github.com:thomaskwscott/examples
thomaskwscott Nov 19, 2020
e40005a
Update multiregion/docs/multiregion.rst
thomaskwscott Nov 19, 2020
f18bb79
Update multiregion/docs/multiregion.rst
thomaskwscott Nov 19, 2020
aabed67
KC-1029 updated replicas image to include observer promotion images
thomaskwscott Nov 24, 2020
c7d9885
Merge branch 'master' of github.com:thomaskwscott/examples
thomaskwscott Nov 24, 2020
8b776e1
Pull out build_docker_images.sh; tweak jmx metrics; docs improvements
ybyzek Nov 24, 2020
b8c1f5b
Remove threads from kafka-consumer-perf-test due to KAFKA-10126
ybyzek Nov 24, 2020
9230261
Add colon in describe topic output
ybyzek Nov 24, 2020
b3b6757
Update JMX docker conatiner check; add output to start script
ybyzek Nov 24, 2020
fe0151b
Reorder topics in jmx_metrics
ybyzek Nov 24, 2020
f012501
Add more JMX metric output to docs
ybyzek Nov 24, 2020
a9ea6b9
Delete unnecessary consumes from run-consumer.sh
ybyzek Nov 24, 2020
fd0a896
Add more JMX metrics to script run
ybyzek Nov 24, 2020
cb08b46
Increase timeout in run-consumer.sh
ybyzek Nov 24, 2020
45cbc7e
Update docs with more JMX metrics
ybyzek Nov 24, 2020
854d52b
Add permanent-failover.sh
ybyzek Nov 24, 2020
df954f8
Promote section headings
ybyzek Nov 24, 2020
74794f8
Merge pull request #1 from confluentinc/PR-857-YB
thomaskwscott Nov 25, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 20 additions & 0 deletions multiregion/config/placement-multi-region-async-op-leader-is-observer.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
{
"version": 2,
"replicas": [
{
"count": 2,
"constraints": {
"rack": "west"
}
}
],
"observers": [
{
"count": 2,
"constraints": {
"rack": "east"
}
}
],
"observerPromotionPolicy":"leader-is-observer"
}
20 changes: 20 additions & 0 deletions multiregion/config/placement-multi-region-async-op-under-min-isr.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
{
"version": 2,
"replicas": [
{
"count": 2,
"constraints": {
"rack": "west"
}
}
],
"observers": [
{
"count": 2,
"constraints": {
"rack": "east"
}
}
],
"observerPromotionPolicy":"under-min-isr"
}
20 changes: 20 additions & 0 deletions multiregion/config/placement-multi-region-async-op-under-replicated.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
{
"version": 2,
"replicas": [
{
"count": 2,
"constraints": {
"rack": "west"
}
}
],
"observers": [
{
"count": 2,
"constraints": {
"rack": "east"
}
}
],
"observerPromotionPolicy":"under-replicated"
}
234 changes: 222 additions & 12 deletions multiregion/docs/multiregion.rst
View file
Open in desktop

Large diffs are not rendered by default.

30 changes: 30 additions & 0 deletions multiregion/scripts/create-topics.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,3 +36,33 @@ docker-compose exec broker-west-1 kafka-topics \
--bootstrap-server broker-west-1:19091 \
--topic multi-region-default \
--config min.insync.replicas=1

echo -e "\n==> Creating topic multi-region-async-op-under-min-isr"

docker-compose exec broker-west-1 kafka-topics \
--create \
--bootstrap-server broker-west-1:19091 \
--topic multi-region-async-op-under-min-isr \
--partitions 1 \
--replica-placement /etc/kafka/demo/placement-multi-region-async-op-under-min-isr.json \
--config min.insync.replicas=2

echo -e "\n==> Creating topic multi-region-async-op-under-replicated"

docker-compose exec broker-west-1 kafka-topics \
--create \
--bootstrap-server broker-west-1:19091 \
--topic multi-region-async-op-under-replicated \
--partitions 1 \
--replica-placement /etc/kafka/demo/placement-multi-region-async-op-under-replicated.json \
--config min.insync.replicas=1

echo -e "\n==> Creating topic multi-region-async-op-leader-is-observer"

docker-compose exec broker-west-1 kafka-topics \
--create \
--bootstrap-server broker-west-1:19091 \
--topic multi-region-async-op-leader-is-observer \
--partitions 1 \
--replica-placement /etc/kafka/demo/placement-multi-region-async-op-leader-is-observer.json \
--config min.insync.replicas=1
15 changes: 15 additions & 0 deletions multiregion/scripts/describe-topics.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,3 +19,18 @@ echo -e "\n==> Describe topic multi-region-default\n"

docker-compose exec broker-east-3 kafka-topics --describe \
--bootstrap-server broker-east-3:19093 --topic multi-region-default

echo -e "\n==> Describe topic multi-region-async-op-under-min-isr\n"

docker-compose exec broker-east-3 kafka-topics --describe \
--bootstrap-server broker-east-3:19093 --topic multi-region-async-op-under-min-isr

echo -e "\n==> Describe topic multi-region-async-op-under-replicated\n"

docker-compose exec broker-east-3 kafka-topics --describe \
--bootstrap-server broker-east-3:19093 --topic multi-region-async-op-under-replicated

echo -e "\n==> Describe topic multi-region-async-op-leader-is-observer\n"

docker-compose exec broker-east-3 kafka-topics --describe \
--bootstrap-server broker-east-3:19093 --topic multi-region-async-op-leader-is-observer
4 changes: 2 additions & 2 deletions multiregion/scripts/jmx_metrics.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
#!/bin/bash

for metric in ReplicasCount InSyncReplicasCount CaughtUpReplicasCount
for metric in ReplicasCount InSyncReplicasCount CaughtUpReplicasCount ObserversInIsrCount
do

echo -e "\n\n==> Monitor $metric \n"

for topic in single-region multi-region-sync multi-region-async multi-region-default
for topic in single-region multi-region-sync multi-region-async multi-region-default multi-region-async-op-under-min-isr multi-region-async-op-under-replicated multi-region-async-op-leader-is-observer
do
BW1=$(docker-compose exec broker-west-1 kafka-run-class kafka.tools.JmxTool --jmx-url service:jmx:rmi:///jndi/rmi://localhost:8091/jmxrmi --object-name kafka.cluster:type=Partition,name=$metric,topic=$topic,partition=0 --one-time true | tail -n 1 | awk -F, '{print $2;}' | head -c 1)
BW2=$(docker-compose exec broker-west-2 kafka-run-class kafka.tools.JmxTool --jmx-url service:jmx:rmi:///jndi/rmi://localhost:8092/jmxrmi --object-name kafka.cluster:type=Partition,name=$metric,topic=$topic,partition=0 --one-time true | tail -n 1 | awk -F, '{print $2;}' | head -c 1)
Expand Down
21 changes: 21 additions & 0 deletions multiregion/scripts/run-consumer.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,3 +16,24 @@ docker-compose exec broker-east-3 kafka-consumer-perf-test --topic multi-region-
--broker-list broker-west-1:19091,broker-east-3:19093 \
--timeout 20000 \
--consumer.config /etc/kafka/demo/consumer-east.config

docker-compose exec broker-east-3 kafka-consumer-perf-test --topic multi-region-async-op-under-min-isr \
--messages 5000 \
--threads 1 \
--broker-list broker-west-1:19091,broker-east-3:19093 \
--timeout 20000 \
--consumer.config /etc/kafka/demo/consumer-east.config

docker-compose exec broker-east-3 kafka-consumer-perf-test --topic multi-region-async-op-under-replicated \
--messages 5000 \
--threads 1 \
--broker-list broker-west-1:19091,broker-east-3:19093 \
--timeout 20000 \
--consumer.config /etc/kafka/demo/consumer-east.config

docker-compose exec broker-east-3 kafka-consumer-perf-test --topic multi-region-async-op-leader-is-observer \
--messages 5000 \
--threads 1 \
--broker-list broker-west-1:19091,broker-east-3:19093 \
--timeout 20000 \
--consumer.config /etc/kafka/demo/consumer-east.config
30 changes: 30 additions & 0 deletions multiregion/scripts/run-producer.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,3 +38,33 @@ docker-compose exec broker-west-1 kafka-producer-perf-test --topic multi-region-
bootstrap.servers=broker-west-1:19091,broker-east-3:19093 \
compression.type=none \
batch.size=8196

docker-compose exec broker-west-1 kafka-producer-perf-test --topic multi-region-async-op-under-min-isr \
--num-records 5000 \
--record-size 5000 \
--throughput -1 \
--producer-props \
acks=all \
bootstrap.servers=broker-west-1:19091,broker-east-3:19093 \
compression.type=none \
batch.size=8196

docker-compose exec broker-west-1 kafka-producer-perf-test --topic multi-region-async-op-under-replicated \
--num-records 5000 \
--record-size 5000 \
--throughput -1 \
--producer-props \
acks=all \
bootstrap.servers=broker-west-1:19091,broker-east-3:19093 \
compression.type=none \
batch.size=8196

docker-compose exec broker-west-1 kafka-producer-perf-test --topic multi-region-async-op-leader-is-observer \
--num-records 5000 \
--record-size 5000 \
--throughput -1 \
--producer-props \
acks=all \
bootstrap.servers=broker-west-1:19091,broker-east-3:19093 \
compression.type=none \
batch.size=8196
13 changes: 12 additions & 1 deletion multiregion/scripts/start.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,8 +53,19 @@ sleep 5

${DIR}/jmx_metrics.sh

echo -e "\nDegrade west region"
docker-compose stop broker-west-1

echo "Sleeping 30 seconds"
sleep 30

${DIR}/describe-topics.sh

echo "Sleeping 30 seconds"
sleep 30

echo -e "\nFail west region"
docker-compose stop broker-west-1 broker-west-2 zookeeper-west
docker-compose stop broker-west-2 zookeeper-west

echo "Sleeping 30 seconds"
sleep 30
Expand Down