KC-1029 updating MRC demo with observer promotion

[thomaskwscott]

Description

What behavior does this PR change, and why?

This PR adds observer promotion to the MRC demo

Author Validation

Describe the validation already done, or needs to be done, by the PR submitter.

• Documentation: https://github.com/confluentinc/docs-platform/pull/127 & Asana ticket filed for Interactive team to update diagram

• multiregion

Reviewer Tasks

Describe the tasks/validation that the PR submitter is requesting to be done by the reviewer.

• Documentation: https://github.com/confluentinc/docs-platform/pull/127 & Asana ticket filed for Interactive team to update diagram

• multiregion

• Depending on time, validate if right branch is master or 6.1.x

[ybyzek]

@thomaskwscott thanks for the PR. I made a first pass through docs changes. I have not run the demo yet -- are nightly Docker images stable for 6.1?

[ybyzek]

@jsancio @skaundinya15 -- do you approve this PR?

@thomaskwscott thanks for iterating on the feedback. There are a few more comments, were you able to address those (if you agree)?

[thomaskwscott]

@jsancio @skaundinya15 -- do you approve this PR?

@thomaskwscott thanks for iterating on the feedback. There are a few more comments, were you able to address those (if you agree)?

@ybyzek I only see 2 outstanding comments that I have already addressed and replied to. Did i miss some?

[ybyzek]

@thomaskwscott Interactive will update the MRC diagram next week, so will leave the PR open till then.

[skaundinya15]

Overall this looks good to me, great work @thomaskwscott! Will let @ybyzek give the final sign off.

[ybyzek]

@thomaskwscott in this PR can you please replace the image at multiregion/docs/images/multi-region-topic-replicas-v2.png with the attached image? Note the slight name change in the legend to use wildcard to describe multiregion async topics (it now includes the new topics created by this PR)

[thomaskwscott]

@thomaskwscott in this PR can you please replace the image at multiregion/docs/images/multi-region-topic-replicas-v2.png with the attached image? Note the slight name change in the legend to use wildcard to describe multiregion async topics (it now includes the new topics created by this PR)

Done!

[ybyzek]

@thomaskwscott running the PR, my environment produces the following WARNING messages when trying to consume. I suspect this is independent of the PR (seems like new CP 6.1 behavior) but would like to determine whether the same messages appear when you run this PR?

==> Consume from east: Multi-region Async Replication reading from Leader in west (topic: multi-region-async) 

WARNING: option [threads] and [num-fetch-threads] have been deprecated and will be ignored by the test
start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec
WARNING: Exiting before consuming the expected number of messages: timeout (20000 ms) exceeded. You can use the --timeout option to increase the timeout.
2020-11-24 18:15:09:135, 2020-11-24 18:15:29:224, 0.0000, 0.0000, 0, 0.0000, 0, 20089, 0.0000, 0.0000


==> Consume from east: Multi-region Async Replication reading from Observer in east (topic: multi-region-async) 

WARNING: option [threads] and [num-fetch-threads] have been deprecated and will be ignored by the test
start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec
2020-11-24 18:15:36:337, 2020-11-24 18:16:12:003, 23.8419, 0.6685, 5000, 140.1895, 1606241753568, -1606241717902, -0.0000, -0.0000
WARNING: option [threads] and [num-fetch-threads] have been deprecated and will be ignored by the test
start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec
2020-11-24 18:16:14:576, 2020-11-24 18:16:19:897, 23.8419, 4.4807, 5000, 939.6730, 1606241778685, -1606241773364, -0.0000, -0.0000
WARNING: option [threads] and [num-fetch-threads] have been deprecated and will be ignored by the test
start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec
2020-11-24 18:16:22:309, 2020-11-24 18:16:27:492, 23.8419, 4.6000, 5000, 964.6923, 1606241786352, -1606241781169, -0.0000, -0.0000
WARNING: option [threads] and [num-fetch-threads] have been deprecated and will be ignored by the test
start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec
2020-11-24 18:16:29:888, 2020-11-24 18:16:35:761, 23.8419, 4.0596, 5000, 851.3537, 1606241794496, -1606241788623, -0.0000, -0.0000

UPDATE: https://issues.apache.org/jira/browse/KAFKA-10126

[skaundinya15]

Just one QQ, otherwise LGTM. Thanks for the work on this @thomaskwscott!

[skaundinya15]

QQ: Should this have a non zero value if we've promoted observers in this example?

[thomaskwscott]

yes, but these metrics are shown before region failure and hence no observers have been promoted.

[ybyzek]

@skaundinya15 I'm reworking the docs with more JMX meric output, so this will become clearer based on when the JMX metrics are captured. Please stay tuned...PR in progress ;)

[skaundinya15]

Makes sense, thanks @thomaskwscott. Looking forward to that @ybyzek 💯

[ybyzek]

@thomaskwscott thomaskwscott#1 is ready for your review & merge.

[ybyzek]

@thomaskwscott LGTM, let's ship it!