Got ERROR when running test for rocketmq #55

rhelbrent · 2022-05-13T07:26:08Z

rhelbrent
May 13, 2022

I'm trying to set up openchaos testing for rocketmq environment, after running command

chaos.sh --driver driver-rocketmq/rocketmq.yaml --install

at OS level, or running command

bin/chaos.sh --driver driver-rocketmq/rocketmq.yaml --install --restart

at container level, always got ERRORs:

"Remote command failed (1): cd /opt/rocketmq-4.9.3;curl https://archive.apache.org/dist/rocketmq/4.9.3/rocketmq-all-4.9.3-bin-release.zip -o rocketmq.zip;unzip rocketmq.zip;rm -f rocketmq.zip;mv rocketmq-all*/* .;rmdir rocketmq-all*"
"Queue model setupCluster fail"

Found detail logs in ~/logs/openchaos.log:

[root@client1 logs]# cat openchaos.log 
2022-05-12 17:10:04.001 [main] INFO - --------------- CHAOS TEST --- DRIVER : RocketMQ---------------
2022-05-12 17:10:04.033 [main] INFO - Initial driver: {
  "name" : "RocketMQ",
  "driverClass" : "io.openchaos.driver.rocketmq.RocketMQDriver",
  "nodes" : [ "rocketmq1", "rocketmq2" ],
  "metaNodes" : [ "rocketmq1", "rocketmq2" ],
  "isOrderTest" : false,
  "endToEndLatencyCheck" : true,
  "pull" : false,
  "metaNodesParticipateInFault" : false,
  "isUploadImage" : false,
  "ossEndPoint" : null,
  "ossAccessKeyId" : null,
  "ossAccessKeySecret" : null,
  "bucketName" : null
}
2022-05-12 17:10:04.048 [main] INFO - Node rocketmq1 download rocketmq for nameserver ...
2022-05-12 17:10:31.312 [main] ERROR - Node rocketmq1 setup nameserver node failed
java.rmi.RemoteException: Remote command failed (1): cd rocketmq-chaos-test;curl https://archive.apache.org/dist/rocketmq/4.9.3/rocketmq-all-4.9.3-bin-release.zip -o rocketmq.zip;unzip rocketmq.zip;rm -f rocketmq.zip;mv rocketmq-all*/* .;rmdir rocketmq-all*; nested exception is: 
	java.rmi.ServerException: 1
	at org.apache.sshd.client.channel.ClientChannel.validateCommandExitStatusCode(ClientChannel.java:142) ~[ChaosControl.jar:?]
	at org.apache.sshd.client.session.ClientSession.executeRemoteCommand(ClientSession.java:287) ~[ChaosControl.jar:?]
	at io.openchaos.common.utils.SshUtil.execCommandInDir(SshUtil.java:93) ~[ChaosControl.jar:?]
	at io.openchaos.driver.rocketmq.RocketMQMetaNode.setup(RocketMQMetaNode.java:53) ~[ChaosControl.jar:?]
	at java.util.HashMap$Values.forEach(HashMap.java:981) [?:1.8.0_202]
	at io.openchaos.model.QueueModel.setupCluster(QueueModel.java:121) [ChaosControl.jar:?]
	at io.openchaos.ChaosControl.ready(ChaosControl.java:292) [ChaosControl.jar:?]
	at io.openchaos.ChaosControl.main(ChaosControl.java:171) [ChaosControl.jar:?]
Caused by: java.rmi.ServerException: 1
	... 8 more
2022-05-12 17:10:31.323 [main] ERROR - Queue model setupCluster fail
java.lang.RuntimeException: java.rmi.RemoteException: Remote command failed (1): cd rocketmq-chaos-test;curl https://archive.apache.org/dist/rocketmq/4.9.3/rocketmq-all-4.9.3-bin-release.zip -o rocketmq.zip;unzip rocketmq.zip;rm -f rocketmq.zip;mv rocketmq-all*/* .;rmdir rocketmq-all*; nested exception is: 
	java.rmi.ServerException: 1
	at io.openchaos.driver.rocketmq.RocketMQMetaNode.setup(RocketMQMetaNode.java:62) ~[ChaosControl.jar:?]
	at java.util.HashMap$Values.forEach(HashMap.java:981) ~[?:1.8.0_202]
	at io.openchaos.model.QueueModel.setupCluster(QueueModel.java:121) [ChaosControl.jar:?]
	at io.openchaos.ChaosControl.ready(ChaosControl.java:292) [ChaosControl.jar:?]
	at io.openchaos.ChaosControl.main(ChaosControl.java:171) [ChaosControl.jar:?]
Caused by: java.rmi.RemoteException: Remote command failed (1): cd rocketmq-chaos-test;curl https://archive.apache.org/dist/rocketmq/4.9.3/rocketmq-all-4.9.3-bin-release.zip -o rocketmq.zip;unzip rocketmq.zip;rm -f rocketmq.zip;mv rocketmq-all*/* .;rmdir rocketmq-all*; nested exception is: 
	java.rmi.ServerException: 1
	at org.apache.sshd.client.channel.ClientChannel.validateCommandExitStatusCode(ClientChannel.java:142) ~[ChaosControl.jar:?]
	at org.apache.sshd.client.session.ClientSession.executeRemoteCommand(ClientSession.java:287) ~[ChaosControl.jar:?]
	at io.openchaos.common.utils.SshUtil.execCommandInDir(SshUtil.java:93) ~[ChaosControl.jar:?]
	at io.openchaos.driver.rocketmq.RocketMQMetaNode.setup(RocketMQMetaNode.java:53) ~[ChaosControl.jar:?]
	... 4 more
Caused by: java.rmi.ServerException: 1
	at org.apache.sshd.client.channel.ClientChannel.validateCommandExitStatusCode(ClientChannel.java:142) ~[ChaosControl.jar:?]
	at org.apache.sshd.client.session.ClientSession.executeRemoteCommand(ClientSession.java:287) ~[ChaosControl.jar:?]
	at io.openchaos.common.utils.SshUtil.execCommandInDir(SshUtil.java:93) ~[ChaosControl.jar:?]
	at io.openchaos.driver.rocketmq.RocketMQMetaNode.setup(RocketMQMetaNode.java:53) ~[ChaosControl.jar:?]
	... 4 more
2022-05-12 17:10:31.323 [main] ERROR - Failed to ready chaos test.
java.lang.RuntimeException: java.lang.RuntimeException: java.rmi.RemoteException: Remote command failed (1): cd rocketmq-chaos-test;curl https://archive.apache.org/dist/rocketmq/4.9.3/rocketmq-all-4.9.3-bin-release.zip -o rocketmq.zip;unzip rocketmq.zip;rm -f rocketmq.zip;mv rocketmq-all*/* .;rmdir rocketmq-all*; nested exception is: 
	java.rmi.ServerException: 1
	at io.openchaos.model.QueueModel.setupCluster(QueueModel.java:164) ~[ChaosControl.jar:?]
	at io.openchaos.ChaosControl.ready(ChaosControl.java:292) [ChaosControl.jar:?]
	at io.openchaos.ChaosControl.main(ChaosControl.java:171) [ChaosControl.jar:?]
Caused by: java.lang.RuntimeException: java.rmi.RemoteException: Remote command failed (1): cd rocketmq-chaos-test;curl https://archive.apache.org/dist/rocketmq/4.9.3/rocketmq-all-4.9.3-bin-release.zip -o rocketmq.zip;unzip rocketmq.zip;rm -f rocketmq.zip;mv rocketmq-all*/* .;rmdir rocketmq-all*; nested exception is: 
	java.rmi.ServerException: 1
	at io.openchaos.driver.rocketmq.RocketMQMetaNode.setup(RocketMQMetaNode.java:62) ~[ChaosControl.jar:?]
	at java.util.HashMap$Values.forEach(HashMap.java:981) ~[?:1.8.0_202]
	at io.openchaos.model.QueueModel.setupCluster(QueueModel.java:121) ~[ChaosControl.jar:?]
	... 2 more
Caused by: java.rmi.RemoteException: Remote command failed (1): cd rocketmq-chaos-test;curl https://archive.apache.org/dist/rocketmq/4.9.3/rocketmq-all-4.9.3-bin-release.zip -o rocketmq.zip;unzip rocketmq.zip;rm -f rocketmq.zip;mv rocketmq-all*/* .;rmdir rocketmq-all*; nested exception is: 
	java.rmi.ServerException: 1
	at org.apache.sshd.client.channel.ClientChannel.validateCommandExitStatusCode(ClientChannel.java:142) ~[ChaosControl.jar:?]
	at org.apache.sshd.client.session.ClientSession.executeRemoteCommand(ClientSession.java:287) ~[ChaosControl.jar:?]
	at io.openchaos.common.utils.SshUtil.execCommandInDir(SshUtil.java:93) ~[ChaosControl.jar:?]
	at io.openchaos.driver.rocketmq.RocketMQMetaNode.setup(RocketMQMetaNode.java:53) ~[ChaosControl.jar:?]
	at java.util.HashMap$Values.forEach(HashMap.java:981) ~[?:1.8.0_202]
	at io.openchaos.model.QueueModel.setupCluster(QueueModel.java:121) ~[ChaosControl.jar:?]
	... 2 more
Caused by: java.rmi.ServerException: 1
	at org.apache.sshd.client.channel.ClientChannel.validateCommandExitStatusCode(ClientChannel.java:142) ~[ChaosControl.jar:?]
	at org.apache.sshd.client.session.ClientSession.executeRemoteCommand(ClientSession.java:287) ~[ChaosControl.jar:?]
	at io.openchaos.common.utils.SshUtil.execCommandInDir(SshUtil.java:93) ~[ChaosControl.jar:?]
	at io.openchaos.driver.rocketmq.RocketMQMetaNode.setup(RocketMQMetaNode.java:53) ~[ChaosControl.jar:?]
	at java.util.HashMap$Values.forEach(HashMap.java:981) ~[?:1.8.0_202]
	at io.openchaos.model.QueueModel.setupCluster(QueueModel.java:121) ~[ChaosControl.jar:?]
	... 2 more
2022-05-12 17:10:31.325 [main] INFO - Teardown client
2022-05-12 17:10:31.328 [main] ERROR - Failed to run chaos test.
java.lang.NullPointerException: null
	at io.openchaos.driver.rocketmq.RocketMQDriver.shutdown(RocketMQDriver.java:185) ~[ChaosControl.jar:?]
	at io.openchaos.model.QueueModel.shutdown(QueueModel.java:223) ~[ChaosControl.jar:?]
	at io.openchaos.ChaosControl.clearAfterException(ChaosControl.java:483) ~[ChaosControl.jar:?]
	at io.openchaos.ChaosControl.ready(ChaosControl.java:371) ~[ChaosControl.jar:?]
	at io.openchaos.ChaosControl.main(ChaosControl.java:171) [ChaosControl.jar:?]
2022-05-12 17:10:31.328 [main] INFO - Teardown client
2022-05-12 17:10:31.328 [main] INFO - Test stop
2022-05-12 17:10:31.331 [main] INFO - Teardown client
2022-05-12 17:11:05.095 [Thread-1] INFO - Teardown client
2022-05-12 17:36:36.766 [main] INFO - --------------- CHAOS TEST --- DRIVER : RocketMQ---------------
2022-05-12 17:36:36.795 [main] INFO - Initial driver: {
  "name" : "RocketMQ",
  "driverClass" : "io.openchaos.driver.rocketmq.RocketMQDriver",
  "nodes" : [ "rocketmq1", "rocketmq2" ],
  "metaNodes" : [ "rocketmq1", "rocketmq2" ],
  "isOrderTest" : false,
  "endToEndLatencyCheck" : true,
  "pull" : false,
  "metaNodesParticipateInFault" : false,
  "isUploadImage" : false,
  "ossEndPoint" : null,
  "ossAccessKeyId" : null,
  "ossAccessKeySecret" : null,
  "bucketName" : null
}
2022-05-12 17:36:36.810 [main] INFO - Node rocketmq1 download rocketmq for nameserver ...
2022-05-12 17:36:43.532 [main] ERROR - Node rocketmq1 setup nameserver node failed
java.rmi.RemoteException: Remote command failed (1): cd /opt/rocketmq-4.9.3;curl https://archive.apache.org/dist/rocketmq/4.9.3/rocketmq-all-4.9.3-bin-release.zip -o rocketmq.zip;unzip rocketmq.zip;rm -f rocketmq.zip;mv rocketmq-all*/* .;rmdir rocketmq-all*; nested exception is: 
	java.rmi.ServerException: 1
	at org.apache.sshd.client.channel.ClientChannel.validateCommandExitStatusCode(ClientChannel.java:142) ~[ChaosControl.jar:?]
	at org.apache.sshd.client.session.ClientSession.executeRemoteCommand(ClientSession.java:287) ~[ChaosControl.jar:?]
	at io.openchaos.common.utils.SshUtil.execCommandInDir(SshUtil.java:93) ~[ChaosControl.jar:?]
	at io.openchaos.driver.rocketmq.RocketMQMetaNode.setup(RocketMQMetaNode.java:53) ~[ChaosControl.jar:?]
	at java.util.HashMap$Values.forEach(HashMap.java:981) [?:1.8.0_202]
	at io.openchaos.model.QueueModel.setupCluster(QueueModel.java:121) [ChaosControl.jar:?]
	at io.openchaos.ChaosControl.ready(ChaosControl.java:292) [ChaosControl.jar:?]
	at io.openchaos.ChaosControl.main(ChaosControl.java:171) [ChaosControl.jar:?]
Caused by: java.rmi.ServerException: 1
	... 8 more
2022-05-12 17:36:43.537 [main] ERROR - Queue model setupCluster fail
java.lang.RuntimeException: java.rmi.RemoteException: Remote command failed (1): cd /opt/rocketmq-4.9.3;curl https://archive.apache.org/dist/rocketmq/4.9.3/rocketmq-all-4.9.3-bin-release.zip -o rocketmq.zip;unzip rocketmq.zip;rm -f rocketmq.zip;mv rocketmq-all*/* .;rmdir rocketmq-all*; nested exception is: 
	java.rmi.ServerException: 1
	at io.openchaos.driver.rocketmq.RocketMQMetaNode.setup(RocketMQMetaNode.java:62) ~[ChaosControl.jar:?]
	at java.util.HashMap$Values.forEach(HashMap.java:981) ~[?:1.8.0_202]
	at io.openchaos.model.QueueModel.setupCluster(QueueModel.java:121) [ChaosControl.jar:?]
	at io.openchaos.ChaosControl.ready(ChaosControl.java:292) [ChaosControl.jar:?]
	at io.openchaos.ChaosControl.main(ChaosControl.java:171) [ChaosControl.jar:?]
Caused by: java.rmi.RemoteException: Remote command failed (1): cd /opt/rocketmq-4.9.3;curl https://archive.apache.org/dist/rocketmq/4.9.3/rocketmq-all-4.9.3-bin-release.zip -o rocketmq.zip;unzip rocketmq.zip;rm -f rocketmq.zip;mv rocketmq-all*/* .;rmdir rocketmq-all*; nested exception is: 
	java.rmi.ServerException: 1
	at org.apache.sshd.client.channel.ClientChannel.validateCommandExitStatusCode(ClientChannel.java:142) ~[ChaosControl.jar:?]
	at org.apache.sshd.client.session.ClientSession.executeRemoteCommand(ClientSession.java:287) ~[ChaosControl.jar:?]
	at io.openchaos.common.utils.SshUtil.execCommandInDir(SshUtil.java:93) ~[ChaosControl.jar:?]
	at io.openchaos.driver.rocketmq.RocketMQMetaNode.setup(RocketMQMetaNode.java:53) ~[ChaosControl.jar:?]
	... 4 more
Caused by: java.rmi.ServerException: 1
	at org.apache.sshd.client.channel.ClientChannel.validateCommandExitStatusCode(ClientChannel.java:142) ~[ChaosControl.jar:?]
	at org.apache.sshd.client.session.ClientSession.executeRemoteCommand(ClientSession.java:287) ~[ChaosControl.jar:?]
	at io.openchaos.common.utils.SshUtil.execCommandInDir(SshUtil.java:93) ~[ChaosControl.jar:?]
	at io.openchaos.driver.rocketmq.RocketMQMetaNode.setup(RocketMQMetaNode.java:53) ~[ChaosControl.jar:?]
	... 4 more
2022-05-12 17:36:43.538 [main] ERROR - Failed to ready chaos test.
java.lang.RuntimeException: java.lang.RuntimeException: java.rmi.RemoteException: Remote command failed (1): cd /opt/rocketmq-4.9.3;curl https://archive.apache.org/dist/rocketmq/4.9.3/rocketmq-all-4.9.3-bin-release.zip -o rocketmq.zip;unzip rocketmq.zip;rm -f rocketmq.zip;mv rocketmq-all*/* .;rmdir rocketmq-all*; nested exception is: 
	java.rmi.ServerException: 1
	at io.openchaos.model.QueueModel.setupCluster(QueueModel.java:164) ~[ChaosControl.jar:?]
	at io.openchaos.ChaosControl.ready(ChaosControl.java:292) [ChaosControl.jar:?]
	at io.openchaos.ChaosControl.main(ChaosControl.java:171) [ChaosControl.jar:?]
Caused by: java.lang.RuntimeException: java.rmi.RemoteException: Remote command failed (1): cd /opt/rocketmq-4.9.3;curl https://archive.apache.org/dist/rocketmq/4.9.3/rocketmq-all-4.9.3-bin-release.zip -o rocketmq.zip;unzip rocketmq.zip;rm -f rocketmq.zip;mv rocketmq-all*/* .;rmdir rocketmq-all*; nested exception is: 
	java.rmi.ServerException: 1
	at io.openchaos.driver.rocketmq.RocketMQMetaNode.setup(RocketMQMetaNode.java:62) ~[ChaosControl.jar:?]
	at java.util.HashMap$Values.forEach(HashMap.java:981) ~[?:1.8.0_202]
	at io.openchaos.model.QueueModel.setupCluster(QueueModel.java:121) ~[ChaosControl.jar:?]
	... 2 more
Caused by: java.rmi.RemoteException: Remote command failed (1): cd /opt/rocketmq-4.9.3;curl https://archive.apache.org/dist/rocketmq/4.9.3/rocketmq-all-4.9.3-bin-release.zip -o rocketmq.zip;unzip rocketmq.zip;rm -f rocketmq.zip;mv rocketmq-all*/* .;rmdir rocketmq-all*; nested exception is: 
	java.rmi.ServerException: 1
	at org.apache.sshd.client.channel.ClientChannel.validateCommandExitStatusCode(ClientChannel.java:142) ~[ChaosControl.jar:?]
	at org.apache.sshd.client.session.ClientSession.executeRemoteCommand(ClientSession.java:287) ~[ChaosControl.jar:?]
	at io.openchaos.common.utils.SshUtil.execCommandInDir(SshUtil.java:93) ~[ChaosControl.jar:?]
	at io.openchaos.driver.rocketmq.RocketMQMetaNode.setup(RocketMQMetaNode.java:53) ~[ChaosControl.jar:?]
	at java.util.HashMap$Values.forEach(HashMap.java:981) ~[?:1.8.0_202]
	at io.openchaos.model.QueueModel.setupCluster(QueueModel.java:121) ~[ChaosControl.jar:?]
	... 2 more
Caused by: java.rmi.ServerException: 1
	at org.apache.sshd.client.channel.ClientChannel.validateCommandExitStatusCode(ClientChannel.java:142) ~[ChaosControl.jar:?]
	at org.apache.sshd.client.session.ClientSession.executeRemoteCommand(ClientSession.java:287) ~[ChaosControl.jar:?]
	at io.openchaos.common.utils.SshUtil.execCommandInDir(SshUtil.java:93) ~[ChaosControl.jar:?]
	at io.openchaos.driver.rocketmq.RocketMQMetaNode.setup(RocketMQMetaNode.java:53) ~[ChaosControl.jar:?]
	at java.util.HashMap$Values.forEach(HashMap.java:981) ~[?:1.8.0_202]
	at io.openchaos.model.QueueModel.setupCluster(QueueModel.java:121) ~[ChaosControl.jar:?]
	... 2 more
2022-05-12 17:36:43.539 [main] INFO - Teardown client
2022-05-12 17:36:43.541 [main] ERROR - Failed to run chaos test.
java.lang.NullPointerException: null
	at io.openchaos.driver.rocketmq.RocketMQDriver.shutdown(RocketMQDriver.java:185) ~[ChaosControl.jar:?]
	at io.openchaos.model.QueueModel.shutdown(QueueModel.java:223) ~[ChaosControl.jar:?]
	at io.openchaos.ChaosControl.clearAfterException(ChaosControl.java:483) ~[ChaosControl.jar:?]
	at io.openchaos.ChaosControl.ready(ChaosControl.java:371) ~[ChaosControl.jar:?]
	at io.openchaos.ChaosControl.main(ChaosControl.java:171) [ChaosControl.jar:?]
2022-05-12 17:36:43.542 [main] INFO - Teardown client
2022-05-12 17:36:43.542 [main] INFO - Test stop
2022-05-12 17:36:43.544 [main] INFO - Teardown client

rhelbrent · 2022-05-13T07:35:07Z

rhelbrent
May 13, 2022
Author

Try to change "installDir: rocketmq-chaos-test" in rocketmq.yaml, same errors reported again.
The authentication from control node to cluster node is correct, I can find directory /root/rocketmq-chaos-test created.

[root@rocketmq1 rocketmq-4.9.3]# pwd
/root/rocketmq-chaos-test/rocketmq-4.9.3
[root@rocketmq1 rocketmq-4.9.3]# ls -l
total 40
drwxr-xr-x 2 root root   126 Feb 22 01:25 benchmark
drwxr-xr-x 3 root root  4096 Feb 22 00:25 bin
drwxr-xr-x 7 root root   201 Feb 22 00:25 conf
drwxr-xr-x 2 root root  4096 Feb 22 01:25 lib
-rw-r--r-- 1 root root 17327 Feb 22 00:25 LICENSE
-rw-r--r-- 1 root root  1338 Feb 22 00:25 NOTICE
-rw-r--r-- 1 root root  6069 Feb 22 00:25 README.md

BTW, I've used "installDir: /opt/rocketmq-4.9.3/" in rocketmq.yaml, which is the RocketMQ home directory in node rocketmq1, after running chaos.sh, it removed all original files and directories then create its own without any operation pre check, I think it is a very significant defects and will cause customer's data loss!

0 replies

rhelbrent · 2022-05-13T07:37:46Z

rhelbrent
May 13, 2022
Author

Content of "rocketmq.yaml"

[root@client1 driver-rocketmq]# cat /opt/openchaos-openchaos-0.6.4/driver-rocketmq/rocketmq.yaml
name: RocketMQ
driverClass: io.openchaos.driver.rocketmq.RocketMQDriver
endToEndLatencyCheck: true

nodes:
- rocketmq1
- rocketmq2

metaNodes:
- rocketmq1
- rocketmq2

rocketmqVersion: 4.9.3
installDir: rocketmq-chaos-test
nameServerPort: 9876

clusterName: DefaultCluster # same value as the config in broker.properties
vipChannelEnabled: false

brokerClusterName: DefaultCluster
brokerName: c1-m1
storePathRootDir: /store/c1-m1
storePathCommitLog: /commitlog/c1-m1
enableDLegerCommitLog: true
dLegerGroup: c1-m1
sendMessageThreadPoolNums: 32

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Got ERROR when running test for rocketmq #55

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Replies: 2 comments

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Select a reply

Uh oh!

Got ERROR when running test for rocketmq #55

Uh oh!

Uh oh!

rhelbrent May 13, 2022

Replies: 2 comments

Uh oh!

Uh oh!

rhelbrent May 13, 2022 Author

Uh oh!

Uh oh!

rhelbrent May 13, 2022 Author

rhelbrent
May 13, 2022

rhelbrent
May 13, 2022
Author

rhelbrent
May 13, 2022
Author