Skip to content

Commit 4c96f84

Browse files
authored
Addressed PR comments
1 parent 10c23d3 commit 4c96f84

File tree

1 file changed

+45
-16
lines changed

1 file changed

+45
-16
lines changed

articles/hdinsight/spark/zookeeper-troubleshoot-quorum-fails.md

Lines changed: 45 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -49,39 +49,68 @@ Message
4949
* High CPU usage on the zookeeper servers
5050
* In the Ambari UI, if you see near 100% sustained CPU usage on the zookeeper servers, then the zookeeper sessions open during that time can expire and timeout
5151
* Zookeeper clients are reporting frequent timeouts
52-
* The transaction logs and the snapshots are being written to the same disk. This can cause I/O bottlenecks
52+
* In the logs for resource manager, namenode and others, you will see frequent client connection timeouts
53+
* This could result in quorum loss, frequent failovers and other issues
5354

5455
## Check for zookeeper status
5556
* Find the zookeeper servers from the /etc/hosts file or from Ambari UI
5657
* Run the following command
5758
* echo stat | nc {zk_host_ip} 2181 (or 2182)
58-
* Port 2181 is the apache zookeeper instance
59-
* Port 2182 is used by the HDI zookeeper (to provide HA for services that are not natively HA)
60-
59+
* Port 2181 is the apache zookeeper instance
60+
* Port 2182 is used by the HDI zookeeper (to provide HA for services that are not natively HA)
61+
* If the command shows no output, then it means that the zookeeper servers are not running
62+
* If the servers are running, the result will include statics of client connections and other statistics
63+
````
64+
Zookeeper version: 3.4.6-8--1, built on 12/05/2019 12:55 GMT
65+
Clients:
66+
/10.2.0.57:50988[1](queued=0,recved=715,sent=715)
67+
/10.2.0.57:46632[1](queued=0,recved=138340,sent=138347)
68+
/10.2.0.34:14688[1](queued=0,recved=264653,sent=353420)
69+
/10.2.0.52:49680[1](queued=0,recved=134812,sent=134814)
70+
/10.2.0.57:50614[1](queued=0,recved=19812,sent=19812)
71+
/10.2.0.56:35034[1](queued=0,recved=2586,sent=2586)
72+
/10.2.0.52:63982[1](queued=0,recved=72215,sent=72217)
73+
/10.2.0.57:53024[1](queued=0,recved=19805,sent=19805)
74+
/10.2.0.57:45126[1](queued=0,recved=19621,sent=19621)
75+
/10.2.0.56:41270[1](queued=0,recved=1348743,sent=1348788)
76+
/10.2.0.53:59097[1](queued=0,recved=72215,sent=72217)
77+
/10.2.0.56:41088[1](queued=0,recved=788,sent=802)
78+
/10.2.0.34:10246[1](queued=0,recved=19575,sent=19575)
79+
/10.2.0.56:40944[1](queued=0,recved=717,sent=717)
80+
/10.2.0.57:45466[1](queued=0,recved=19861,sent=19861)
81+
/10.2.0.57:59634[0](queued=0,recved=1,sent=0)
82+
/10.2.0.34:14704[1](queued=0,recved=264622,sent=353355)
83+
/10.2.0.57:42244[1](queued=0,recved=49245,sent=49248)
84+
85+
Latency min/avg/max: 0/3/14865
86+
Received: 238606078
87+
Sent: 239139381
88+
Connections: 18
89+
Outstanding: 0
90+
Zxid: 0x1004f99be
91+
Mode: follower
92+
Node count: 133212
93+
````
6194
## CPU load peaks up every hour
6295
* Login to the zookeeper server and check the /etc/crontab
6396
* Are there any hourly jobs running at this time?
6497
* If so, randomize the start time across different zookeeper servers
6598

6699
## Purging old snapshots
67-
* HDI Zookeepers are configured to auto purge old snapshots
100+
* Zookeepers are configured to auto purge old snapshots
68101
* By default, last 30 snapshots are retained
69-
* This controlled by the configuration key autopurge.snapRetainCount
102+
* This is controlled by the configuration key autopurge.snapRetainCount
70103
* /etc/zookeeper/conf/zoo.cfg for hadoop zookeeper
71104
* /etc/hdinsight-zookeeper/conf/zoo.cfg for HDI zookeeper
72-
* Set this to a value =3 and restart the zookeeper servers
73-
* Hadoop zookeeper can be restarted through Ambari
105+
* Set this to a value of 3 and restart the zookeeper servers
106+
* Hadoop zookeeper config can be updated and the service can be restarted through Ambari
74107
* HDI zookeeper has to be stopped manually and restarted manually
75108
* sudo lsof -i :2182 will give you the process id to kill
76109
* sudo python /opt/startup_scripts/startup_hdinsight_zookeeper.py
77-
* Manually purging snapshots
78-
* **DO NOT delete the snapshot files directly as this could result in data loss**
79-
* zookeeper
80-
* sudo java -cp /usr/hdp/current/zookeeper-server/zookeeper.jar:/usr/hdp/current/zookeeper-server/lib/* org.apache.zookeeper.server.PurgeTxnLog /hadoop/zookeeper/ /hadoop/zookeeper/ 3
81-
* hdinsight-zookeeper
82-
* sudo java -cp /usr/hdp/current/zookeeper-server/zookeeper.jar:/usr/hdp/current/zookeeper-server/lib/* org.apache.zookeeper.server.PurgeTxnLog /hadoop/hdinsight-zookeeper/ /hadoop/hdinsight-zookeeper/ 3
83-
84-
## CancelledKeyException in the zookeeper server log
110+
* Manually purging snapshots is not required
111+
* **DO NOT delete the snapshot files directly as this could result in data loss**
112+
113+
## CancelledKeyException in the zookeeper server log doesn't require snapshot cleanup
85114
* This exception usually means that the client is no longer active and the server is unable to send a message
86115
* This is indication of a symptom that the zookeeper client is terminating sessions prematurely
87116
* Look for the other symptoms outlined in this document

0 commit comments

Comments
 (0)