|
| 1 | +--- |
| 2 | +title: Issues with region servers in Azure HDInsight |
| 3 | +description: Issues with region servers in Azure HDInsight |
| 4 | +ms.service: hdinsight |
| 5 | +ms.topic: troubleshooting |
| 6 | +author: hrasheed-msft |
| 7 | +ms.author: hrasheed |
| 8 | +ms.date: 08/07/2019 |
| 9 | +--- |
| 10 | + |
| 11 | +# Issues with region servers in Azure HDInsight |
| 12 | + |
| 13 | +This article describes troubleshooting steps and possible resolutions for issues when interacting with Azure HDInsight clusters. |
| 14 | + |
| 15 | +## Scenario: Unassigned regions |
| 16 | + |
| 17 | +### Issue |
| 18 | + |
| 19 | +When running `hbase hbck` command, you see an error message similar to: |
| 20 | + |
| 21 | +``` |
| 22 | +multiple regions being unassigned or holes in the chain of regions |
| 23 | +``` |
| 24 | + |
| 25 | +From the Apache HBase Master UI, it can be seen that the count of regions being unbalanced across all the region servers. |
| 26 | + |
| 27 | +### Cause |
| 28 | + |
| 29 | +Holes may be the result of offline regions. |
| 30 | + |
| 31 | +### Resolution |
| 32 | + |
| 33 | +Fix the assignments. Follow the steps below to bring the unassigned regions back to normal state: |
| 34 | + |
| 35 | +1. Sign in to HDInsight HBase cluster using SSH. |
| 36 | + |
| 37 | +1. Run `hbase zkcli` command to connect with zookeeper shell. |
| 38 | + |
| 39 | +1. Run `rmr /hbase/regions-in-transition` or `rmr /hbase-unsecure/regions-in-transition` command. |
| 40 | + |
| 41 | +1. Exit zookeeper shell by using `exit` command. |
| 42 | + |
| 43 | +1. Open Ambari UI and restart Active HBase Master service from Ambari. |
| 44 | + |
| 45 | +1. Run `hbase hbck` command again (without any further options). Check the output and ensure that all regions are being assigned. |
| 46 | + |
| 47 | +--- |
| 48 | + |
| 49 | +## Scenario: Dead region servers |
| 50 | + |
| 51 | +### Issue |
| 52 | + |
| 53 | +Region servers fail to start. |
| 54 | + |
| 55 | +### Cause |
| 56 | + |
| 57 | +Multiple splitting WAL directories. |
| 58 | + |
| 59 | +1. Get list of current wals: `hadoop fs -ls -R /hbase/WALs/ > /tmp/wals.out`. |
| 60 | + |
| 61 | +1. Inspect the `wals.out` file. If there are too many splitting directories (starting with *-splitting), the region server is probably failing because of these directories. |
| 62 | + |
| 63 | +### Resolution |
| 64 | + |
| 65 | +1. Stop HBase from Ambari portal. |
| 66 | + |
| 67 | +1. Execute `hadoop fs -ls -R /hbase/WALs/ > /tmp/wals.out` to get fresh list of WALs. |
| 68 | + |
| 69 | +1. Move the *-splitting directories to a temporary folder, `splitWAL`, and delete the *-splitting directories. |
| 70 | + |
| 71 | +1. Execute `hbase zkcli` command to connect with zookeeper shell. |
| 72 | + |
| 73 | +1. Execute `rmr /hbase-unsecure/splitWAL`. |
| 74 | + |
| 75 | +1. Restart HBase service. |
| 76 | + |
| 77 | +## Next steps |
| 78 | + |
| 79 | +If you didn't see your problem or are unable to solve your issue, visit one of the following channels for more support: |
| 80 | + |
| 81 | +* Get answers from Azure experts through [Azure Community Support](https://azure.microsoft.com/support/community/). |
| 82 | + |
| 83 | +* Connect with [@AzureSupport](https://twitter.com/azuresupport) - the official Microsoft Azure account for improving customer experience. Connecting the Azure community to the right resources: answers, support, and experts. |
| 84 | + |
| 85 | +* If you need more help, you can submit a support request from the [Azure portal](https://portal.azure.com/?#blade/Microsoft_Azure_Support/HelpAndSupportBlade/). Select **Support** from the menu bar or open the **Help + support** hub. For more detailed information, review [How to create an Azure support request](https://docs.microsoft.com/azure/azure-supportability/how-to-create-azure-support-request). Access to Subscription Management and billing support is included with your Microsoft Azure subscription, and Technical Support is provided through one of the [Azure Support Plans](https://azure.microsoft.com/support/plans/). |
0 commit comments