OpenTSDB import fails with PleaseThrottleException

[HariSekhon]

Hi,

When importing data to HBase 1.1.2 on Hortonworks HDP 2.3 I get the following exception as soon as HBase goes to region split:

org.hbase.async.PleaseThrottleException: 10000 RPCs waiting on "MyTable,,<region>." to come back online
...
Caused by: org.hbase.async.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: MyTable,,<region>. is closing
....

I believe the correct behaviour is to catch the PleaseThrottleException + do an exponential backoff.

Is this something that can be added to the OpenTSDB import tool?

Thanks

Hari

[manolama]

Yeah, this something we need to add, re the backoff. It does mean that HBase is just backed up so if you can pre-split your regions that should help importing.

[HariSekhon]

Yes we did start doing pre-splits as a short term workaround as well as for performance, but I think this is a generic enough issue that the correct handling would significantly improve the robustness of the opentsdb loader.