Riak KV Handoff Timeouts caused by Yokozuna [JIRA: RIAK-2732]

[jcapricebasho]

An open source user reported Riak KV Handoffs repeatedly timing out and restarting when the yz_solr:entropy_data/2 case in yz_entropy:get_entropy_data/2 failed to return in the default 60 seconds set via YZ_SOLR_REQUEST_TIMEOUT.

Each time handoffs would start at around 6MB/s, gradually slowing down to about 100KB/s before timing out at around the same place each time. Before handoff times out, the below message is logged:

2016-08-04 18:10:58.057 [error] <0.1278.0>@yz_entropy:get_entropy_data:74 failed to iterate over entropy data due to request exceeding timeout 60000 for filter params [{continuation,none},{limit,100},{partition,19}]

Turning Search off allows handoff to complete successfully.

[mengzyou]

How about this error, we also encounter logs of such errors:

2017-07-24 07:41:14.783 [error] <0.12084.307>@yz_entropy:get_entropy_data:74 failed to iterate over entropy data due to request exceeding timeout 60000 for filter params [{continuation,none},{limit,100},{partition,65}]
2017-07-24 07:42:15.838 [error] <0.12084.307>@yz_entropy:get_entropy_data:74 failed to iterate over entropy data due to request exceeding timeout 60000 for filter params [{continuation,none},{limit,100},{partition,65}]
2017-07-24 08:48:37.996 [error] <0.4253.310>@yz_entropy:get_entropy_data:74 failed to iterate over entropy data due to request exceeding timeout 60000 for filter params [{continuation,none},{limit,100},{partition,49}]
2017-07-24 09:44:57.673 [error] <0.17029.311>@yz_entropy:get_entropy_data:74 failed to iterate over entropy data due to request exceeding timeout 60000 for filter params [{continuation,<<"MyAxOCBZMkZ5WDNKMWJnPT0gWTJGeVgzSjFiakl3TVRjdE1EY3RNRFU9IE1URXpOamRrTjJZdE1qQmhNaTAwWkdRMUxXSm1ObU10WldKaE16UTVZakF5TVRVMiBnMklGTUt0RA">>},{limit,100},{partition,18}]
2017-07-24 09:49:55.593 [error] <0.17029.311>@yz_entropy:get_entropy_data:74 failed to iterate over entropy data due to request exceeding timeout 60000 for filter params [{continuation,<<"MyAxOCBZMmhoY21kcGJtZGZjR2xzWlE9PSBZMmhoY21kcGJtZGZjR2xzWlRJd01UY3RNRGN0TWpRPSBaRFZoTkRNNVpUTXRaamhpTUMwME5UbGpMV0ZsWXpNdE5EWXhPVGd5TVRRMlptWm0gZzJJQU4remY">>},{limit,100},{partition,18}]