Riak_kv K8s memory consumption

[azowarg]

Hello, I would like to ask for some advice. We have a 5-node riak kv nodes in ring installed in a kubernetes cluster. About 500 GB of data is stored on each node, but there are also butch of corrupted keys. We use "leveldb" as the backend and have enabled tictacaae.

storage_backend = leveldb
anti_entropy = passive
tictacaae_active = active

1. The first problem is related to memory consumption in the K8s environment. In our case with corrupted keys, we have the opinion that riak make is trying to restore them, but it uses all the requested memory and goes beyond the pod memory limits. With this behavior regarding memory consumption, the pod receives an OOM Killer signal. Is it possible to set memory limits for the riak application? We tried several workarounds related to the leveldb.maximum_memory.percentage or level db.maximum_memory settings, but without any result.
2. Is there a scenario for some kind of "forced" key recovery? How do I track the progress of the "recovery"? Looks like riak doing something but we don't have control on it and can't make any influence.

Thanks in advance, I am ready to provide any additional information.

[martinsumner]

What version of Riak? How many keys overall? What is the average size of the keys (including bucket and type names)? What is the per-pod memory limit.

Enabling the leveldb limits doesn't alter the leveled backend demand for memory, and enabling tictacaae will enable the leveled backend. There has been a lot of work on memory efficiency and AAE efficiency recently, but much of that has yet to make it through to the basho repos.

There are some specific operations that can be memory intensive (like AAE store rebuilds). Is there a pattern of the memory suddenly spiking?

If memory is very tight, you probably have more control over memory if you use the legacy anti_entropy rather than tictacaae. The tictacaae is designed to work in native mode (when the storage_backend is leveled), running in parallel mode (i.e. with another backend) like this, has had far less non-functional testing.

In terms of recovery, AAE is used to discover differences, and then those differences are resolved by AAE prompting read repair. Read repair will read the values for a suspect key from all N vnodes that hold them, and over-write any out-of-date values (using the vector clocks to determine what is out of date).

The process of AAE discovering the differences are:

• build a key store of keys/clocks (using the leveled backend) for each vnode.
• build a cached (merkle-style) hash tree of the key store of keys/clocks for each partition supported by the vnode.
• exchange the partition-specific tree with another vnode which also holds such a partition-specific tree.
• discover any broken "segments" in the tree - portions of the hash-space where there are deltas.
• compare the keys/clocks in those segments by querying the key-store to find keys within the segment.
• prompt read-repair for any key where the clocks are different.

There are some logs in the riak_kv_vnode module that summarise what happens in each cycle, as well as some stats in riak stats. If you're getting memory spikes you might want to look for logs about rebuilds starting/completing and see if they correlate.

[azowarg]

Thank you for the quick response!
Version of riak is 3.0.12. More then 100kk keys in all and some buckets holds more then 10kk keys inside.
At the first boot, no more than 5-6 gb of memory are consumed by one pod. As soon as requests to the node begin, memory consumption starts to gradually increase (we assume that the corrupted keys are accessed and the internal read_repair procedure starts. We also launched a separate copy of our cluster with data backup without load and requests from our services. A similar cluster behavior is starts if, for example, you run the procedure manually riak_client:aae_fold({repair_key_range, <<"any_bucket">>, all, all, all}).). For a while, we used the 20gb limits on the pod and wait for ending of internal read_repair processes . Over the past couple of weeks, we have moved the pods to 30Gb limits, so the idea was to allocate more to check the behavior of the cluster. But over time, memory consumption comes to pods limits and it does not decrease over time.
Here is riak admin top output form node with 30gb memory consumtion.

========================================================================================
 'riak@riak-4.riak-headless'                                               11:53:25
 Load:  cpu         0               Memory:  total     7972862    binary    3530027
        procs   17207                        processes 4339132    code        18238
        runq        0                        atom         5685    ets         38894

Pid            Name or Initial Func    Time    Reds  Memory    MsgQ Current Function
----------------------------------------------------------------------------------------
<8769.17478.0> aae_runner:init/1        '-'******** 1574136       0 gen:do_call/4
<8769.16084.0> aae_runner:init/1        '-'********  973164       0 gen_server:loop/7
<8769.16962.0> aae_runner:init/1        '-'********  601780       0 gen_server:loop/7
<8769.17360.0> aae_runner:init/1        '-'********  372252       0 gen_server:loop/7
<8769.770.0>   riak_core_vnode_work     '-'******** 2546416       0 gen_server:loop/7
<8769.16319.0> riak_core_vnode_work     '-'********   13800       0 gen_server:loop/7
<8769.17365.0> riak_core_vnode_work     '-'********    8864       0 gen_server:loop/7
<8769.16980.0> riak_core_vnode_work     '-'********    8864       0 gen_server:loop/7
<8769.16307.0> riak_core_vnode_work     '-'********    8864       0 gen_server:loop/7
<8769.17332.0> riak_core_vnode_work     '-'********   13800       0 gen_server:loop/7
========================================================================================

========================================================================================
 'riak@riak-4.riak-headless'                                               11:53:31
 Load:  cpu         1               Memory:  total     7965241    binary    3517755
        procs   17206                        processes 4344504    code        18238
        runq        0                        atom         5685    ets         38937

Pid            Name or Initial Func    Time    Reds  Memory    MsgQ Current Function
----------------------------------------------------------------------------------------
<8769.17478.0> aae_runner:init/1        '-'19648973  973224       0 gen:do_call/4
<8769.15481.0> leveled_sst:init/1       '-'  432374  372404       0 gen_fsm:loop/8
<8769.15280.0> leveled_sst:init/1       '-'  427503  372404       0 gen_fsm:loop/8
<8769.3200.0>  leveled_sst:init/1       '-'  424963  372404       0 gen_fsm:loop/8
<8769.3224.0>  leveled_sst:init/1       '-'  424282  372404       0 gen_fsm:loop/8
<8769.3189.0>  leveled_sst:init/1       '-'  423038  372404       0 gen_fsm:loop/8
<8769.3267.0>  leveled_sst:init/1       '-'  422548  372404       0 gen_fsm:loop/8
<8769.15246.0> leveled_sst:init/1       '-'  422333  372404       0 gen_fsm:loop/8
<8769.15491.0> leveled_sst:init/1       '-'  422023  372404       0 gen_fsm:loop/8
<8769.15740.0> leveled_sst:init/1       '-'  421316  230548       0 gen_fsm:loop/8
========================================================================================

All context about behavior and seeking of corrupted keys can be find in our previous issue from my collegue. So we tried to "force" the recovery of these keys manually making requests for them or using aae_fold queries like riak_client:aae_fold({object_stats, <<"any_bucket">>, all, {date, {{2023, 8, 5}, {12, 0, 0}}, {{2023, 8, 5}, {13, 0, 0}}}}).
But it is impossible not to predict the recovery. For example, riak may give you this key in a couple of days by curl request, or maybe not. Also additional entries from logs appears sometimes.

Inbound clock entry for <<212,115,185,63,164,208,25,139>> in <<"example_bucket_1">>/<<"example_key_1">> greater than local.Epochs: {In:3543632 Local:0}. Counters: {In:41 Local:0}.
Inbound clock entry for <<212,115,185,63,164,208,25,139>> in <<"example_bucket_1">>/<<"example_key_1">> greater than local.Epochs: {In:0 Local:0}. Counters: {In:11 Local:0}.
Inbound clock entry for <<212,115,185,63,164,208,25,139>> in <<"example_bucket_1">>/<<"example_key_1">> greater than local.Epochs: {In:0 Local:0}. Counters: {In:1 Local:0}.
Inbound clock entry for <<124,245,8,174,118,96,56,16>> in <<"example_bucket_2">>/<<"example_key_2">> greater than local.Epochs: {In:7965961 Local:0}. Counters: {In:1 Local:0}.
Inbound clock entry for <<124,245,8,174,118,96,56,16>> in <<"example_bucket_2">>/<<"example_key_2">> greater than local.Epochs: {In:7895995 Local:0}. Counters: {In:1 Local:0}.
Inbound clock entry for <<124,245,8,174,118,96,56,16>> in <<"example_bucket_2">>/<<"example_key_2">> greater than local.Epochs: {In:7917020 Local:0}. Counters: {In:1 Local:0}.

[martinsumner]

The underlying issue of corrupting objects is fixed in this PR #1882. This is in the nhs-riak 3.2.2 release, but hasn't been accepted into the basho repo yet.

The key amnesia logs I assume are from 3 different vnodes. The Inbound counter on one of them is showing nearly 8m updates for that vnode alone. Is that expected through normal activity? I assume not, and in fact these keys are stuck in some continuous AAE exchange/repair key churn. Do you think this is the case? What is the output of this log - https://github.com/basho/riak_kv/blob/develop-3.0/src/riak_kv_vnode.erl#L1215-L1226

If you are in some continuous exchange churn you might be able to improve your symptoms by increasing your exchange tick to reduce the frequency of activity - https://github.com/basho/riak_kv/blob/develop-3.0/priv/riak_kv.schema#L150-L162.

There are a bunch of improvements to exchange efficiency that are in the latest nhs-riak 3.2.2 release that would help too. However, I have no way of getting these upstream into the basho repo at present. You may wish to consider the relative risks of building your own release from nhs-riak if you need these changes.

[azowarg]

The output for this log. exchanges_expected and exchanges_completed are always 6 for different partitions and in output for different nodes in general.

Tictac AAE loop completed for partition=1438665674247607560106752257205091097473808596992 with exchanges expected=6 exchanges completed=6 total deltas=790 total exchange_time=886 seconds loop duration=3360 seconds (elapsed)
Tictac AAE loop completed for partition=0 with exchanges expected=6 exchanges completed=6 total deltas=501 total exchange_time=341 seconds loop duration=3360 seconds (elapsed)
Tictac AAE loop completed for partition=68507889249886074290797726533575766546371837952 with exchanges expected=6 exchanges completed=6 total deltas=0 total exchange_time=5 seconds loop duration=3360 seconds (elapsed)
Tictac AAE loop completed for partition=137015778499772148581595453067151533092743675904 with exchanges expected=6 exchanges completed=7 total deltas=2934 total exchange_time=1949 seconds loop duration=3360 seconds (elapsed)
Tictac AAE loop completed for partition=159851741583067506678528028578343455274867621888 with exchanges expected=6 exchanges completed=6 total deltas=627 total exchange_time=779 seconds loop duration=3360 seconds (elapsed)
Tictac AAE loop completed for partition=1118962191081472546749696200048404186924073353216 with exchanges expected=6 exchanges completed=6 total deltas=0 total exchange_time=5 seconds loop duration=3360 seconds (elapsed)
Tictac AAE loop completed for partition=456719261665907161938651510223838443642478919680 with exchanges expected=6 exchanges completed=6 total deltas=707 total exchange_time=209 seconds loop duration=3360 seconds (elapsed)
Tictac AAE loop completed for partition=1187470080331358621040493926581979953470445191168 with exchanges expected=6 exchanges completed=6 total deltas=2133 total exchange_time=1344 seconds loop duration=3360 seconds (elapsed)

[martinsumner]

AAE is working very hard. You may be able to ease your symptoms by increasing the exchangetick.

i.e. from riak remote_console (on each node):

application:set_env(riak_kv, tictacaae_exchangetick, 1440000).

The change can be made permanently by changing the setting in riak.conf - https://github.com/nhs-riak/riak_kv/blob/nhse-develop/priv/riak_kv.schema#L150-L162.

The long-term answer is to upgrade, to a version with the necessary fixes and performance improvements (i.e. Riak 3.2.2 with OTP22), but that would require you to build your own packages from the nhs-riak repo, given the current delays bringing the basho repo up-to-date.

[azowarg]

Thanks for the advice, we are going to try to build a new image with the new version.

Just for clarification, I found min 8 vnodes in the cluster that output entries with Inbound clock entry for.
Someone sends a log more often, someone less.
So maybe there are any tools to work and help with our case exactly in these vnodes?

And could you please explain in more detail what exactly parameters mean in the output of this log.
Inbound clock entry for << node_id >> in <<"example_bucket_1">>/<<"example_key_1">> greater than local.Epochs: {In:3543632 Local:0}. Counters: {In:41 Local:0}.
Especially how "In" and "local" values are calculated and their meaning.

[martinsumner]

There's an explanation of incrementing key epochs here. The linked issue (KV679) and related PRs contain more information.

"Local" is referring to the vector clock of the object being replaced in the vnode, "In" is the new PUT arriving at the vnode.

If you look at the log_key_amnesia function in the riak_kv_vnode, you can see it is called in two cases. If the object is not_found, then the Local values are assumed to be 0/0. So in this case, a PUT arrived - that according to the vector clock, this vnode has coordinated 41 changes to this object ... but instead found it wasn't present.

The KV679 issue that this is there to solve is complicated. It covers various corner cases, which include data being deleted due to backend corruption/failure. But also the affect of deleting keys (when delete_mode =/= keep), and then writing new values to the object, only for "older" tombstones to be resurrected (known as the dombstone scenario).

If you've highlighted 8 vnodes, it might be worth looking at their relative location in the ring i.e. which preflists they share. If they don't share preflists, you might have potential solutions involving partition repair. If they share preflists, then there might be some further clues as to what is happening.

[azowarg]

After a couple of weeks of testing, we tried several scenarios.
First, we decided to trigger data migration/transferring by adding a node to our test cluster. After it became a cluster member we started to exclude the remaining nodes from the cluster one by one. I mean, we have taken a node and left it from the cluster, and after successfully ring rebuild, we have joined this node to the cluster again. The first node has finished the leave/join process in 4 days and remains approximately in 24-36 hours. As a result, we found that log_key_amnesia outputs disappeared and don't appear anymore. Unfortunately, this doesn't make any changes to our "keys" case and resource consumption. Results, for example, for aae_fold erase_keys count shows any number of keys (this cluster doesn't connect to our services and write any inbound data)

(riak@riak-1.riak-headless)2> riak_client:aae_fold({erase_keys,<<"bucket">>,all,all,all,count}).
{ok,122292978}
(riak@riak-1.riak-headless)3> riak_client:aae_fold({erase_keys,<<"bucket">>,all,all,all,count}).
{ok,118788850}
(riak@riak-1.riak-headless)4> riak_client:aae_fold({erase_keys,<<"bucket">>,all,all,all,count}).
{ok,118154097}
(riak@riak-1.riak-headless)5> riak_client:aae_fold({erase_keys,<<"bucket">>,all,all,all,count}).
{ok,118154221}
(riak@riak-1.riak-headless)6> riak_client:aae_fold({erase_keys,<<"bucket">>,all,all,all,count}).
{ok,117811339}
(riak@riak-1.riak-headless)7> riak_client:aae_fold({erase_keys,<<"bucket">>,all,all,all,count}).
{ok,122261776}

We have changed the version of riak as you noticed to the 3.2.2 version from your NHS repo. It works but we don't see any visible effect on our data corruption case, but noticed that CPU consumption became less.
Then we have tried to rebuild tictac_aae trees. We stopped tictac on every node, restarted, deleted tictac_aae directory, and enabled tictac again.
The replica cluster at the beginning has this directory space allocation for each of 5 nodes.

0
220G	./tictac_aae
256G	./leveldb
1
272G	./leveldb
249G	./tictac_aae
2
264G	./leveldb
174G	./tictac_aae
3
199G	./tictac_aae
273G	./leveldb
4
170G	./tictac_aae
251G	./leveldb

Replica cluster for test after adding and leave/join procedure:

0
132G	./tictac_aae
202G	./leveldb
1
202G	./leveldb
142G	./tictac_aae
2
219G	./leveldb
158G	./tictac_aae
3
136G	./tictac_aae
100G	./leveldb
4
155G	./tictac_aae
112G	./leveldb
5
223G	./leveldb
58G	        ./tictac_aae

After deleting the directory and rebuilding:

0
141G	./tictac_aae
202G	./leveldb
1
202G	./leveldb
142G	./tictac_aae
2
219G	./leveldb
150G	./tictac_aae
3
136G	./tictac_aae
201G	./leveldb
4
155G	./tictac_aae
222G	./leveldb
5
223G	./leveldb
155G	./tictac_aae

Is it normal that our tictac_aae directories have, in my opinion, so big size in comparison with leveldb directories with data?
We don't enable storeheads.

      {tictacaae_primaryonly,true},
      {tictacaae_rangeboost,2},
      {tictacaae_repairloops,4},
      {tictacaae_maxresults,64},
      {tictacaae_rebuild_blocktime,1000},
      {tictacaae_rebuildtick,3600000},
      {tictacaae_exchangetick,480000}, <-- # tried to increase as you mentioned, seems it has little effect on memory consumption.
      {tictacaae_storeheads,false},
      {tictacaae_rebuilddelay,345600},
      {tictacaae_rebuildwait,336},
      {tictacaae_parallelstore,leveled_ko},

This still does not affect keys but feels like memory consumption decreased. And Tictac AAE loop completed logs look as before:

Jul 4, 2024 @ 14:11:37.079 | Tictac AAE loop completed for partition=342539446249430371453988632667878832731859189760 with exchanges expected=6 exchanges completed=6 total deltas=0 total exchange_time=6 seconds loop duration=3360 seconds (elapsed)
Jul 4, 2024 @ 14:11:26.912 | Tictac AAE loop completed for partition=570899077082383952423314387779798054553098649600 with exchanges expected=6 exchanges completed=6 total deltas=0 total exchange_time=6 seconds loop duration=3360 seconds (elapsed)
Jul 4, 2024 @ 14:10:56.878 | Tictac AAE loop completed for partition=844930634081928249586505293914101120738586001408 with exchanges expected=6 exchanges completed=6 total deltas=0 total exchange_time=5 seconds loop duration=3360 seconds (elapsed)
Jul 4, 2024 @ 14:10:50.598 | Tictac AAE loop completed for partition=1347321821914426127719021955160323408745312813056 with exchanges expected=6 exchanges completed=6 total deltas=0 total exchange_time=5 seconds loop duration=3360 seconds (elapsed)
Jul 4, 2024 @ 14:10:27.824 | Tictac AAE loop completed for partition=1187470080331358621040493926581979953470445191168 with exchanges expected=6 exchanges completed=6 total deltas=4013 total exchange_time=472 seconds loop duration=3360 seconds (elapsed)
Jul 4, 2024 @ 14:02:26.658 | Tictac AAE loop completed for partition=137015778499772148581595453067151533092743675904 with exchanges expected=6 exchanges completed=6 total deltas=2478 total exchange_time=615 seconds loop duration=3360 seconds (elapsed)

[martinsumner]

I would expect tictac_aae folders to be much smaller than leveldb folders, unless your object sizes are very small (i.e. much less than 1KB). If you're not using store_heads - it is just keys and clocks I think (and a very small amount of additional metadata).

[martinsumner]

I don't understand what is happening. There is an environment variable in riak_kv of log_readrepair - you can set that to true, and it will log out the actual keys that it is repairing.

You may then be able to get some clues - are they all from one bucket, are they from a specific set of keys that have different activity patterns etc. It also logs the details of the vector clocks that differ, which may offer some further clues.

[azowarg]

Does every log appearance for each unique key mean that this key is corrupted and will be passed to the recovering process?
I have received 15k output examples in at least 15 minutes for different buckets. Does "none" appearance in "Clocks" field normal behavior?

  | Jul 4, 2024 @ 17:41:14.342 | Prompted read repair Bucket=<<"bucket_1">> Key=<<"key_1">> Clocks none [{<<124,245,8,174,26,199,201,241,0,160,152,128>>,{1,63868219612}}]
  | Jul 4, 2024 @ 17:41:14.342 | Prompted read repair Bucket=<<"bucket_1">> Key=<<"key_2">> Clocks none [{<<124,245,8,174,26,199,201,241,0,160,223,7>>,{1,63868261878}}]
  | Jul 4, 2024 @ 17:41:14.342 | Prompted read repair Bucket=<<"bucket_1">> Key=<<"key_3">> Clocks none [{<<109,136,98,215,109,134,234,195,0,122,222,116>>,{1,63868281693}}]
  | Jul 4, 2024 @ 17:41:14.342 | Prompted read repair Bucket=<<"bucket_1">> Key=<<"key_4">> Clocks none [{<<109,136,98,215,109,134,234,195,0,124,133,202>>,{1,63868471749}}]
  | Jul 4, 2024 @ 17:41:14.342 | Prompted read repair Bucket=<<"bucket_1">> Key=<<"key_5">> Clocks none [{<<109,136,98,215,109,134,234,195,0,121,220,87>>,{1,63868114394}}]
  | Jul 4, 2024 @ 17:41:14.342 | Prompted read repair Bucket=<<"bucket_1">> Key=<<"key_6">> Clocks none [{<<197,14,211,70,72,91,101,25,0,131,235,82>>,{1,63868230684}}]
  | Jul 4, 2024 @ 17:41:14.342 | Prompted read repair Bucket=<<"bucket_1">> Key=<<"key_7">> Clocks none [{<<109,136,98,215,109,134,234,195,0,123,243,186>>,{1,63868400663}}]
  | Jul 4, 2024 @ 17:41:14.341 | Prompted read repair Bucket=<<"bucket_1">> Key=<<"key_8">> Clocks none [{<<124,245,8,174,26,199,201,241,0,154,156,182>>,{1,63867293695}}]
  | Jul 4, 2024 @ 17:41:14.341 | Prompted read repair Bucket=<<"bucket_1">> Key=<<"key_9">> Clocks none [{<<109,136,98,215,109,134,234,195,0,119,187,209>>,{1,63867708430}}]

[martinsumner]

A clock of none means the key doesn't exist in one vnode. There's only one vnode in the other clock, and this has a counter of 1 - so this is a fresh write.

However, the timestamp in the clock is not as expected:

calendar:gregorian_seconds_to_datetime(63868219612).
{{2023,11,26},{12,6,52}}

So this is a key that has a single update (the initial write), made on 26th November 2023 at 12:06:52 - that somehow isn't in the other vnode.

Might the key have been deleted recently? Do you expect the key to exist? Did something odd happen late November (most of these timestamps are from similar date/times).

[martinsumner]

Sorry, I ignored your question about the recovery process.

The recovery process is read repair, and this is logged at the start of that recovery process - so the AAE has discovered this difference between the clocks, and it will now prompt a GET of an object (internally, but using the same GET_FSM as an external fetch).

It is expected that GET_FSM will discover the same difference, and and the end of that process it will prompt any vnode that doesn't have the object (i.e. the one with clock of none) to take the retrieved the object.

there are some stats in riak admin status of read_repairs/skipped_read_repairs these should start incrementing as the repair process resolves.

Obviously if something is broken with the repair process related to the state of these objects, then if you wait you will see a repeating pattern of the same keys being passed to repair.

[azowarg]

Might the key have been deleted recently? Do you expect the key to exist? Did something odd happen late November (most of these timestamps are from similar date/times).

We never delete keys so it should be present in the base. I don't know if specific dates are a clue in case I found a lot of different timestamps.
Just for example:

(riak@riak-0.riak-headless)5> calendar:gregorian_seconds_to_datetime(63853864562).
{{2023,6,13},{8,36,2}}
(riak@riak-0.riak-headless)6> calendar:gregorian_seconds_to_datetime(63865469412).
{{2023,10,25},{16,10,12}}
(riak@riak-0.riak-headless)7> calendar:gregorian_seconds_to_datetime(63868333718).
{{2023,11,27},{19,48,38}}
(riak@riak-0.riak-headless)8> calendar:gregorian_seconds_to_datetime(63864702431).
{{2023,10,16},{19,7,11}}
(riak@riak-0.riak-headless)9> calendar:gregorian_seconds_to_datetime(63858361067).
{{2023,8,4},{9,37,47}}
(riak@riak-0.riak-headless)10> calendar:gregorian_seconds_to_datetime(63852677904).
{{2023,5,30},{14,58,24}}
(riak@riak-0.riak-headless)11> calendar:gregorian_seconds_to_datetime(63846658343).
{{2023,3,21},{22,52,23}}
(riak@riak-0.riak-headless)12> calendar:gregorian_seconds_to_datetime(63848078575).
{{2023,4,7},{9,22,55}}

Stats from riak admin status | grep read_repair seems to look normal. Counters are increasing slowly after some periods but for some reason *_outofdate_* counters have "undefined" values. Is it ok?

read_repairs : 0
read_repairs_counter : 0
read_repairs_counter_total : 0
read_repairs_fallback_notfound_count : 123629
read_repairs_fallback_notfound_one : 0
read_repairs_fallback_outofdate_count : undefined
read_repairs_fallback_outofdate_one : undefined
read_repairs_hll : 0
read_repairs_hll_total : 0
read_repairs_map : 0
read_repairs_map_total : 0
read_repairs_primary_notfound_count : 213861
read_repairs_primary_notfound_one : 0
read_repairs_primary_outofdate_count : undefined
read_repairs_primary_outofdate_one : undefined
read_repairs_set : 0
read_repairs_set_total : 0
read_repairs_total : 336530
skipped_read_repairs : 0
skipped_read_repairs_total : 0

For something odd and abnormal I can remember the situation when one of the nodes can't successfully start after rollout restart. This situation was not the only one in the lifetime of the cluster.
The node that cannot start the riak process sent log output like this.

Failed to start riak_kv_eleveldb_backend backend for index 43388329858261180384171893471264652146035497

For the solution, we decided to remove the broken index from the data directory and shift the responsibility for index recovery to anti-entropy processes. When I say "remove" I mean like the command find ./ -name 43388329858261180384171893471264652146035497 | xargs -n1 rm -r for the example above. So indexes have been removed from the tictac_aae and leveldb directories.
After removing the corrupted index node became available and continued work.