5 minutes not enough for parallel cleanup after added new node

[aleksbykov]

Argus

Scylla version: 2026.1.0~rc0-20260125.f94296e0ae43 with build-id 9680213fda6f301234c43da8ca27e47953987cd8

Kernel Version: 6.14.0-1018-aws

Need to increase nodetool cleanup timeout for parallel run in method: nodetool_cleanup_on_all_nodes_parallel, because 5 minutes was not enough in the failed job, and several cleanups was aborted by timeout.

https://github.com//issues/12261

Installation details

Cluster size: 8 nodes (i4i.4xlarge)

Scylla Nodes used in this run:

- longevity-cdc-100gb-8h-multi-dc-tpc-db-node-9af7382c-1 (54.75.73.134 | 10.4.3.169) (shards: 14)


- longevity-cdc-100gb-8h-multi-dc-tpc-db-node-9af7382c-2 (34.245.144.83 | 10.4.3.223) (shards: -1)


- longevity-cdc-100gb-8h-multi-dc-tpc-db-node-9af7382c-3 (3.251.88.2 | 10.4.3.46) (shards: 14)


- longevity-cdc-100gb-8h-multi-dc-tpc-db-node-9af7382c-4 (52.215.206.135 | 10.4.0.37) (shards: 14)


- longevity-cdc-100gb-8h-multi-dc-tpc-db-node-9af7382c-5 (3.226.253.176 | 10.12.2.254) (shards: 14)


- longevity-cdc-100gb-8h-multi-dc-tpc-db-node-9af7382c-6 (52.3.234.182 | 10.12.3.175) (shards: 14)


- longevity-cdc-100gb-8h-multi-dc-tpc-db-node-9af7382c-7 (35.170.198.195 | 10.12.1.174) (shards: 14)


- longevity-cdc-100gb-8h-multi-dc-tpc-db-node-9af7382c-8 (3.230.145.204 | 10.12.1.75) (shards: 14)


- longevity-cdc-100gb-8h-multi-dc-tpc-db-node-9af7382c-9 (44.204.96.241 | 10.12.0.40) (shards: 14)

OS / Image: ami-0150598259ac2ded5 ami-0752fe268fe00bbbd (aws: N/A)

Test: longevity-cdc-8h-multi-dc-topology-changes-streaming-vnodes-test
Test id: 9af7382c-2d3f-4b05-b15d-edbfb34fe8b6
Test name: scylla-2026.1/vnodes/tier2/longevity-cdc-8h-multi-dc-topology-changes-streaming-vnodes-test

Test method: longevity_test.LongevityTest.test_custom_time

Test config file(s):

• longevity-cdc-100gb-8h-multi-dc-topology-changes.yaml

Logs:

• longevity-cdc-100gb-8h-multi-dc-tpc-db-node-9af7382c-5

• core.scylla-longevity-cdc-100gb-8h-multi-dc-tpc-db-node-9af7382c-4-2026-01-28_14-22-58.zst

• db-cluster-9af7382c.tar.zst

• schema-logs-9af7382c.tar.zst

• sct-runner-events-9af7382c.tar.zst

• sct-9af7382c.log.tar.zst

• loader-set-9af7382c.tar.zst

• monitor-set-9af7382c.tar.zst

• builder-9af7382c.log.tar.gz

Jenkins job URL

[yarongilor]

reproduced on 2026.1 https://argus.scylladb.com/tests/scylla-cluster-tests/963da125-daa1-4822-a25a-61889ca62c05:

2026-02-12 03:07:33.861: (SoftTimeoutEvent Severity.ERROR) period_type=one-time event_id=63b2ed9b-3677-4e67-87a1-62c39d343241, source=SoftTimeout message=operation 'CLEANUP' is finished and took 1566.3s (soft timeout was set to 60s)
Traceback (most recent call last):
  File "/usr/local/lib/python3.14/threading.py", line 1043, in _bootstrap
    self._bootstrap_inner()
  File "/usr/local/lib/python3.14/threading.py", line 1081, in _bootstrap_inner
    self._context.run(self.run)
  File "/usr/local/lib/python3.14/threading.py", line 1023, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.14/concurrent/futures/thread.py", line 119, in _worker
    work_item.run(ctx)
  File "/usr/local/lib/python3.14/concurrent/futures/thread.py", line 86, in run
    result = ctx.run(self.task)
  File "/usr/local/lib/python3.14/concurrent/futures/thread.py", line 73, in run
    return fn(*args, **kwargs)
  File "/home/ubuntu/scylla-cluster-tests/sdcm/utils/parallel_object.py", line 73, in inner
    return_val = fun(*args, **kwargs)
  File "/home/ubuntu/scylla-cluster-tests/sdcm/nemesis.py", line 2145, in _nodetool_cleanup
    with adaptive_timeout(Operations.CLEANUP, node, timeout=60):
  File "/usr/local/lib/python3.14/contextlib.py", line 148, in __exit__
    next(self.gen)
  File "/home/ubuntu/scylla-cluster-tests/sdcm/utils/adaptive_timeouts/__init__.py", line 249, in adaptive_timeout
    SoftTimeoutEvent(operation=operation.name, soft_timeout=soft_timeout, duration=duration).publish_or_dump()

[jsmolar]

Reproduced again on 2026.1 RC1: https://argus.scylladb.com/tests/scylla-cluster-tests/e92f565c-aeea-4dc1-b9ff-bc8b7608ab72
In this run parallel clean up completed after ~900-1000 seconds for 6 nodes

[timtimb0t]

https://argus.scylladb.com/tests/scylla-cluster-tests/e7d7f820-9f29-4229-a8d5-cb33bfa9eabb
seems like reproduced there

2026-02-12T18:30:42.608 alternator-48h-2026-1-db-node-e7d7f820-1  !INFO | scylla-cluster-tests[17530] nodetool /usr/bin/nodetool -u cassandra -pw 'cassandra'  cleanup  completed after 390.63s on alternator-48h-2026-1-db-node-e7d7f820-1 [10.4.2.112]