disrupt_add_remove_mv is systematically skipped in sanity runs

[timtimb0t]

For example this run: https://argus.scylladb.com/tests/scylla-cluster-tests/5a9c1409-e24e-4534-9509-170cba6551a2
Usually, this nemesis runs one of the first and gets skipped because of:

Non-system keyspace and table are not found. nemesis can't be run

within this nemesis cfstats cmd returns following output:

l:276  c:RemoteLibSSH2CmdRunner p:DEBUG > <10.142.0.59>: Total number of tables: 88
l:276  c:RemoteLibSSH2CmdRunner p:DEBUG > <10.142.0.59>: ----------------
l:276  c:RemoteLibSSH2CmdRunner p:DEBUG > <10.142.0.59>: Keyspace : keyspace1
l:276  c:RemoteLibSSH2CmdRunner p:DEBUG > <10.142.0.59>: 	Read Count: 0
l:276  c:RemoteLibSSH2CmdRunner p:DEBUG > <10.142.0.59>: 	Read Latency: NAN ms
l:276  c:RemoteLibSSH2CmdRunner p:DEBUG > <10.142.0.59>: 	Write Count: 3375552
l:276  c:RemoteLibSSH2CmdRunner p:DEBUG > <10.142.0.59>: 	Write Latency: 4.420610317956885E-06 ms
l:276  c:RemoteLibSSH2CmdRunner p:DEBUG > <10.142.0.59>: 	Pending Flushes: 0
l:276  c:RemoteLibSSH2CmdRunner p:DEBUG > <10.142.0.59>: 		Table: standard1
l:276  c:RemoteLibSSH2CmdRunner p:DEBUG > <10.142.0.59>: 		SSTable count: 0
l:276  c:RemoteLibSSH2CmdRunner p:DEBUG > <10.142.0.59>: 		Space used (live): 0
l:276  c:RemoteLibSSH2CmdRunner p:DEBUG > <10.142.0.59>: 		Space used (total): 0
l:276  c:RemoteLibSSH2CmdRunner p:DEBUG > <10.142.0.59>: 		Space used by snapshots (total): 0
l:276  c:RemoteLibSSH2CmdRunner p:DEBUG > <10.142.0.59>: 		Off heap memory used (total): 5682102272
l:276  c:RemoteLibSSH2CmdRunner p:DEBUG > <10.142.0.59>: 		SSTable Compression Ratio: 0.00
l:276  c:RemoteLibSSH2CmdRunner p:DEBUG > <10.142.0.59>: 		Number of partitions (estimate): 0
l:276  c:RemoteLibSSH2CmdRunner p:DEBUG > <10.142.0.59>: 		Memtable cell count: 1707630
l:276  c:RemoteLibSSH2CmdRunner p:DEBUG > <10.142.0.59>: 		Memtable data size: 1441887224
l:276  c:RemoteLibSSH2CmdRunner p:DEBUG > <10.142.0.59>: 		Memtable off heap memory used: 5682102272
l:276  c:RemoteLibSSH2CmdRunner p:DEBUG > <10.142.0.59>: 		Memtable switch count: 0
l:276  c:RemoteLibSSH2CmdRunner p:DEBUG > <10.142.0.59>: 		Local read count: 0
l:276  c:RemoteLibSSH2CmdRunner p:DEBUG > <10.142.0.59>: 		Local read latency: 0.000 ms
l:276  c:RemoteLibSSH2CmdRunner p:DEBUG > <10.142.0.59>: 		Local write count: 3375648
l:276  c:RemoteLibSSH2CmdRunner p:DEBUG > <10.142.0.59>: 		Local write latency: 0.004 ms
l:276  c:RemoteLibSSH2CmdRunner p:DEBUG > <10.142.0.59>: 		Pending flushes: 0
l:276  c:RemoteLibSSH2CmdRunner p:DEBUG > <10.142.0.59>: 		Percent repaired: 0.0
l:276  c:RemoteLibSSH2CmdRunner p:DEBUG > <10.142.0.59>: 		Bloom filter false positives: 0
l:276  c:RemoteLibSSH2CmdRunner p:DEBUG > <10.142.0.59>: 		Bloom filter false ratio: 0.00000
l:276  c:RemoteLibSSH2CmdRunner p:DEBUG > <10.142.0.59>: 		Bloom filter space used: 0
l:276  c:RemoteLibSSH2CmdRunner p:DEBUG > <10.142.0.59>: 		Bloom filter off heap memory used: 0
l:276  c:RemoteLibSSH2CmdRunner p:DEBUG > <10.142.0.59>: 		Index summary off heap memory used: 0
l:276  c:RemoteLibSSH2CmdRunner p:DEBUG > <10.142.0.59>: 		Compression metadata off heap memory used: 0
l:276  c:RemoteLibSSH2CmdRunner p:DEBUG > <10.142.0.59>: 		Compacted partition minimum bytes: 0
l:276  c:RemoteLibSSH2CmdRunner p:DEBUG > <10.142.0.59>: 		Compacted partition maximum bytes: 0
l:276  c:RemoteLibSSH2CmdRunner p:DEBUG > <10.142.0.59>: 		Compacted partition mean bytes: 0
l:276  c:RemoteLibSSH2CmdRunner p:DEBUG > <10.142.0.59>: 		Average live cells per slice (last five minutes): 0.0
l:276  c:RemoteLibSSH2CmdRunner p:DEBUG > <10.142.0.59>: 		Maximum live cells per slice (last five minutes): 0
l:276  c:RemoteLibSSH2CmdRunner p:DEBUG > <10.142.0.59>: 		Average tombstones per slice (last five minutes): 0.0
l:276  c:RemoteLibSSH2CmdRunner p:DEBUG > <10.142.0.59>: 		Maximum tombstones per slice (last five minutes): 0
l:276  c:RemoteLibSSH2CmdRunner p:DEBUG > <10.142.0.59>: 		Dropped Mutations: 0
l:276  c:RemoteLibSSH2CmdRunner p:DEBUG > <10.142.0.59>: 
l:276  c:RemoteLibSSH2CmdRunner p:DEBUG > <10.142.0.59>: ----------------

It seems all the data is still in the memtable and needs to be flushed before attempting to create the MV.
Maybe we can modify this nemesis to invoke a flush command if cfstats returns nothing?

[timtimb0t]

Hi @fruch , could you please share your opinion?

[fruch]

Which test case it is, we should have machinery that waits for enough data to be flushed before starting nemesis.

Maybe it's not configured for this test ?

[timtimb0t]

Here is argus run: https://argus.scylladb.com/tests/scylla-cluster-tests/5a9c1409-e24e-4534-9509-170cba6551a2
job: https://jenkins.scylladb.com/job/scylla-master/job/longevity/job/longevity-10gb-3h-gce-test/
configuration: test-cases/longevity/longevity-10gb-3h.yaml

Maybe it's not configured for this test ?

I don’t think so. For example, in runs 1015 and 1028 this nemesis was executed and the test passed. From my perspective, sanity tests should consistently cover all mandatory features, and MVs seem to be one of them. On the other hand, this nemesis runs from time to time.
Maybe the nemesis_during_prepare: true is the reason?
I think at this point the value is inherited from test_default. We can set it explicitly in the test YAML, but that would likely increase the overall test runtime.

UPD: the same may be said about this job too:
https://argus.scylladb.com/tests/scylla-cluster-tests/b1ced3a7-fb66-4a4b-ba8c-a5fc109b052a
https://jenkins.scylladb.com/job/scylla-master/job/longevity/job/longevity-10gb-3h-azure-test/