error persisting newly received tx org.rocksdb.RocksDBException:

[damageco]

Bug description

After a few hours, my node start to raise RocksDBException on new transaction persisting.

IRI version

1.8.5

Hardware Spec

Bare metal machine.
8 CPU cores i7
8 GB RAM
1To + 2To disk
1 GBit/s
Traffic Unlimited

Errors

'm was using 1.8.4, I've upgraded to 1.8.5 yesterday, but the errors always appears after of few hours.
I've tried to change the Linux distribution, RedHat, Ubuntu, same problem.

Got this stacktrace : ERROR ReceivedStage:57 - error persisting newly received tx
org.rocksdb.RocksDBException: block checksum mismatch: expected 1900677990, got 1492066657 in /iri/data//002100.sst offset 26773563 size 1276

[GalRogozinski]

This seems like an underlying issue of RocksDB issue. According to this ticket it will be hard to solve...

but the errors always appears after of few hours

Are you starting from the same DB?
Because once it has been corrupted it is probably hard to fix?

The only thing I can recommend atm is to try to recompile IRI with different versions of RocksDb.
You can do this by recompiling IRI with mvn clean package and changing the version in the pom.xml in this section

<dependency>
            <groupId>org.rocksdb</groupId>
            <artifactId>rocksdbjni</artifactId>
            <version>6.3.6</version>
</dependency>

Perhaps upgrading to a later version can help?

[damageco]

Ok, let's try it. I've upgraded to 6.6.4 and start a new node. By the way, the last 6.7.3 version of RocksDb change the public API and breaks compilation.

After 5 minutes running I've got this error :
03/29 12:31:32.268 [Milestone Solidifier] ERROR MilestoneSolidifier:465 - Error while solidifying milestone #1361816
org.rocksdb.RocksDBException: block checksum mismatch: expected 1674591948, got 3341790192 in mainnetdb/000463.sst offset 46314903 size 3030

I dig the net, but I can't find way to fix the corrupted files. So, yes, I always have to start with a new snapshot database.

[GalRogozinski]

If it keeps happening to you after a snapshot report please :-)

[dsampaolo]

It should be RocksDB related. I'm using ArangoDB (which uses RocksDB) and I have exactly the same problem. This doesn't seem to be related to iotaledger...