Investigate `Error: Corruption: SST file is ahead of WALs in CF metadata`

[mlegner]

Description

When running walrus-node db-tool drop-column-families, we sometimes get the error Error: Corruption: SST file is ahead of WALs in CF metadata. In some cases, this was resolved by briefly starting the node and stopping it again.

Code of Conduct

• I agree to follow this project's Code of Conduct.

[github-actions]

Thank you for opening this issue, a team member will review it shortly. Until then, please do not interact with any users that claim to be from Sui or Walrus support and do not click on any links!

[dennyxcodes]

It appears the db-tool is failing to open a database that wasn't shut down cleanly. This issue stems from an inconsistency in the RocksDB options used by the db-tool versus the main walrus-node.

The db-tool was using default RocksDB options, which prevented it from properly recovering the database from the write-ahead logs (WALs). This mismatch in configuration is what caused the corruption error.

This may be solved by ensuring the db-tool is configured to use the exact same RocksDB options as the walrus-node when performing recovery or maintenance tasks.

[mlegner]

Thanks for the insight, @dennyxcodes.

@liquid-helium: Could you take a look if this can be resolved?