Skip to content

Conversation

@lhotari
Copy link
Member

@lhotari lhotari commented Apr 15, 2025

Fix #4409

Motivation

Upgrade RocksDB to v.9.9.3 to pick up latest performance and stability improvements and fixes in deleteRanges.
Hopefully deleteRanges() is stable enough for us to try again perf improvements #3653 that were reverted previously.

Additional context

The upgrade/downgrade tests pass now. The problem was that format_version wasn't set in all cases and that resulted in using the default format_version=6 of RocksDB 9.x. There were multiple bugs in setting the format_version. The related fixes to address this are #4466, #4480, #4559 and #4560.

The RocksDB 9.0.0 release notes mention the default format_version change:

format_version=6 is the new default setting in BlockBasedTableOptions, for more robust data integrity checking. DBs and SST files written with this setting cannot be read by RocksDB versions before 8.6.0.

Changes

  • Changed RocksDB version, updated code to use changed API.
  • Refactored compatibility tests for easier maintainability when new versions get added
  • Dropped some upgrade/downgrade compatibility tests for very old versions
  • Updated instructions for running compatibility tests on Mac Apple Silicon with Docker Desktop

@lhotari lhotari changed the title [WIP] Upgrade to RocksDB 9.9.3 Upgrade to RocksDB 9.9.3 Apr 16, 2025
@lhotari lhotari self-assigned this Apr 16, 2025
@lhotari lhotari added this to the 4.18.0 milestone Apr 16, 2025
hezhangjian pushed a commit that referenced this pull request Apr 16, 2025
### Motivation

While working on #4580, I noticed that the Commons Compress version and the Commons Codec library version aren't compatible. It causes a ClassDefNotFoundError in running tests which were using some specific methods of docker-java.
While exploring this, I noticed that Apache Commons library versions haven't been kept up-to-date for a long time and it's better to handle that for 4.18.0 release.

### Changes

- upgrade commons-cli from 1.2 to 1.9.0
- upgrade commons-codec from 1.6 to 1.18.0
- upgrade commons-io from 2.17.0 to 2.19.0
- upgrade commons-lang3 from 3.6 to 3.17.0
- upgrade commons-compress from 1.26.0 to 1.27.0
lhotari added a commit that referenced this pull request Apr 16, 2025
While working on #4580, I noticed that the Commons Compress version and the Commons Codec library version aren't compatible. It causes a ClassDefNotFoundError in running tests which were using some specific methods of docker-java.
While exploring this, I noticed that Apache Commons library versions haven't been kept up-to-date for a long time and it's better to handle that for 4.18.0 release.

- upgrade commons-cli from 1.2 to 1.9.0
- upgrade commons-codec from 1.6 to 1.18.0
- upgrade commons-io from 2.17.0 to 2.19.0
- upgrade commons-lang3 from 3.6 to 3.17.0
- upgrade commons-compress from 1.26.0 to 1.27.0

(cherry picked from commit bcd6b52)
lhotari added a commit that referenced this pull request Apr 16, 2025
While working on #4580, I noticed that the Commons Compress version and the Commons Codec library version aren't compatible. It causes a ClassDefNotFoundError in running tests which were using some specific methods of docker-java.
While exploring this, I noticed that Apache Commons library versions haven't been kept up-to-date for a long time and it's better to handle that for 4.18.0 release.

- upgrade commons-cli from 1.2 to 1.9.0
- upgrade commons-codec from 1.6 to 1.18.0
- upgrade commons-io from 2.17.0 to 2.19.0
- upgrade commons-lang3 from 3.6 to 3.17.0
- upgrade commons-compress from 1.26.0 to 1.27.0

(cherry picked from commit bcd6b52)
@lhotari
Copy link
Member Author

lhotari commented Apr 17, 2025

@hangc0276 @dlg99 @eolivelli @Ghatage Please review. The upgrade/downgrade tests pass now. Previously there was a problem that format_version wasn't set in all cases and that resulted in using the default format_version=6 of RocksDB 9.x. This was fixed by PRs #4466, #4480, #4559 and #4560.

@lhotari lhotari requested a review from Ghatage April 17, 2025 07:08
Copy link
Member

@StevenLuMT StevenLuMT left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great jobs

@merlimat merlimat merged commit fabe44b into apache:master Apr 22, 2025
23 of 24 checks passed
priyanshu-ctds pushed a commit to datastax/bookkeeper that referenced this pull request Jul 11, 2025
While working on apache#4580, I noticed that the Commons Compress version and the Commons Codec library version aren't compatible. It causes a ClassDefNotFoundError in running tests which were using some specific methods of docker-java.
While exploring this, I noticed that Apache Commons library versions haven't been kept up-to-date for a long time and it's better to handle that for 4.18.0 release.

- upgrade commons-cli from 1.2 to 1.9.0
- upgrade commons-codec from 1.6 to 1.18.0
- upgrade commons-io from 2.17.0 to 2.19.0
- upgrade commons-lang3 from 3.6 to 3.17.0
- upgrade commons-compress from 1.26.0 to 1.27.0

(cherry picked from commit bcd6b52)
(cherry picked from commit fab4780)
sandeep-ctds pushed a commit to datastax/bookkeeper that referenced this pull request Jul 22, 2025
While working on apache#4580, I noticed that the Commons Compress version and the Commons Codec library version aren't compatible. It causes a ClassDefNotFoundError in running tests which were using some specific methods of docker-java.
While exploring this, I noticed that Apache Commons library versions haven't been kept up-to-date for a long time and it's better to handle that for 4.18.0 release.

- upgrade commons-cli from 1.2 to 1.9.0
- upgrade commons-codec from 1.6 to 1.18.0
- upgrade commons-io from 2.17.0 to 2.19.0
- upgrade commons-lang3 from 3.6 to 3.17.0
- upgrade commons-compress from 1.26.0 to 1.27.0

(cherry picked from commit bcd6b52)
(cherry picked from commit fab4780)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Task: upgrade RocksDB

4 participants