Skip to content

Fix SST files not being cleaned up in the locations folder #4555

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
zymap merged 6 commits into from
Apr 1, 2025
Merged

Fix SST files not being cleaned up in the locations folder #4555

zymap merged 6 commits into from
Apr 1, 2025

Conversation

@danpi
Copy link
Contributor

@danpi danpi commented Feb 20, 2025

Descriptions of the changes in this PR:

Fix #4554

Motivation

Resolve issue with SST files remaining uncleared in the locations folder.

Changes

  1. Reuse the cleanup logic of GarbageCollectorThread to trigger entryLocationCompact() when a major compaction occurs.
  2. Add the entryLocationCompaction configuration option to control the above behavior, disabled by default.

@danpi
Copy link
Contributor Author

danpi commented Feb 20, 2025

I believe this issue is related to optimization #3653.

When using deleteRange to replace the one-by-one deletion method, only one action is written to RocksDB. If RocksDB is performing tiered compaction, this action is not guaranteed to be picked up, causing SST files to remain undeleted.

To resolve this issue:
Option 1: Revert to the previous logic of iteratively deleting each key.
Option 2: Use compactRange to forcefully clean up levels L0 to L6.
I chose to perform a compactRange synchronously when executing a major compaction.

@hangc0276
Copy link
Contributor

hangc0276 commented Feb 27, 2025 •
edited
Loading

I believe this issue is related to optimization #3653.

When using deleteRange to replace the one-by-one deletion method, only one action is written to RocksDB. If RocksDB is performing tiered compaction, this action is not guaranteed to be picked up, causing SST files to remain undeleted.

To resolve this issue: Option 1: Revert to the previous logic of iteratively deleting each key. Option 2: Use compactRange to forcefully clean up levels L0 to L6. I chose to perform a compactRange synchronously when executing a major compaction.

@danpi Great job! I think it should be a bug in RocksDB. We can use option 2. Please check the failed CI, thanks.

@danpi
Copy link
Contributor Author

danpi commented Mar 6, 2025

@hangc0276 Addressed the feedback above. PTAL, thanks.

hangc0276
hangc0276 reviewed Mar 6, 2025
View reviewed changes
bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/GarbageCollectorThread.java Outdated
long majorCompactionMaxTimeMillis;
long lastMajorCompactionTime;

boolean enableEntryLocationCompaction = false;
Copy link
Contributor

@hangc0276 hangc0276 Mar 6, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove this one and use entryLocationCompactionInterval > 0 instead?

bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/GarbageCollectorThread.java Outdated
minorCompacting.set(false);
}
}
if (enableEntryLocationCompaction && (curTime - lastEntryLocationCompactionTime
Copy link
Contributor

@hangc0276 hangc0276 Mar 6, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We'd better introduce a random factor for it. If we roll restart the BookKeeper cluster, and all the bookies start time are the same, which will make all the bookies triggered RocksDB compaction at the same time and impact the read latency.

Copy link
Contributor

@hangc0276 hangc0276 Mar 6, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The random factor can be any time in [0, entryLocationCompactionInterval]

@hangc0276
Copy link
Contributor

hangc0276 commented Mar 6, 2025

@hangc0276 Addressed the feedback above. PTAL, thanks.

@danpi Thanks for your contribution. I left more comments; please take a look. Thanks.

@danpi
Copy link
Contributor Author

danpi commented Mar 10, 2025

@hangc0276 Addressed the feedback above. PTAL, thanks.

@danpi Thanks for your contribution. I left more comments; please take a look. Thanks.

@hangc0276 Add randomCompactionDelay to avoid all the bookies triggering compaction simultaneously. PTAL, thanks.

@hangc0276
Copy link
Contributor

hangc0276 commented Mar 20, 2025

@danpi Would you please fix the check style issue?

Error:  /home/runner/work/bookkeeper/bookkeeper/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/GarbageCollectorThread.java:220:30: '=' is not preceded with whitespace. [WhitespaceAround]
Error:  /home/runner/work/bookkeeper/bookkeeper/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/GarbageCollectorThread.java:301: Line is longer than 120 characters (found 121). [LineLength]
Error:  /home/runner/work/bookkeeper/bookkeeper/bookkeeper-server/src/main/java/org/apache/bookkeeper/conf/ServerConfiguration.java:3243: Line is longer than 120 characters (found 122). [LineLength]
Audit done.
[INFO] There are 3 errors reported by Checkstyle 9.3 with buildtools/src/main/resources/bookkeeper/checkstyle.xml ruleset.
Error:  src/main/java/org/apache/bookkeeper/bookie/GarbageCollectorThread.java:[220,30] (whitespace) WhitespaceAround: '=' is not preceded with whitespace.
Error:  src/main/java/org/apache/bookkeeper/bookie/GarbageCollectorThread.java:[301] (sizes) LineLength: Line is longer than 120 characters (found 121).
Error:  src/main/java/org/apache/bookkeeper/conf/ServerConfiguration.java:[3243] (sizes) LineLength: Line is longer than 120 characters (found 122).

@hangc0276 hangc0276 mentioned this pull request Mar 20, 2025
Merged
@horizonzy
Copy link
Member

horizonzy commented Mar 26, 2025

@danpi Could you help to fix the checkstyle.

@zymap zymap merged commit ede1ba9 into apache:master Apr 1, 2025
23 checks passed
@zymap zymap added this to the 4.18.0 milestone Apr 2, 2025
zymap added a commit that referenced this pull request Apr 7, 2025
@danpi @zymap
Fix SST files not being cleaned up in the locations folder (#4555)
138849a 
* fix entry location compaction

* replace entryLocationCompactionEnable with entryLocationCompactionInterval

* Add randomCompactionDelay to avoid all the bookies triggering compaction simultaneously

* Fix the style issue

* Fix the style issue

* Fix test

---------

Co-authored-by: houbonan <[email protected]>
Co-authored-by: zymap <[email protected]>
(cherry picked from commit ede1ba9)
zymap added a commit that referenced this pull request Apr 7, 2025
@danpi @zymap
Fix SST files not being cleaned up in the locations folder (#4555)
fbd33b5 
* fix entry location compaction

* replace entryLocationCompactionEnable with entryLocationCompactionInterval

* Add randomCompactionDelay to avoid all the bookies triggering compaction simultaneously

* Fix the style issue

* Fix the style issue

* Fix test

---------

Co-authored-by: houbonan <[email protected]>
Co-authored-by: zymap <[email protected]>
(cherry picked from commit ede1ba9)
priyanshu-ctds pushed a commit to datastax/bookkeeper that referenced this pull request Jul 11, 2025
@danpi @zymap @priyanshu-ctds
Fix SST files not being cleaned up in the locations folder (apache#4555)
39da363 
* fix entry location compaction

* replace entryLocationCompactionEnable with entryLocationCompactionInterval

* Add randomCompactionDelay to avoid all the bookies triggering compaction simultaneously

* Fix the style issue

* Fix the style issue

* Fix test

---------

Co-authored-by: houbonan <[email protected]>
Co-authored-by: zymap <[email protected]>
(cherry picked from commit ede1ba9)
(cherry picked from commit 138849a)
sandeep-ctds pushed a commit to datastax/bookkeeper that referenced this pull request Jul 22, 2025
@danpi @zymap @sandeep-ctds
Fix SST files not being cleaned up in the locations folder (apache#4555)
9a42316 
* fix entry location compaction

* replace entryLocationCompactionEnable with entryLocationCompactionInterval

* Add randomCompactionDelay to avoid all the bookies triggering compaction simultaneously

* Fix the style issue

* Fix the style issue

* Fix test

---------

Co-authored-by: houbonan <[email protected]>
Co-authored-by: zymap <[email protected]>
(cherry picked from commit ede1ba9)
(cherry picked from commit 138849a)
@lhotari lhotari mentioned this pull request Sep 22, 2025
Merged
@lhotari lhotari mentioned this pull request Dec 12, 2025
Closed
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@hangc0276 hangc0276 hangc0276 approved these changes

@dlg99 dlg99 dlg99 approved these changes

@lhotari lhotari Awaiting requested review from lhotari

@eolivelli eolivelli Awaiting requested review from eolivelli

@zymap zymap Awaiting requested review from zymap

Assignees

@danpi danpi

Labels

area/bookie cherry-picked/branch-4.16 cherry-picked/branch-4.17 release/4.16.7 release/4.17.2 type/bug

Projects

None yet

Milestone

4.18.0

Development

Successfully merging this pull request may close these issues.

[BUG]Large number of SST files in locations not being cleared after upgrade

6 participants

@danpi @hangc0276 @horizonzy @dlg99 @lhotari @zymap