introduce clear separation between proposed compaction, eligibility status, and compaction mode, along with improved task state tracking by cecemei · Pull Request #18968 · apache/druid

cecemei · 2026-01-30T08:44:44Z

This PR refactors compaction eligibility evaluation by introducing clear separation between proposed compaction (what segments to compact), eligibility status (whether compaction is needed), and compaction mode (how to compact), along with improved task state tracking.

Motivation

The previous CompactionStatus implementation mixed multiple concerns:

Eligibility checks: Is compaction needed based on config?
Policy decisions: Should we compact this interval now?
Task state tracking: Is a compaction task already running?

This made it difficult to reason about compaction decisions and track the state of compaction candidates through the system.

Key Changes

1. Introduced `CompactionMode` Enum

New enum to represent different compaction strategies:

FULL_COMPACTION: Compact all segments in the interval
NOT_APPLICABLE: No compaction should be performed

The mode determines whether a candidate will be queued for compaction or skipped.

2. Refactored `CompactionCandidate`

New structure:

ProposedCompaction: Nested class containing the segment list, intervals, and statistics
- Represents the raw "what should we compact?" without policy decisions
CompactionStatus: Eligibility evaluation result (COMPLETE, ELIGIBLE, NOT_ELIGIBLE)
- Tracks compacted vs uncompacted segments and statistics
- Contains reason for ineligibility when applicable
CompactionMode: How to perform compaction (FULL_COMPACTION or NOT_APPLICABLE)
policyNote: Optional note from the policy explaining why this candidate was chosen/skipped

Benefits:

Clear separation between the proposed segments and the decision to compact them
Easier to track why candidates are skipped or accepted
More maintainable and testable code structure

3. Updated `CompactionStatus`

New responsibilities:

Evaluates if segments need compaction based on config settings
Returns state: COMPLETE, ELIGIBLE, or NOT_ELIGIBLE
For eligible intervals, tracks:
- Compacted segment statistics
- Uncompacted segment statistics
Provides detailed reason when not eligible

Separation from policy:

CompactionStatus checks config-based requirements (granularity, rollup, etc.)
Policy then decides whether to actually compact eligible intervals

4. Simplified `CompactionCandidateSearchPolicy`

Changed interface:

Removed: checkEligibilityForCompaction(candidate, taskStatus) → Eligibility
Added: createCandidate(proposedCompaction, eligibility) → CompactionCandidate

Benefits:

Policies now create the final candidate after receiving eligibility info
Can add policy-specific notes and select compaction mode
Cleaner separation: status checks config compliance, policy makes scheduling decisions

5. Improved Task State Tracking

New TaskState enum in CompactionCandidate:

READY: No task running, can start compaction
TASK_IN_PROGRESS: Compaction task already running
RECENTLY_COMPLETED: Task recently finished, segments not yet updated

Benefits:

Clear distinction between eligibility and task state
Prevents launching duplicate tasks for the same interval
Easier to understand candidate lifecycle

Code Flow Example

1. DataSourceCompactibleSegmentIterator finds segments in a time chunk
   → Creates ProposedCompaction

2. CompactionStatus evaluates config compliance
   → Returns ELIGIBLE with compacted/uncompacted stats

3. Policy receives ProposedCompaction + CompactionStatus
   → Calls createCandidate() to produce final CompactionCandidate
   → Sets mode = FULL_COMPACTION if should compact, NOT_APPLICABLE if skip
   → Adds policyNote explaining decision

4. CompactionStatusTracker checks task state
   → Returns READY/TASK_IN_PROGRESS/RECENTLY_COMPLETED

5. Scheduler decides whether to launch task based on mode + task state

Testing

Added CompactionCandidateTest for new candidate structure
Added CompactionStatusBuilderTest for eligibility evaluation
Updated existing tests to work with new architecture

This PR has:

kfaraz

Thanks for the PR, @cecemei !

I like the idea of separating the CompactionStatus (i.e. the current degree of compaction of an interval) from the Eligibility (i.e. whether an interval should be picked for compaction or not).

I have left some suggestions to aid with the separation. Let me know if they make sense.

server/src/main/java/org/apache/druid/server/compaction/CompactionCandidateSearchPolicy.java

kfaraz · 2026-02-04T07:29:05Z

server/src/main/java/org/apache/druid/client/indexing/ClientCompactionIntervalSpec.java

  @JsonCreator
  public ClientCompactionIntervalSpec(
      @JsonProperty("interval") Interval interval,
+      @JsonProperty("uncompactedSegments") @Nullable List<SegmentDescriptor> uncompactedSegments,


It is a little confusing to add segments here since there is already a SpecificSegmentsSpec which specifies the set of segments that should be compacted.

I feel we should add a new type of CompactionInputSpec which will be used for incremental compaction only. It would have both a non-null interval as well as a non-empty set of segment IDs to compact incrementally. The CompactionTask would handle the different input specs accordingly.

SpecificSegmentsSpec is not supported by msq, and it somewhat felt a bit deprecated to me, maybe because of the segment lock stuff. i kinda like to specify an interval since it gives some certainty, also wonder maybe we should check non-null for interval here? i didnt see any instance with null interval.

i kinda like to specify an interval since it gives some certainty

Yes, I agree, in the new scheme of things, it is always nice to specify the interval.
That's why I advise adding a new implementation of CompactionInputSpec which will be used only for incremental compaction so that it is easy to distinguish the two on the task side as well. We need not piggyback on the existing CompactionIntervalSpec.

server/src/main/java/org/apache/druid/server/compaction/CompactionCandidateSearchPolicy.java

server/src/main/java/org/apache/druid/server/compaction/CompactionCandidate.java

kfaraz · 2026-02-04T09:14:40Z

server/src/main/java/org/apache/druid/server/compaction/CompactionStatus.java

+     *
+     * @return Pair of eligibility status and compaction status with reason for first failed check
+     */
+    Pair<CompactionCandidateSearchPolicy.Eligibility, CompactionStatus> evaluate()


Since the CompactionStatus and eligibility are going to be two separate things now, this class should not deal with eligibility at all. We can retain the old code in this class.

Once the CompactionStatus has been computed, that should be passed to CompactionStatusTracker.computeEligiblity() to determine the eligibility.

server/src/main/java/org/apache/druid/server/compaction/CompactionStatusTracker.java

server/src/main/java/org/apache/druid/server/compaction/CompactionStatus.java

...r/src/main/java/org/apache/druid/server/compaction/DataSourceCompactibleSegmentIterator.java

server/src/main/java/org/apache/druid/server/coordinator/duty/CompactSegments.java

kfaraz · 2026-02-04T10:48:51Z

Currently, CompactionStatus conflates two concerns:
Status: What's the task status of a CompactionCandidate?
Eligibility: Should we compact this interval now, and how?

@cecemei , just so that we are on the same page, the CompactionStatus does not exactly contain information of the compaction "task" (except for the RUNNING state, which we can now remove in this PR). It contains info of the current level of compaction in the interval, i.e. number of compacted segments, number of uncompacted segments and overall state (COMPLETE/PENDING/SKIPPED).

It would be nice to get rid of the SKIPPED state too since it is more of an eligibility thing, but I guess we can leave it for now.

kfaraz

Thanks for the update, @cecemei !
I have left some follow-up suggestions to clean up the PR.

...r/src/main/java/org/apache/druid/server/compaction/DataSourceCompactibleSegmentIterator.java

server/src/main/java/org/apache/druid/server/compaction/CompactionCandidate.java

server/src/main/java/org/apache/druid/server/compaction/CompactionStatus.java

server/src/main/java/org/apache/druid/server/compaction/CompactionEligibility.java

...r/src/main/java/org/apache/druid/server/compaction/DataSourceCompactibleSegmentIterator.java

server/src/main/java/org/apache/druid/server/compaction/CompactionCandidate.java

server/src/main/java/org/apache/druid/server/compaction/CompactionStatusTracker.java

server/src/main/java/org/apache/druid/server/coordinator/duty/CompactSegments.java

server/src/main/java/org/apache/druid/server/compaction/CompactionEligibility.java

server/src/test/java/org/apache/druid/server/compaction/CompactionEligibilityEvaluateTest.java

cecemei · 2026-02-06T02:51:42Z

Thanks for the PR, @cecemei !

I like the idea of separating the CompactionStatus (i.e. the current degree of compaction of an interval) from the Eligibility (i.e. whether an interval should be picked for compaction or not).

I have left some suggestions to aid with the separation. Let me know if they make sense.

I addressed most comments except for the coordinator based compaction and incremental compaction feature change.

The major change is i moved a lot of CompactionStatus stuff to CompactionEligibility, and update the CHECK to return String (null for previous CompactionStatus.COMPLETE, and non-null for previous CompactionStatus.pending...), it seems simpler since after all we're just looking for a reason to do compaction.

capistrant · 2026-02-06T15:21:58Z

server/src/main/java/org/apache/druid/server/compaction/CompactionEligibility.java

+   * <p>
+   * This method performs a two-stage evaluation:
+   * <ol>
+   * <li>First, uses {@link Evaluator} to check if the candidate needs compaction


did we make any changes to the Evaluator logic in moving it to this file or is it just lift and place in new location?

...r/src/main/java/org/apache/druid/server/compaction/DataSourceCompactibleSegmentIterator.java

kfaraz · 2026-02-06T17:56:05Z

Had a brief discussion with @cecemei on this. I think there has been some confusion regarding the roles of CompactionStatus, CompactionEligibility and the task status. It is in part due to the fact that CompactionStatus was originally serving all of these roles but not perfectly.

In short, we shouldn't need to make any major modification to any class in this PR except CompactionEligibility.

I have shared some details offline. We should be able to get this resolved by Monday.

server/src/main/java/org/apache/druid/server/compaction/CompactionStatusTracker.java

cecemei · 2026-02-08T08:53:43Z

Hi @kfaraz @capistrant , I made some major changes to this pr, and hopefully it's in a much better state, please take a look when you're available, thanks! The incremental compaction related change has been removed from this pr, i'm going to make another one for that.

server/src/main/java/org/apache/druid/server/compaction/CompactionMode.java

server/src/main/java/org/apache/druid/server/coordinator/duty/CompactSegments.java

server/src/main/java/org/apache/druid/server/compaction/CompactionStatus.java

server/src/test/java/org/apache/druid/server/coordinator/duty/CompactSegmentsTest.java

kfaraz

@cecemei , I feel a lot of changes here are really not needed and probably make the definitions of various compaction entities confusing.

The only change that is really needed here is adding a CompactionMode enum to Eligibility.
The existing terminology CompactionCandidate, CompactionStatus, Eligibility, CompactionTaskStatus captures the respective purposes nicely and also provides a clear distinction. Let's stick with the existing terminology unless there is some real gap in the current nomenclature/functionality.
If you feel it needs some explanation, we may add/update javadocs.
You may either remove all the refactors from this PR (so that we can quickly unblock incremental compaction in #18996) OR update this PR to only make suggested modifications that do not break the existing definitions.

P.S. Existing definitions

Class	Definition
`CompactionCandidate`	A potential candidate for compaction, identified by an interval and some segment IDs
`CompactionStatus`	Current status of a `CompactionCandidate` indicated by number of compacted/uncompacted segments/bytes in the interval.
`Eligibility`	Given a `CompactionCandidate` and `CompactionStatus`, does the policy think it is eligible for compaction?
`CompactionTaskStatus`	Status of past/ongoing compaction tasks for a `CompactionCandidate`

All of these are distinct from each other and should preferably remain so.
There is room for improvement in these classes, but we should do them in a manner that does not completely change the existing semantics.

indexing-service/src/main/java/org/apache/druid/indexing/compact/CompactionJobQueue.java

kfaraz · 2026-02-11T18:28:10Z

server/src/main/java/org/apache/druid/server/compaction/CompactionMode.java

+    return createCandidate(proposedCompaction, eligibility, null);
+  }
+
+  public CompactionCandidate createCandidate(


We shouldn't really have this method here. The CompactionMode is an attribute that we associate to a CompactionCandidate. One should not be responsible for creating the other.

kfaraz · 2026-02-11T18:30:08Z

server/src/main/java/org/apache/druid/server/compaction/CompactionCandidateSearchPolicy.java

-             '}';
-    }
-  }
+  CompactionCandidate createCandidate(CompactionCandidate.ProposedCompaction candidate, CompactionStatus eligibility);


It doesn't make sense for the policy to be creating candidates since the candidates are identified much earlier by the DatasourceCompactibleSegmentIterator itself. The policy can only check the eligibility of a candidate for compaction. (I suppose the name CompactionCandidateSearchPolicy might be misleading here, since it is really the CompactionCandidatePickAndPriorizePolicy but it can be a mouthful, so we can stick with what we have.)

We should retain the original method here. The caller (CompactionJobQueue) should just check the eligibility and then decide whether to launch a job or not. If yes, then whether full or incremental.

gotcha - can just return CompactionMode here and let callsite handle creation of candidate.

No we should continue to use the existing method in CompactionCandidateSearchPolicy i.e. checkEligibility().

Just add a new enum inside the Eligibility class.

kfaraz · 2026-02-11T19:29:06Z

server/src/main/java/org/apache/druid/server/compaction/CompactionCandidate.java

+   * Used by {@link CompactionStatusTracker#computeCompactionTaskState(CompactionCandidate)}.
+   * The callsite then determines whether to launch compaction task or not.
+   */
+  public enum TaskState


Please do not add a new TaskState, use CompactionTaskStatus instead. If it doesn't contain any info, let's add it there.

server/src/main/java/org/apache/druid/server/compaction/CompactionCandidate.java

server/src/main/java/org/apache/druid/server/compaction/CompactionCandidateSearchPolicy.java

kfaraz · 2026-02-11T20:01:48Z

...r/src/main/java/org/apache/druid/server/compaction/DataSourceCompactibleSegmentIterator.java

-            CompactionCandidate candidatesWithStatus = CompactionCandidate
-                .from(partialEternitySegments, null)
-                .withCurrentStatus(CompactionStatus.skipped("Segments have partial-eternity intervals"));
+            CompactionCandidate candidatesWithStatus =


This class probably doesn't need to change in this PR.
The Eligibility should be computed and used only by the CompactionJobQueue (and maybe CompactSegments for backward compatibility).

server/src/main/java/org/apache/druid/server/compaction/CompactionStatusTracker.java

…ecemei

…ecemei) (#19054) * Add CompactionMode to compaction Eligibility * Deprecate old stuff * Clean up comment * Javadocs update * Fix test * revert job queue change * Eligibility --------- Co-authored-by: cecemei <yingqian.mei@gmail.com>

indexing-service/src/main/java/org/apache/druid/indexing/compact/CompactionJobQueue.java

+      return false;
+    }
+    // Check if the job is already running or succeeded
+    final TaskState candidateState = statusTracker.computeCompactionTaskState(job.getCandidate().getCandidate());


server/src/main/java/org/apache/druid/server/coordinator/duty/CompactSegments.java

-      } else {
-        // As these segments will be compacted, we will aggregate the statistic to the Compacted statistics
-        snapshotBuilder.addToComplete(entry);
+      final TaskState compactionTaskState = statusTracker.computeCompactionTaskState(candidate);


server/src/test/java/org/apache/druid/server/compaction/CompactionStatusTrackerTest.java

-        = statusTracker.computeCompactionStatus(candidateSegments, policy);
-    Assert.assertEquals(CompactionStatus.State.PENDING, status.getState());
-    Assert.assertEquals("Not compacted yet", status.getReason());
+    TaskState status = statusTracker.computeCompactionTaskState(candidateSegments.getCandidate());


server/src/test/java/org/apache/druid/server/compaction/CompactionStatusTrackerTest.java

-        "Segment timeline not updated since last compaction task succeeded",
-        status.getReason()
-    );
+    status = statusTracker.computeCompactionTaskState(candidateSegments.getCandidate());


server/src/test/java/org/apache/druid/server/compaction/CompactionStatusTrackerTest.java

    statusTracker.onSegmentTimelineUpdated(DateTimes.nowUtc());
-    status = statusTracker.computeCompactionStatus(candidateSegments, policy);
-    Assert.assertEquals(CompactionStatus.State.PENDING, status.getState());
+    status = statusTracker.computeCompactionTaskState(candidateSegments.getCandidate());


wip

35dad44

github-actions bot added the Area - Ingestion label Jan 30, 2026

cecemei added 2 commits February 1, 2026 13:05

test

0a4a0df

eligibility

ca0bc0c

cecemei marked this pull request as ready for review February 2, 2026 06:11

cecemei changed the title ~~wip~~ Add compaction eligibility concept and incremental compaction support Feb 2, 2026

Double.compare

ab79904

kfaraz reviewed Feb 4, 2026

View reviewed changes

cecemei added 4 commits February 4, 2026 15:27

CompactionEligibility

08d70b8

candidate

5aba7d9

format

5001d31

policy-status

e38257b

kfaraz reviewed Feb 5, 2026

View reviewed changes

server/src/main/java/org/apache/druid/server/coordinator/duty/CompactSegments.java Show resolved Hide resolved

cecemei added 3 commits February 5, 2026 16:34

eligible

502854d

revert drop existing change

3ec2d34

javadoc

f9f1bc9

github-advanced-security bot found potential problems Feb 6, 2026

View reviewed changes

test

5bfcf0c

capistrant reviewed Feb 6, 2026

View reviewed changes

cecemei added 2 commits February 7, 2026 00:33

Merge branch 'master' into compact3

5487f8e

compaction mode

a39a35e

github-advanced-security bot found potential problems Feb 7, 2026

View reviewed changes

server/src/main/java/org/apache/druid/server/compaction/CompactionStatusTracker.java Fixed Show fixed Hide fixed

cecemei added 4 commits February 7, 2026 03:58

format

aa6a5ed

revert incremental compaction change

eb8b118

revert incremental compaction change 2

599090c

fix bug

ac6d147

format

e44fb34

revert incremental compaction change 3

55405ef

cecemei mentioned this pull request Feb 9, 2026

add incremental compaction support #18996

Draft

10 tasks

capistrant reviewed Feb 10, 2026

View reviewed changes

kfaraz requested changes Feb 11, 2026

View reviewed changes

kfaraz reviewed Feb 11, 2026

View reviewed changes

server/src/main/java/org/apache/druid/server/compaction/CompactionStatusTracker.java Show resolved Hide resolved

GWphua mentioned this pull request Feb 13, 2026

Minor compaction #19016

Open

9 tasks

cecemei added 6 commits February 24, 2026 12:24

Merge branch 'master' into compact3

851d8ae

javadoc

5eb1e4b

build & test

d6df4b8

revert CompactionCandidate.TaskState

51512f4

CompactionCandidateAndStatus

7030709

null not allowed for serialization

249f867

github-advanced-security bot found potential problems Feb 25, 2026

View reviewed changes

server/src/main/java/org/apache/druid/server/compaction/CompactionStatusTracker.java Fixed Show fixed Hide fixed

cecemei added 3 commits February 25, 2026 00:48

style

360559e

style

a424bf8

strict-compile

2c29a1b

cecemei requested a review from kfaraz February 25, 2026 10:20

kfaraz mentioned this pull request Feb 25, 2026

Add CompactionMode to compaction Eligibility (split from #18968 by @cecemei) #19054

Merged

10 tasks

Merge branch 'master' into compact3

5358209

github-advanced-security bot found potential problems Feb 26, 2026

View reviewed changes

kfaraz mentioned this pull request Feb 26, 2026

Refactor: Ensure that CompactionCandidate.currentStatus is non-null #19058

Open

10 tasks

Conversation

cecemei commented Jan 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Key Changes

1. Introduced CompactionMode Enum

2. Refactored CompactionCandidate

3. Updated CompactionStatus

4. Simplified CompactionCandidateSearchPolicy

5. Improved Task State Tracking

Code Flow Example

Testing

Uh oh!

kfaraz left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

kfaraz Feb 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kfaraz commented Feb 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kfaraz left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cecemei commented Feb 6, 2026

Uh oh!

capistrant Feb 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kfaraz commented Feb 6, 2026

Uh oh!

Uh oh!

cecemei commented Feb 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kfaraz left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cecemei commented Jan 30, 2026 •

edited

Loading

1. Introduced `CompactionMode` Enum

2. Refactored `CompactionCandidate`

3. Updated `CompactionStatus`

4. Simplified `CompactionCandidateSearchPolicy`

kfaraz Feb 4, 2026 •

edited

Loading

kfaraz commented Feb 4, 2026 •

edited

Loading

capistrant Feb 6, 2026 •

edited

Loading

cecemei commented Feb 8, 2026 •

edited

Loading

kfaraz Feb 11, 2026 •

edited

Loading