Add Supervisor UI component to view timeline of rules applied by a cascading reindexing supervisor by capistrant · Pull Request #19051 · apache/druid

capistrant · 2026-02-25T01:57:18Z

Description

New Supervisor API (cascading reindexing supervisors only)

GET /druid/indexer/v1/supervisor/{supervisorId}/reindexingTimeline

This generates a timeline of search intervals for reindexing with the effective sets of rules that are used to create the underlying inline compaction configs for the different intervals. This is the business end of the console UI described below

Console visualization of reindexing timeline (cascading reindexing supervisors only)

For cascading compaction supervisors you can get a timeline view of how rules from the rule provider(s) will apply to your datasource. You can view how skip offsets (if they are configured) will change the timeline too. This visualization is a great help for operators in understanding in how their underlying rules will be applied across the datasource history. I will add some screen shots for reference

Refactor `CompactionSupervisorTest` Embedded Tests

Split out the cascading template tests to their own class from the inline compaction supervisor tests. Sharing one file was getting convoluted. They share a base class for what makes sense.

Release note

Key changed/added classes in this PR

CascadingReindexingTemplate
ReindexingTimelineView
SupervisorResource
web-console/src/components/reindexing-timeline/reindexing-timeline.scss

This PR has:

been self-reviewed.
added documentation for new or modified features or behaviors.
a release note entry in the PR description.
added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
added or updated version, license, or notice information in licenses.yaml
added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
added integration tests.
been tested in a test Druid cluster.

…code" This reverts commit 0d0d0c5.

This reverts commit 3f2747c.

…for new endpoint

…upervisor timeline api

...service/src/test/java/org/apache/druid/indexing/compact/CascadingReindexingTemplateTest.java

kfaraz

Pretty cool!
Left some minor suggestions.

Will try this out locally.
Would also be nice to get a review from @vogievetsky on the web-console stuff.

kfaraz · 2026-02-25T04:13:04Z

indexing-service/src/main/java/org/apache/druid/indexing/compact/ReindexingTimelineView.java

+    private final AppliedSkipOffset applied;
+    private final NotAppliedSkipOffset notApplied;


Nit: These two classes feel a little unnecessary. We could have just have the actual fields in the class SkipOffsetInfo directly: type, period, a boolean for isApplied, a nullable effectiveEndTime mutually exclusive with a nullable errorMessage.

...-service/src/main/java/org/apache/druid/indexing/overlord/supervisor/SupervisorResource.java

kfaraz · 2026-02-25T04:21:08Z

...service/src/test/java/org/apache/druid/indexing/compact/CascadingReindexingTemplateTest.java

+        "tuning-7d",
+        null,
+        Period.days(7),
+        new UserCompactionTaskQueryTuningConfig(


Didn't we add a builder for this class in the previous PR?

no, it is coming in #18996 though

kfaraz · 2026-02-25T04:22:33Z

...service/src/test/java/org/apache/druid/indexing/compact/CascadingReindexingTemplateTest.java

+  /**
+   * Test validation error when granularity timeline is invalid
+   */


Not really needed since the test name captures this already.

kfaraz · 2026-02-25T04:22:50Z

...service/src/test/java/org/apache/druid/indexing/compact/CascadingReindexingTemplateTest.java

+  /**
+   * Test that skipOffsetFromNow correctly skips intervals and populates skipOffset.applied
+   */


Not really needed as the test name already captures this.

kfaraz · 2026-02-25T04:23:39Z

...service/src/test/java/org/apache/druid/indexing/compact/CascadingReindexingTemplateTest.java

+    // Older data (P90D) has DAY granularity, newer data (P30D) has HOUR granularity
+    // This means as we move from past to present, granularity gets finer (valid)
+    // But then if we add MONTH for recent data, it becomes coarser (invalid)
+    ReindexingRuleProvider provider = InlineReindexingRuleProvider.builder()


Style tip: Move the .builder() to the next line to make the whole thing feel less crowded.

Suggested change

ReindexingRuleProvider provider = InlineReindexingRuleProvider.builder()

ReindexingRuleProvider provider = InlineReindexingRuleProvider

.builder()

kfaraz · 2026-02-25T04:29:35Z

...ing-service/src/main/java/org/apache/druid/indexing/compact/CascadingReindexingTemplate.java

+   * @param referenceTime the reference time to use for computing rule periods (typically DateTime.now())
+   * @return a view of the reindexing timeline with intervals and their configs
+   */
+  public ReindexingTimelineView getReindexingTimelineView(DateTime referenceTime)


I feel like this method should not live here as it kind of pollutes the template definition.

Either have the timeline be built using a static method like ReindexingTimelineView.fromTemplate(template, referenceTime) or have a separate builder class altogether.

hmm. I wonder about how we'd manage to re-use a lot of the underlying private implementation details that are quite convoluted (unfortunately). Like creating the base timeline from segment gran rules and what not is all shared between the timeline creation and the task creation. I agree that it feels a little weird here, but I didn't know how else to approach it without either duping a lot of internal logic or exposing it to both somehow. will keep thinking though

kfaraz · 2026-02-25T04:31:24Z

@capistrant , I took the liberty to slightly update the title for clarity. Hope it seems less ambiguous now.

kfaraz · 2026-02-25T07:24:09Z

Was trying out the embedded test in this PR to check out the timeline.

@capistrant , in the snapshot below, what does the "4 rules applied" mean?
The widget below it shows only 2 items - a segment granularity and a deletion rule.

Maybe we should include an info popup for each of the chips.
Also, the chip says `"1" deletion rule". When can there be more than 1 for a single interval?

Co-authored-by: Kashif Faraz <kashif.faraz@gmail.com>

...-service/src/main/java/org/apache/druid/indexing/overlord/supervisor/SupervisorResource.java

capistrant · 2026-02-25T16:40:03Z

Was trying out the embedded test in this PR to check out the timeline.

@capistrant , in the snapshot below, what does the "4 rules applied" mean? The widget below it shows only 2 items - a segment granularity and a deletion rule.

Maybe we should include an info popup for each of the chips. Also, the chip says `"1" deletion rule". When can there be more than 1 for a single interval?

4 rules applied means that the config created for the interval is made up of 4 different reindexing rules. In this case based on the test I assume you are running, it is

segmentGran rule
deletion rule
ioConfig rule
tuningConfig rule

You could confirm by clicking the view raw rules button.

I think this is a UX issue. Hand up, I don't have any experience curating a good UX :). If I had to guess what UX decision is best here, it is to remove the N rules applied chip. if someone really wants to see the explicit underlying rules they can click view raw rules.

As for deletion rules, N deletion rules can apply to any given interval. They accumulate (unlike other rule types). Every deletion rule that applies a search interval will be combined together.

deleteWhere "countryName=france" olderThan P7D"
deleteWhere "countryName=italy" olderThan P14D"

Any interval where both of those apply would have 2 deletion rules.

The chips are intended to give very quick reference to what is in the config for the given search interval. They are not strictly tied to rules even. In my pictures you see metrics and dims in separate chips, but both belong to the same rule. So maybe the deletion rules chip should actually be like 2 deletion filters or 2 deletion clauses. I didn't know how valuable chips would be for tuningConfig and ioConfig so those are not surfaced, likely leading to your initial confusion. You'd have to view the config payload or the raw rules to see them.

…the same

capistrant added 8 commits February 24, 2026 10:30

Revert "Remove support for the reindexing timeline creation from app …

ce4a48e

…code" This reverts commit 0d0d0c5.

Revert "Remove web-console changes from this PR"

b1a5854

This reverts commit 3f2747c.

fixup tests

3144518

Improve web console code

b610bd0

refactor some reindexing timeline code and update supervisor api doc …

d14ca99

…for new endpoint

Refactor compaction supervisor embedded tests. create new tests for s…

76d01bb

…upervisor timeline api

refactorings

dd615fc

Refactoring the web console a bit

49dd1f6

github-actions bot added Area - Documentation Area - Web Console Area - Ingestion labels Feb 25, 2026

capistrant changed the title ~~Reindexing timeline UI~~ Cascading Reindexing Supervisor UI component fore viewing timeline of rule applications Feb 25, 2026

github-advanced-security bot found potential problems Feb 25, 2026

View reviewed changes

kfaraz reviewed Feb 25, 2026

View reviewed changes

kfaraz changed the title ~~Cascading Reindexing Supervisor UI component fore viewing timeline of rule applications~~ Cascading Reindexing Supervisor UI component for viewing timeline of rule applications Feb 25, 2026

kfaraz changed the title ~~Cascading Reindexing Supervisor UI component for viewing timeline of rule applications~~ Supervisor UI component to view timeline of rules applied for a cascading reindexing supervisor Feb 25, 2026

kfaraz changed the title ~~Supervisor UI component to view timeline of rules applied for a cascading reindexing supervisor~~ Add Supervisor UI component to view timeline of rules applied by a cascading reindexing supervisor Feb 25, 2026

capistrant and others added 3 commits February 25, 2026 09:35

Apply suggestions from code review

984371a

Co-authored-by: Kashif Faraz <kashif.faraz@gmail.com>

fixup formatting and some linter checks

0b1ae3c

Some refactoring of the timeline view around skip offsets

e975bbd

github-advanced-security bot found potential problems Feb 25, 2026

View reviewed changes

capistrant added 3 commits February 25, 2026 10:43

fix some string formatting issues

52e016b

fix UTs

9a6fe4a

Add a serde test for the new timeline view and fix a jackson bug for …

b2daa0e

…the same

		private final AppliedSkipOffset applied;
		private final NotAppliedSkipOffset notApplied;

	ReindexingRuleProvider provider = InlineReindexingRuleProvider.builder()
	ReindexingRuleProvider provider = InlineReindexingRuleProvider
	.builder()

Conversation

capistrant commented Feb 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

New Supervisor API (cascading reindexing supervisors only)

Console visualization of reindexing timeline (cascading reindexing supervisors only)

Refactor CompactionSupervisorTest Embedded Tests

Release note

Key changed/added classes in this PR

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kfaraz left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kfaraz commented Feb 25, 2026

Uh oh!

kfaraz commented Feb 25, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

capistrant commented Feb 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

capistrant commented Feb 25, 2026 •

edited

Loading

Refactor `CompactionSupervisorTest` Embedded Tests