Adding support for multi-tier rollups in ISM

[tandonks]

Description

This PR implements multi-tier rollup functionality in OpenSearch Index Management, enabling hierarchical data aggregations where rollup indices can serve as source indices for subsequent rollup operations. This allows for progressive data summarization (e.g., raw data → 1-minute → 5-minute → 10-minute intervals) within a single ISM policy or through chained rollup jobs.

Previously, the ISM Rollup action was a terminal operation that only supported rolling up raw indices. This limitation forced users to rely on complex external automation chains (Raw Data → Rollup → _reindex → Second Rollup Job) or redundantly roll up raw data multiple times at different granularities, leading to:

• Operational overhead maintaining chained reindex and rollup jobs
• Compute inefficiency from reprocessing the same raw data repeatedly
• Lack of automation in retention policies
• Data management complexity and inconsistency risks

Solution Overview

This implementation extends the rollup framework to support multi-level rollups by:

1. Smart Initial Timestamp Computation: Automatically detects rollup indices and fetches the earliest timestamp from the date_histogram field instead of scanning raw documents
2. Consistent Metric and Field Naming: Maps user-specified raw field names to aggregated field names in source rollup indices (e.g., passenger_count → passenger_count.sum)
3. Source Index Field Support: Added optional source_index field to ISMRollup schema, enabling explicit source specification for multi-tier chaining
4. Template Variable Resolution: Supports Mustache templates ({{ctx.index}}, {{ctx.source_index}}) for dynamic index naming in ISM policies
5. Interval Compatibility Enforcement: Validates that target intervals are exact multiples of source intervals (e.g., 1m → 5m → 10m)
6. Metric Availability Validation: Ensures target rollups only request metrics available in source rollups

Example Policy

{
  "policy": {
    "description": "Multi-tier rollup: 1m → 5m → 10m",
    "default_state": "rollup_1m",
    "states": [
      {
        "name": "rollup_1m",
        "actions": [
          {
            "rollup": {
              "ism_rollup": {
                "description": "Rollup raw data into 1-minute intervals",
                "target_index": "my_index_rollup_1m-{{ctx.index}}",
                "page_size": 100,
                "dimensions": [
                  {
                    "date_histogram": {
                      "source_field": "timestamp",
                      "fixed_interval": "1m",
                      "timezone": "UTC"
                    }
                  },
                  {
                    "terms": {
                      "source_field": "category"
                    }
                  }
                ],
                "metrics": [
                  {
                    "source_field": "value",
                    "metrics": [
                      { "sum": {} },
                      { "min": {} },
                      { "max": {} },
                      { "value_count": {} },
                      { "avg": {} }
                    ]
                  }
                ]
              }
            }
          }
        ],
        "transitions": [
          {
            "state_name": "rollup_5m"
          }
        ]
      },
      {
        "name": "rollup_5m",
        "actions": [
          {
            "rollup": {
              "ism_rollup": {
                "description": "Rollup 1m data into 5-minute intervals",
                "source_index": "my_index_rollup_1m-{{ctx.index}}",
                "target_index": "my_index_rollup_5m-{{ctx.index}}",
                "page_size": 100,
                "dimensions": [
                  {
                    "date_histogram": {
                      "source_field": "timestamp",
                      "fixed_interval": "5m",
                      "timezone": "UTC"
                    }
                  },
                  {
                    "terms": {
                      "source_field": "category"
                    }
                  }
                ],
                "metrics": [
                  {
                    "source_field": "value",
                    "metrics": [
                      { "sum": {} },
                      { "min": {} },
                      { "max": {} },
                      { "value_count": {} },
                      { "avg": {} }
                    ]
                  }
                ]
              }
            }
          }
        ],
        "transitions": [
          {
            "state_name": "rollup_10m"
          }
        ]
      },
      {
        "name": "rollup_10m",
        "actions": [
          {
            "rollup": {
              "ism_rollup": {
                "description": "Rollup 5m data into 10-minute intervals",
                "source_index": "my_index_rollup_5m-{{ctx.index}}",
                "target_index": "my_index_rollup_10m-{{ctx.index}}",
                "page_size": 100,
                "dimensions": [
                  {
                    "date_histogram": {
                      "source_field": "timestamp",
                      "fixed_interval": "10m",
                      "timezone": "UTC"
                    }
                  },
                  {
                    "terms": {
                      "source_field": "category"
                    }
                  }
                ],
                "metrics": [
                  {
                    "source_field": "value",
                    "metrics": [
                      { "sum": {} },
                      { "min": {} },
                      { "max": {} },
                      { "value_count": {} },
                      { "avg": {} }
                    ]
                  }
                ]
              }
            }
          }
        ],
        "transitions": []
      }
    ]
  }
}

Related Issues

Resolves #1490

Check List

• New functionality includes testing.
• New functionality has been documented.
• API changes companion pull request created.
• Commits are signed per the DCO using --signoff.
• Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[codecov]

Codecov Report

❌ Patch coverage is 83.56164% with 36 lines in your changes missing coverage. Please review.
✅ Project coverage is 76.21%. Comparing base (143bf54) to head (c266f5b).

Files with missing lines	Patch %	Lines
...nagement/step/rollup/AttemptCreateRollupJobStep.kt	64.10%	10 Missing and 4 partials ⚠️
...ch/indexmanagement/rollup/RollupMetadataService.kt	76.92%	8 Missing and 1 partial ⚠️
.../opensearch/indexmanagement/rollup/RollupRunner.kt	81.81%	3 Missing and 5 partials ⚠️
...management/rollup/interceptor/RollupInterceptor.kt	81.81%	0 Missing and 2 partials ⚠️
...nsearch/indexmanagement/rollup/util/RollupUtils.kt	96.15%	0 Missing and 2 partials ⚠️
.../rollup/util/RollupFieldValueExpressionResolver.kt	92.30%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1533      +/-   ##
==========================================
+ Coverage   76.12%   76.21%   +0.09%     
==========================================
  Files         375      375              
  Lines       17567    17761     +194     
  Branches     2410     2451      +41     
==========================================
+ Hits        13372    13536     +164     
- Misses       2953     2974      +21     
- Partials     1242     1251       +9     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.