Support different downsampling methods through ILM

[gmarouli]

Following #136813, we expose to ILM the new sampling method config in the downsampling API. This will allow users to configure the sampling method in their downsample action of their ILM policies. For example:

PUT _ilm/policy/datastream_policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_docs": 1
          },
          "downsample": {
  	          "fixed_interval": "1h",
  	          "force_merge_index": false,
                  "sampling_method": "aggregate"
  	  }
        }
      }
    }
  }
}

[github-actions]

🔍 Preview links for changed docs

• docs/reference/elasticsearch/index-lifecycle-actions/ilm-downsample.md

[github-actions]

ℹ️ Important: Docs version tagging

👋 Thanks for updating the docs! Just a friendly reminder that our docs are now cumulative. This means all 9.x versions are documented on the same page and published off of the main branch, instead of creating separate pages for each minor version.

We use applies_to tags to mark version-specific features and changes.

Expand for a quick overview

When to use applies_to tags:

✅ At the page level to indicate which products/deployments the content applies to (mandatory)
✅ When features change state (e.g. preview, ga) in a specific version
✅ When availability differs across deployments and environments

What NOT to do:

❌ Don't remove or replace information that applies to an older version
❌ Don't add new information that applies to a specific version without an applies_to tag
❌ Don't forget that applies_to tags can be used at the page, section, and inline level

🤔 Need help?

• Check out the cumulative docs guidelines
• Reach out in the #docs Slack channel

[elasticsearchmachine]

Hi @gmarouli, I've created a changelog YAML for you.

[elasticsearchmachine]

Pinging @elastic/es-storage-engine (Team:StorageEngine)

[elasticsearchmachine]

Pinging @elastic/es-data-management (Team:Data Management)

[nielsbauman]

Thanks a lot for working on this, Mary! Looking good to me, I just have a few pretty minor comments.

Who would have thought that a new functionality would require fewer changes in ILM than in DLM 😄

[nielsbauman]

Should we link to a page that explains the difference between the two methods? Or do we not have such a page yet?

[gmarouli]

We do not have a page about this yet. But this is a very good point and I will keep a note about that.

[nielsbauman]

Small optional nit:

Suggested change

	return switch (between(0, 2)) {
	case 0 -> null;
	case 1 -> DownsampleConfig.SamplingMethod.AGGREGATE;
	case 2 -> DownsampleConfig.SamplingMethod.LAST_VALUE;
	default -> throw new IllegalArgumentException("Unknown randomisation branch");
	};
	return randomBoolean() ? null : randomFrom(DownsampleConfig.SamplingMethod.values());

[gmarouli]

I used to have it like this but it favours null, I will change it though to something fair that still picks up the values dynamically. Nice that you challenged it.

I am considering something along the lines of:

static DownsampleConfig.SamplingMethod randomSamplingMethod() {
        if (between(0, DownsampleConfig.SamplingMethod.values().length) == 0) {
            return null;
        } else {
            return randomFrom(DownsampleConfig.SamplingMethod.values());
        }
    }

[gmarouli]

Who would have thought that a new functionality would require fewer changes in ILM than in DLM 😄

Hahah, actually, if we did the same in DLM just adding it to the each round, it required almost nothing. But the users would need to pay the price. So, I still think DLM wins, it's just unfortunate that we did not see this when we were first coming up with the configuration.

[nielsbauman]

I understand this comment, but I think it might be hard to understand without the context of this PR.

Also, wouldn't it be valuable to have a similar test that explicitly changes the sampling method between the two phases? Now we sometimes implicitly do that through the default value (which is what your comment hints at), but I feel like that's an important edge case that is worth explicitly verifying. What do you think?

[gmarouli]

Yes you are right, initially, I thought the sampling method should not matter, but as you say it's good to check that the test case captures this. I changed it to always change the sampling method

[nielsbauman]

Thanks. I think we should also update this comment as Sometimes is no longer correct, right?

[nielsbauman]

LGTM! Thanks a lot for the work and the iterations :)

[gmarouli]

Thanks for the review @nielsbauman . Very helpful!