[Failure store] Introduce dedicated failure store lifecycle configuration

[gmarouli]

The failure store is a set of data stream indices that are used to store certain type of ingestion failures. Until this moment they were sharing the configuration of the backing indices. We understand that the two data sets have different lifecycle needs.

We believe that typically the failures will need to be retained much less than the data. Considering this we believe the lifecycle needs of the failures also more limited and they fit better the simplicity of the data stream lifecycle feature.

This allows the user to only set the desired retention and we will perform the rollover and other maintenance tasks without the user having to think about them. Furthermore, having only one lifecycle management feature allows us to ensure that these data is managed by default.

This PR introduces the following:

Configuration

We extend the failure store configuration to allow lifecycle configuration too, this configuration reflects the user's configuration only as shown below:

PUT _data_stream/*/options
{
  "failure_store": {
     "lifecycle": {
       "data_retention": "5d"
     }
  }
}

GET _data_stream/*/options

{
  "data_streams": [
    {
      "name": "my-ds",
      "options": {
        "failure_store": {
          "lifecycle": {
            "data_retention": "5d"
          }
        }
      }
    }
  ]
}

To retrieve the effective configuration you need to use the GET data streams API, see #126668

Functionality

• The data stream lifecycle (DLM) will manage the failure indices regardless if the failure store is enabled or not. This will ensure that if the failure store gets disabled we will not have stagnant data.
• The data stream options APIs reflect only the user's configuration.
• The GET data stream API should be used to check the current state of the effective failure store configuration.
  Telemetry

We extend the data stream failure store telemetry to also include the lifecycle telemetry.

{
  "data_streams": {
     "available": true,
     "enabled": true,
     "data_streams": 10,
     "indices_count": 50,
     "failure_store": {
       "explicitly_enabled_count": 1,
       "effectively_enabled_count": 15,
       "failure_indices_count": 30
       "lifecycle": { 
         "explicitly_enabled_count": 5,
         "effectively_enabled_count": 20,
         "data_retention": {
           "configured_data_streams": 5,
           "minimum_millis": X,
           "maximum_millis": Y,
           "average_millis": Z,
          },
          "effective_retention": {
            "retained_data_streams": 20,
            "minimum_millis": X,
            "maximum_millis": Y, 
            "average_millis": Z
          },
         "global_retention": {
           "max": {
             "defined": false
           },
           "default": {
             "defined": true,  <------ this is the default value applicable for the failure store
             "millis": X
           }
        }
      }
   }
}

Implementation details

• We ensure that partially reset failure store will create valid failure store configuration.
• We ensure that when a node communicates with a note with a previous version it will ensure it will not send an invalid failure store configuration enabled: null.

[elasticsearchmachine]

Pinging @elastic/es-data-management (Team:Data Management)

[elasticsearchmachine]

Hi @gmarouli, I've created a changelog YAML for you.

[jbaiera]

I have left a couple of non binding questions, but otherwise this LGTM!

[jbaiera]

This doesn't have the default failure store retention setting yet does it?

[gmarouli]

Not yet, it defaults to the global default right now. In the follow up PR it will only use the failures default. Do you think we should handle it differently?

[jbaiera]

Nah, follow up is perfectly fine!

[jbaiera]

Previously we weren't checking if the lifecycle was enabled, I'm assuming there isn't any important logic that follows this that needs executing if the data lifecycle configuration is present but disabled? Also, is present-but-disabled an invalid configuration state for the normal lifecycle?

[gmarouli]

Good questions.

• A lifecycle can be present by disabled, either for maintenance or if a user wants it disable for some reason.
• If a lifecycle is not present or if it's disabled, then the lifecycle service should not perform any operations for this data stream, so I think it makes sense to skip it if both the data and failures lifecycle are not enabled.
• I will double check the code to ensure the data lifecycle being null is handled properly, because this is something that could not happen before.
• We could keep just the null check here of course, it doesn't cost that much, but it creates this doubt that null and disabled behave differently, that's why I would prefer to include it here.

[gmarouli]

Confirmed, all actions check DataStream::isIndexManagedByDataStreamLifecycle() before executing any operation and this method handled a null lifecycle.

[jbaiera]

I'm so happy 😌

[jbaiera]

Was looking for where this was now used that it needed to be moved and I think I missed it. Where is this used now?

[gmarouli]

It's for serverless.

[elasticsearchmachine]

💔 Backport failed

Status	Branch	Result
❌	8.19	Commit could not be cherrypicked due to conflicts

You can use sqren/backport to manually backport by running backport --upstream elastic/elasticsearch --pr 127314

[gmarouli]

💚 All backports created successfully

Status	Branch	Result
✅	8.19

Questions ?

Please refer to the Backport tool documentation