Data lifecycle docs: Automate rollover tutorial

[kilfoyle]

Automate rollover tutorial (see 25:00): Manage time series data with data streams

• Apparently Search customers are using this to move their Hot content to Warm, etc., so it's not just for time series data
• Screenshots should show advanced settings.
• Explain why the current screenshot shows “Move data into phase when” is set to zero days.
• Explain that the time is calculated from after last rollover!
• Create index template section, screenshot shows all empty values.
• We should at least explain those (or provide links to what they are).
• All examples show Dev Tools, but it'd be better to have tabs showing both UI and Dev Tools.
• Customers using index management will sometimes change the lifecycle policy on a backing index, thinking that the new policy will apply to the next write index when it's created. But rather, the next index will take the policy defined by the index template.
  If they want to customize the index lifecycle policy for a specific data set they should refer to “Customize built-in policies” tutorial.

[yetanothertw]

Thank you for updating the screenshot here, David! :)

I've marked the first task as completed as well because I've reviewed the feedback video again and I think our Managing general content with data streams tutorial covers that use case. From that tutorial we also link to Configure a lifecycle policy for more general information that describe how to create a policy different from the one in the tutorial itself (for different purposes).

[yetanothertw]

The current screenshot doesn't show that “Move data into phase when” is set to zero days -- and I don't think that's necessary. However, I have mentioned that the time is calculated from the last rollover when the data moves phases (in the snippet for creating a lifecycle policy).

[yetanothertw]

Looking at ways to add UI steps to the rollover general content tutorial

  {
  "lifecycle": {
    "name": "indextods"
  },
  "default_pipeline": "ingest_time_1"
}

[yetanothertw]

{
"_source": {
  "excludes": [],
  "includes": [],
  "enabled": true
  },
  "_routing": {
    "required": false
    },
  "dynamic": true,
  "numeric_detection": false,
  "date_detection": true,
  "dynamic_date_formats": [
    "strict_date_optional_time",
    "yyyy/MM/dd HH:mm:ss Z||yyyy/MM/dd Z"
  ]
}

Fixed this error msg:

[yetanothertw]

I've tried to replicate some of the steps for creating an ingest pipeline, creating a lifecycle policy, and configuring an index template in the UI, but the example that is generated after I create the template doesn't correspond to the one in the API call example. I've tried various options/steps and this is what I get:

{
  "template": {
    "settings": {
      "index": {
        "lifecycle": {
          "name": "indextods"
        },
        "mode": "standard",
        "routing": {
          "allocation": {
            "include": {
              "_tier_preference": "data_hot"
            }
          }
        },
        "default_pipeline": "ingest_time_1"
      }
    },
    "mappings": {
      "dynamic": "true",
      "_data_stream_timestamp": {
        "enabled": true
      },
      "dynamic_date_formats": [
        "strict_date_optional_time",
        "yyyy/MM/dd HH:mm:ss Z||yyyy/MM/dd Z"
      ],
      "dynamic_templates": [],
      "date_detection": true,
      "numeric_detection": false,
      "properties": {
        "@timestamp": {
          "type": "date"
        }
      }
    },
    "aliases": {}
  }
}

The amount of steps and screenshots required to provide an equivalent experience via the UI is not feasible. I'll make some changes, but I don't think a 1:1 equivalence is worth pursuing. (As a user, I'd be intimidated by the need to perform 6 steps in the UI when I could just use one API call instead.)

[shainaraskas]

looking at your comments above, I wonder why we even offer the UI if it doesn't simplify things for users.

1. I question whether we need screenshots for the UI instructions (esp when there are many screens with low information density)
2. how far off is the created ingest pipeline index template from the one in the tutorial? is it substantively different from a narrative pov? would we consider changing the put to something similar, where they do differ?

at minimum, we should call out that there is a UI for this, and if we have instructions on creating index templates, we can link to those if we don't feel like walking them through using the tool in this tutorial. consider in the narrative just pointing out the "key" settings that make the template fit for purpose.