[BUG] Observability plugin's system index creating is impacting on OpenSearch node starts

[zane-neo]

What is the bug?
User reported an issue that OpenSearch cluster is not able to spin up even if the issue in the log has been fixed: opensearch-project/OpenSearch#14791, after investigation this is an issue caused by observability's system index creation: opensearch-project/OpenSearch#14791 (comment).

@andrross has concern on the PR to fix this issue: opensearch-project/OpenSearch#15258, so this issue is to collect more details:

1. Is this system index a critical one and if it's not created, we should block the cluster from spinning up?
2. Is it possible to make the index creation a lazy operation until the time it's been used in plugin?
   Also please chime in the discussion in PR if you have more insights on this issue.

How can one reproduce the bug?
Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

What is the expected behavior?
A clear and concise description of what you expected to happen.

What is your host/environment?

• OS: [e.g. iOS]
• Version [e.g. 22]
• Plugins

Do you have any screenshots?
If applicable, add screenshots to help explain your problem.

Do you have any additional context?
Add any other context about the problem.

[dbwiddis]

IMO failing to start up a node on a plugin index creation failure seems very wrong. See opensearch-project/OpenSearch#16340 for a bug that could fatally disable a cluster because of this choice.

[dblock]

[Catch All Triage - 1, 2]