Fix denial of service caused by a 404 from elasticsearch to managed logstashes

[zwsyscall]

This PR fixes a bug caused by #17052 where a single 404 return from elasticsearch can lead to a prolonged denial of service in Logstash as its pipelines are removed.
Returns @cached_pipelines.

• Closes Unexpected pipeline shutdown with central-managed configurations #17980
• Relates CPM handle 404 response gracefully with user-friendly log #17052

[cla-checker-service]

💚 CLA has been signed

[github-actions]

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

• run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

[mergify]

This pull request does not have a backport label. Could you fix it @zwsyscall? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit.
• If no backport is necessary, please add the backport-skip label

[zwsyscall]

This should be backported to 8.17, 8.18, 8.19, and 9.0, I can't add the tags myself.

[yaauie]

@zwsyscall thank you for your contribution -- it was helpful in chasing down the root issue.

Unfortunately, the Elasticsearch endpoint emits a 404 when search emits no results, and when someone deletes all of the pipeline definitions they expect an empty list to cause the pipelines to shut down.

We have decided to handle this a bit differently in #18265; when the ElasticsearchSource gets a 404 while using the multi-pipeline GET for server-side filtering, we will fall through to a second get-all and do classic client-side filtering. This allows us to "confirm" the 404 and get the desired behaviour when the list is legitimately empty, while preventing spurious 404's with other causes from causing the pipelines to be removed.