[BUG] Creating index-patterns With CCS in Dashboard causes search threadPool to be exhausted in remote cluster

[kkewwei]

Describe the bug

As mentioned in opensearch-project/OpenSearch-Dashboards#8596

When I set the ccs in the cluster settings(the address is pointed to itself to reproduce)

PUT _cluster/settings
{
  "persistent": {
      "cluster": {
         "remote": {
            "ccs1": {
               "mode": "proxy",
               "proxy_address": "127.0.0.1:9300"
            }
         }
      }
  }
}

Creating a new index-patterns in the dashboard, I find that search thread pool is immediately exhausted in the target cluster:

Dashboard will send such request to target cluster, which cause search threadPool to be exhausted in remote cluster.

POST *:*/_search
{
  "size": 0,
  "timeout": "30000ms",
  "query": {
    "match_all": {
      "boost": 1
    }
  },
  "track_total_hits": 2147483647,
  "aggregations": {
    "indices": {
      "terms": {
        "field": "_index",
        "size": 200,
        "min_doc_count": 1,
        "shard_min_doc_count": 0,
        "show_term_doc_count_error": false,
        "order": [
          {
            "_count": "desc"
          },
          {
            "_key": "asc"
          }
        ]
      }
    }
  }
}

Same issue: opensearch-project/index-management-dashboards-plugin#1178

Related component

Search:Remote Search

To Reproduce

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior

There seems no good way to obtain the index information about the target cluster. As we just want to get the count of indices in the target cluster, if we should add new API in the core code to support?

GET {target_cluster}:*/_stats?metric=docs

Additional Details

Plugins
Please list all plugins currently enabled.

Screenshots
If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

• OS: [e.g. iOS]
• Version [e.g. 22]

Additional context
Add any other context about the problem here.

[sandeshkr419]

Hi @kkewwei - thanks for reporting this.
What is the behavior when not directing it to "proxy_address": "127.0.0.1:9300" and an actual cross-cluster?

[kkewwei]

Hi @kkewwei - thanks for reporting this. What is the behavior when not directing it to "proxy_address": "127.0.0.1:9300" and an actual cross-cluster?

@sandeshkr419 It's same. I have added the ccs for
GET {index_name}/_stats->GET {target_cluster}:*/_stats? in my here. If it's ok, I will pull the PR for it.

[kkewwei]

To provide an update on our progress: The Dashboard currently uses the :/_search endpoint to retrieve indices from CCS clusters, a compatibility measure for Elasticsearch versions prior to 7.10. For later versions, the RestResolveIndexAction interface is supported and can fully replace the :/_search endpoint.

This means the :/_search endpoint can be removed from the Dashboard code without impacting functionality in this scenario.

[kkewwei]

Will solved by opensearch-project/OpenSearch-Dashboards#10816