Curator - Connection error - 2.3.220

[presianbg]

Hi,

After upgrading our distributed deployment (2.3.200 -> 2.3.220) , the search nodes hit the watermark disk usage and stopped processing the data.

Looks like the curator is not able to access the elasticsearch (cross cluster search), this is the log from one of the search nodes:

2023-03-01 08:05:28,973 INFO      GET http://127.0.0.1:9200/ [status:N/A duration:0.000s]
2023-03-01 08:05:28,973 WARNING   Node <Urllib3HttpNode(http://127.0.0.1:9200)> has failed for 3 times in a row, putting on 4 second timeout
2023-03-01 08:05:28,973 WARNING   Retrying request after failure (attempt 2 of 3)
Traceback (most recent call last):
  File "/usr/lib/python3.10/site-packages/elastic_transport/_transport.py", line 329, in perform_request
    meta, raw_data = node.perform_request(
  File "/usr/lib/python3.10/site-packages/elastic_transport/_node/_http_urllib3.py", line 199, in perform_request
    raise err from None
elastic_transport.ConnectionError: Connection error caused by: NewConnectionError(<urllib3.connection.HTTPConnection object at 0x7fb7bfc36f50>: Failed to establish a new connection: [Errno 111] Connection refused)
2023-03-01 08:05:28,974 INFO      GET http://127.0.0.1:9200/ [status:N/A duration:0.000s]
2023-03-01 08:05:28,974 WARNING   Node <Urllib3HttpNode(http://127.0.0.1:9200)> has failed for 4 times in a row, putting on 8 second timeout
2023-03-01 08:05:28,974 CRITICAL  Exception encountered: Connection error caused by: ConnectionError(Connection error caused by: NewConnectionError(<urllib3.connection.HTTPConnection object at 0x7fb7bfc36e60>: Failed to establish a new connection: [Errno 111] Connection refused))

The cluster seems to be working:

so-elasticsearch-query _cluster/health?pretty
{
  "cluster_name" : "so-search5",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 186,
  "active_shards" : 186,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

Tried with rebooting the machine to no avail.

Any guidance how to fix it is welcomed!

Cheers,
PY

[presianbg]

I was able to delete some of the old indices with:

so-elasticsearch-query so-index-name -XDELETE

But still baffled why curator is trying to communicate with elastic via http , shouldn't be https ?

[dougburks]

We're looking into this now.

[dougburks]

We've confirmed this issue and are working on a hotfix.

[dougburks]

We've published a hotfix:
https://blog.securityonion.net/2023/03/security-onion-23220-hotfix-20230301.html