[BUG] AT: upgrade from previous release failed as reconciliation isn't completed due to connection exception

[andrewluckyguy]

Describe the bug
The following condition is observed in opensearchservices.netcracker.com after ~15 min of status polling:

  {
    "lastTransitionTime": "2025-12-31 13:49:30.313270895 +0000 UTC m=+326.286004703",
    "message": "Reconciliation cycle started",
    "reason": "ReconcileCycleStatus",
    "status": "False",
    "type": "In progress"
  }

To Reproduce
Steps to reproduce the behavior:

1. Go to Actions and run Opensearch Pipeline
2. Pay attention to job 'Opensearch-Pipeline / Clean [PREVIOUS]. Upgrade to [CURRENT]'
3. Scroll down to 'Verify Opensearch upgrade' stage
4. Download logs
   There are some connection errors:

[2025-12-31T10:22:54,426][WARN ][o.o.c.NodeConnectionsService] [opensearch-2] failed to connect to {opensearch-0}{dWGel94qRnaI57RJIebHCA}{EVdaHMp7Te2AzKE3sr8u9Q}{10.244.2.6}{10.244.2.6:9300}{dimr}{shard_indexing_pressure_enabled=true} (tried [1] times)
org.opensearch.transport.ConnectTransportException: [opensearch-0][10.244.2.6:9300] connect_exception
...
[2025-12-31T10:22:54,719][INFO ][o.o.c.c.FollowersChecker ] [opensearch-2] FollowerChecker{discoveryNode={opensearch-0}{dWGel94qRnaI57RJIebHCA}{EVdaHMp7Te2AzKE3sr8u9Q}{10.244.2.6}{10.244.2.6:9300}{dimr}{shard_indexing_pressure_enabled=true}, failureCountSinceLastSuccess=1, [cluster.fault_detection.follower_check.retry_count]=3} disconnected
org.opensearch.transport.NodeNotConnectedException: [opensearch-0][10.244.2.6:9300] Node not connected

Expected behavior
Reconciliation is completed with "status": "True"

Screenshots

Environment:
GitHub runners

Links to different runs:
Upgrade from 2.2.7
https://github.com/Netcracker/qubership-opensearch/actions/runs/20620156105
Upgrade from 2.1.1
https://github.com/Netcracker/qubership-opensearch/actions/runs/20616464310

[andrewluckyguy]

Added file with logs from all containers:
opensearch_log_from_containers.txt

[KryukovaPolina]

Tested on a local environment: clean install -> upgrade with a template, as in the job. Not reproducible