Upgrade failure because opensearch is slow to start

[tomhughes]

Chef Server Version

15.9.38

Platform Details

Ubuntu 22.04

Configuration

Standalone

Scenario

Complete upgrade of chef server.

Steps to Reproduce

Run chef-server-ctl upgrade.

Expected Result

Upgrade completes.

Actual Result

Upgrade fails on "add internal user to opensearch security plugin" because opensearch is not responsive.

Analysis

There is already an attempt to address the problem in the form of

chef-server/omnibus/files/server-ctl-cookbooks/infra-server/recipes/opensearch.rb

Line 185 in 7940d04

chef_sleep 10

but at least on the machine we're using 10 seconds is nothing like enough for opensearch to finish starting and be ready to answer requests.

Bumping the sleep to 30 seconds while clearly not ideal as a solution seemed to fix things for us.

[AleksanderDyla]

I can also confirm the issue especially when running on smaller azure VMs. I made a workaround by running the sed -i 's/chef_sleep 10/chef_sleep 30/g' /opt/opscode/embedded/cookbooks/infra-server/recipes/opensearch.rb command which override the sleep to 30 sec before running chef chef-server-ctl reconfigure commands and it works wonders :)