Error 500: driver failed programming external connectivity on endpoint

[HelenH321]

Hi there

Please can someone help?

When rebooting a single node SO instance, the majority of services don't automatically start, and are shown as STARTING by an so-status command.

After a
$ sudo docker restart $(sudo docker ps -a -q)
We are left with 3 not started (2 in ERROR and 1 in PENDING):

Checking Docker status

Docker ------------------------------------------------------------------------------------------ [ OK ]

Checking container statuses

so-aptcacherng ---------------------------------------------------------------------------------- [ OK ]
so-cortex --------------------------------------------------------------------------------------- [ OK ]
so-curator -------------------------------------------------------------------------------------- [ OK ]
so-dockerregistry ------------------------------------------------------------------------------- [ OK ]
so-elastalert ----------------------------------------------------------------------------------- [ OK ]
so-elasticsearch -------------------------------------------------------------------------------- [ OK ]
so-filebeat ------------------------------------------------------------------------------------- [ OK ]
so-fleet ---------------------------------------------------------------------------------------- [ OK ]
so-grafana -------------------------------------------------------------------------------------- [ OK ]
so-idstools ------------------------------------------------------------------------------------- [ OK ]
so-influxdb ------------------------------------------------------------------------------------- [ OK ]
so-kibana --------------------------------------------------------------------------------------- [ OK ]
so-kratos ------------------------------------------------------------------------------------ [ ERROR ]
so-logstash ------------------------------------------------------------------------------------- [ OK ]
so-mysql ---------------------------------------------------------------------------------------- [ OK ]
so-nginx ---------------------------------------------------------------------------------------- [ OK ]
so-playbook ------------------------------------------------------------------------------------- [ OK ]
so-redis ---------------------------------------------------------------------------------------- [ OK ]
so-sensoroni ------------------------------------------------------------------------------------ [ OK ]
so-soc ------------------------------------------------------------------------------------------ [ OK ]
so-soctopus ------------------------------------------------------------------------------------- [ OK ]
so-steno ---------------------------------------------------------------------------------------- [ OK ]
so-strelka-backend ------------------------------------------------------------------------------ [ OK ]
so-strelka-coordinator -------------------------------------------------------------------------- [ OK ]
so-strelka-filestream --------------------------------------------------------------------------- [ OK ]
so-strelka-frontend -------------------------------------------------------------------------- [ ERROR ]
so-strelka-gatekeeper --------------------------------------------------------------------------- [ OK ]
so-strelka-manager ------------------------------------------------------------------------------ [ OK ]
so-suricata ------------------------------------------------------------------------------------- [ OK ]
so-telegraf ------------------------------------------------------------------------------------- [ OK ]
so-thehive --------------------------------------------------------------------------------- [ PENDING ]
so-thehive-es ----------------------------------------------------------------------------------- [ OK ]
so-wazuh ---------------------------------------------------------------------------------------- [ OK ]

and the following messages in /opt/so/log/salt/minion:

  1 2021-07-13 10:00:20,868 [salt.state       :321 ][ERROR   ][18242] Container 'so-kratos' is already configured as specified. Failed to start container 'so-kratos': 'Error 500: driver failed programming external connectivity on endpoint so-kratos (2ef3400a9026fc62b0595857144e07f676ac6d953715165eebb2b1c905b9b869): Bind for 0.0.0.0:4434 failed: port is already allocated'.
  2 2021-07-13 10:26:47,498 [salt.state       :321 ][ERROR   ][18242] Container 'so-strelka-frontend' is already configured as specified. Failed to start container 'so-strelka-frontend': 'Error 500: driver failed programming external connectivity on endpoint so-strelka-frontend (3ddadd7c859b22a6c252d923d465465554e780d73ab7ef9f50a4b82301fae634): Bind for 0.0.0.0:57314 failed: port is already allocated'.
  3 2021-07-13 10:27:28,730 [salt.state       :321 ][ERROR   ][18242] Container 'so-thehive' is already configured as specified. Failed to start container 'so-thehive': 'Error 500: driver failed programming external connectivity on endpoint so-thehive (a556b0b017f8d4e3c4db4f82725e2bc0d77647e2242ac6c164d60b20a63d08da): Bind for 0.0.0.0:9000 failed: port is already allocated'.

I have tried a docker refresh, but this did not seem to help.

Restarting these manually one by one does not seem to help either, and a salt-call state.highstate returns:
local:
Data failed to compile:

The function "state.highstate" is running as PID 8697 and was started at 2021, Jul 13 13:45:36.198127 with j6198127

Any help would be most appreciated.
Ideally we would want all these services to start automatically on boot without any interference - can this be done?
Is there a setting I am missing?
This is on v2.3.51, will an upgrade fix this?

Thanks,
Helen

[weslambert]

I would suggest upgrading before doing anything else, then see if you still experience the same or a similar issue.

[HelenH321]

Hi again all

Thanks @weslambert - I have now upgraded to v2.3.60 and we still see the same problem.

I have been able to get a full clean-sheet of "OK" now by doing more than one

sudo docker restart $(sudo docker ps -a -q)

and

salt-call state.highstate

But it seems like it should do this itself after a reboot?

Is there anything else that anyone can suggest that I could try in order to get this stable after a reboot without interference?

Thanks
Helen