Dynamic sidecar takes sometimes 30 seconds until it is healthy

[sanderegg]

After a long testing session with @YuryHrytsuk we found out that the dynamic-sidecar sometimes takes way longer than it should to be in a healthy state.

On AWS-master this seems to happen every single time:

• right after an autoscaled node is made ready,
• dynamic-sidecar starts,
• fails to connect to the otel-collector due to DNS resolution failing,
• fails to connect to the storage micro-service due to DNS resolution failing, and this can take up to 30 seconds

In order to circumvent the issue, we need to access storage via internal IP/port so we do not wait for docker network DNS being ready

an issue is that the dy-sidecar fails after 30 attempts to connect to storage, even though there is already a docker healthcheck that allows for 180s startup time (will need to be fixed @GitHK )

[YuryHrytsuk]

It seems to perfectly correlate with Bulk sync to node ... timed out

journalctl log

Dec 09 13:27:35 ip-10-0-0-160 dockerd[806]: time="2025-12-09T13:27:35.992864589Z" level=error msg="Bulk sync to node <redacted> timed out"

siedcar logs

log_level=DEBUG | log_timestamp=2025-12-09 13:27:37,109 | log_source=simcore_service_dynamic_sidecar.core.storage:_is_storage_responsive(46) | log_uid=None | log_oec=None | log_trace_id=0 | log_span_id=0 | log_msg=Finished checking storage connection at url='http://<redacted>:<redacted>/v0/' auth=None
log_level=INFO | log_timestamp=2025-12-09 13:27:37,110 | log_source=simcore_service_dynamic_sidecar.modules.service_liveness:wait_for_service_liveness(106) | log_uid=None | log_oec=None | log_trace_id=0 | log_span_id=0 | log_msg=Service 'Storage' found at 'http://<redacted>:<redacted>/v0/' after 21092.46 ms

[YuryHrytsuk]

Docker inter-node communication

https://docs.docker.com/engine/network/drivers/overlay/
https://docs.docker.com/engine/swarm/swarm-tutorial/

[YuryHrytsuk]

Docker Swarm Bulk Sync Timeouts. Which port is being used?

Firstly, we identify which code is responsible for this logic. This we can do via searching for log entries we see in journalctl logs.

https://github.com/moby/moby/blob/a7e55b9e4ed9af0ae49fbc21dd1688f3a660e60a/daemon/libnetwork/networkdb/cluster.go#L607

func (nDB *NetworkDB) bulkSyncNode(networks []string, node string, unsolicited bool) error {
	//...
	mnode := nDB.nodes[node]
	//...
	nDB.memberlist.SendReliable(&mnode.Node, buf)
}

We see that actual network call is done via sendReliable. How does it identify host and port?

https://github.com/hashicorp/memberlist/

//https://github.com/hashicorp/memberlist/blob/master/memberlist.go#L601
func (m *Memberlist) SendReliable(to *Node, msg []byte) error {
	return m.sendUserMsg(to.FullAddress(), msg)
}
//...

//https://github.com/hashicorp/memberlist/blob/master/state.go#L66
func (n *Node) FullAddress() Address {
	return Address{
		Addr: joinHostPort(n.Addr.String(), n.Port),
		Name: n.Name,
	}
}
...
//https://github.com/hashicorp/memberlist/blob/master/state.go#L44C1-L56C2
type Node struct {
	Name  string
	Addr  net.IP
	Port  uint16
	...
}

Host and Port are coming from Node object. This node object is actually stored in nDB.nodes. Let's find out where / how this list is being initialized.

https://github.com/moby/moby/blob/master/daemon/libnetwork/networkdb/event_delegate.go#L39

func (e *eventDelegate) NotifyJoin(mn *memberlist.Node) {
	//...
	e.nDB.nodes[mn.Name] = &node{Node: *mn}
	//...
}

So, nDB.nodes are actually initialized via memberlist.Node object. Let's see where memberlist.Node are initialized to trace Port and Host value assignment.

https://github.com/hashicorp/memberlist/blob/master/state.go#L1149C4-L1149C43

func (m *Memberlist) aliveNode(a *alive, notify chan struct{}, bootstrap bool) {
	//...
	state = &nodeState{
		Node: Node{
			Name: a.Node,
			Addr: a.Addr,
			Port: a.Port,
			Meta: a.Meta,
		},
		State: StateDead,
	}
	//...
	m.config.Events.NotifyJoin(&state.Node)
	//...
}

Let's find who calls aliveNode

https://github.com/hashicorp/memberlist/blob/master/memberlist.go#L431C22-L431C30

func (m *Memberlist) setAlive() error {
	//...
	addr, port, err := m.refreshAdvertise()
	//...
	m.aliveNode(&a, nil, true)
	//...
}

Ok. Let's see what is m.refreshAdvertise

https://github.com/hashicorp/memberlist/blob/master/memberlist.go#L490

func (m *Memberlist) refreshAdvertise() (net.IP, int, error) {
	addr, port, err := m.transport.FinalAdvertiseAddr(
		m.config.AdvertiseAddr, m.config.AdvertisePort
	)
	//...
}

So, it comes from memberlist's config.

https://github.com/hashicorp/memberlist/blob/master/memberlist.go#L248

func Create(conf *Config) (*Memberlist, error) {
	m, err := newMemberlist(conf)
	//...
	if err := m.setAlive(); err != nil {
		_ = m.Shutdown()
		return nil, err
	}
	//...
}

https://github.com/hashicorp/memberlist/blob/master/memberlist.go#L100

func newMemberlist(conf *Config) (*Memberlist, error) {
	//...
	m := &Memberlist{
		config:               conf,
		//...
	}
	//...
	mlist, err := memberlist.Create(config)
	//...
	nDB.memberlist = mlist
	//...
}

So, we need to understand how the config is being created. It contains IP and Port

https://github.com/moby/moby/blob/master/daemon/libnetwork/networkdb/cluster.go#L103

func (nDB *NetworkDB) clusterInit() error {
	config := memberlist.DefaultLANConfig()
	mlist, err := memberlist.Create(config)
	nDB.memberlist = mlist
}}

https://github.com/hashicorp/memberlist/blob/master/config.go#L303

func DefaultLANConfig() *Config {
	return &Config{
		BindAddr:                "0.0.0.0",
		BindPort:                7946,
		AdvertiseAddr:           "",
		AdvertisePort:           7946,
	}
}

So, the answer is 7946 if this analysis is correct

[YuryHrytsuk]

Enabled docker debug logs on autoscaled ec2 https://git.speag.com/oSparc/osparc-ops-deployment-configuration/-/merge_requests/1712

[YuryHrytsuk]

The (suspected) root problem of 10-20 seconds delay

• (Bulk) sync networks race condition leads to bulk sync timeouts (dns resolution is delayed) moby/moby#51701

[YuryHrytsuk]

The (suspected) root problem of 10-20 seconds delay

* [(Bulk) sync networks race condition leads to bulk sync timeouts (dns resolution is delayed) moby/moby#51701](https://github.com/moby/moby/issues/51701)

To workaround this (i.e. avoid docker swarm dns / networks) we can re-introduce #573