Autoscaling: Sometimes unlabelled docker nodes remain in the swarm forever

[sanderegg]

From time to time, there are docker nodes that remain in the docker swarm forever:

• docker node in Drain mode, without any label

Current theory is something along the line of:

• a user starts a service,
• autoscaling creates an EC2,
• the EC2 connects to the swarm,
• the user deletes the service,
• autoscaling terminates the EC2 without setting the node labels,

or:
some EC2 that took too long to connect.

--> the docker node remain visible in the swarm forever and autoscaling will never remove it since it lacks the node labels

docker node ls
ID                            HOSTNAME        STATUS    AVAILABILITY   MANAGER STATUS   ENGINE VERSION
s1aics6zjb5mp1r1r3fqn9dv2     ip-10-1-6-138   Down      Drain                           25.0.3
o6ny1i7978a6ybusp7mti9os7     ip-10-1-7-47    Ready     Active                          25.0.3
0yq3i0z3wx3fa9y2vducgt937 *   prod-man1       Ready     Active         Leader           25.0.3
lgpuh0fylqrxdrarvhxcsd14w     prod-ops1       Ready     Active                          25.0.3
kx1cn9gw2blfdct2x34h6ryr4     prod-sim1       Ready     Active                          25.0.3

[sanderegg]

I found out that this happens when:

• jobs are scheduled,
• autoscaling creates machines,
• in the meantime the jobs are cancelled,
• the machine join the swarm as new docker nodes,
  - the autoscaling does not handle their node labels,
• the autoscaling removes the machines EC2 side,
• the docker nodes remain in the swarm and nobody takes care of them

for info @YuryHrytsuk

[sanderegg]

closed by #6898