internal prod tip: node 03 hard drive shows errors

[YuryHrytsuk]

Description

~[PROD]$ sudo smartctl -a /dev/nvme0
Error Information (NVMe Log 0x01, max 64 entries)
Num   ErrCount  SQId   CmdId  Status  PELoc          LBA  NSID    VS
  0          3    29  0x00cd  0xc502  0x000    414652320     1     -
  1          2    29  0x00cc  0xc502  0x000    414652320     1     -
  2          1    29  0x00c8  0x4502  0x000    414652320     1     -

Definition of done

• FIRST: isolate broken node from workloads https://git.speag.com/oSparc/osparc-ops-deployment-configuration/-/merge_requests/1484
• Substitute hard drive & add node back https://git.speag.com/oSparc/osparc-ops-deployment-configuration/-/merge_requests/1487
• Create incident report --> no need
• Add e2e tests to better monitor Docker Swarm Test https://git.speag.com/oSparc/osparc-ops-deployment-configuration/-/merge_requests/1484

[YuryHrytsuk]

Actions

Isolating unhealthy node:

1. Add necessary docker node labels to heathly node
2. Drain unhealthy Node
3. Configure services that were moved if necessery (e.g. configure graylog content packs)
4. Update inventory to reflect new state

[YuryHrytsuk]

Discovered shortcomings

• Webservers were all down for some time (rabbitmq error) which led to 404
  • shall be handled better with 🎨 Add fallback traefik 503 routes 🚨⚠️ DEVOPS osparc-simcore#7899 and closed by Serve custom html / api content when platform is down #218
  • shall webserver handle better this case OR Serve custom html / api content when platform is down #1084
• configuring node with ansible led to 404 again because keepalived was switched on while no services where running --> already in Ansible & Keepalived: smart ip flipping #901