Introduce Logging Stack: Add fluentd, add loki (🚧🚧 DEVOPS 🚧🚧)

[mrnicegyu11]

What do these changes do?

⚠️ ⚠️ ⚠️ Invasive PR - merge at convenient time ⚠️ ⚠️ ⚠️

Logs will now be sent from the docker daemon to fluend, with fluentd running as a docker swarm service inside the logging stack, via a TCP port that is host-exposed. Fluentd will send the logs to graylog and the newly introduced grafana loki.

Syslogs from the Linux Kernel will for now only be available in graylog.

Computational clusters and the impact of this PR on them have not been fully evaluated, please comment on this quickly @sanderegg . If it doesnt impact them at all, even better.

This unblocks grafana alerting.

🚧🚧 DevOps 🚧🚧:

• This PR requires https://git.speag.com/oSparc/osparc-infra/-/merge_requests/305 to be merged and new AMIs based on this code being built, available and put into the repo.config.template
• Any old machines (persistent or not) will stop sending logs to graylog (and loki for that matter) once this PR is rolled out. Warm and hot buffers must be re-created. The docker daemon on persistant machines must be restarted (short downtime) and the /etc/docker/daemon.json must be updated.

Related issue/s

• Setup OPS stack for tracing osparc-simcore#7636
• Configure our opentelemetry tracing so we can sample it and enable opentelemetry tracing in prod #1090

Related PR/s

• https://git.speag.com/oSparc/osparc-ops-deployment-configuration/-/merge_requests/1511
• https://git.speag.com/oSparc/osparc-infra/-/merge_requests/305

Checklist

• I tested and it works

• Service has resource limits and reservations
• Service has placement constraints or is global
• Service is restartable
• Service restart is zero-downtime
• Service is monitored (via prometheus and grafana)
• Service is not bound to one specific node (e.g. via files or volumes)
• Relevant OPS E2E Test are added
• Service's Public URL is included in maintenance mode
• Service's Public URL is included in testing mode

[bisgaard-itis]

Thanks a lot for the effort.

[YuryHrytsuk]

Thanks!

[matusdrobuliak66]

Sorry, I haven’t gone through it thoroughly, to unblock you a approve

[mrnicegyu11]

All good, I checked with SAN and he had some very reasonable feedback, i will do one more round of tests
mispalced comment on the wrong PR, this will be merged after the upcomming prod release

[sanderegg]

Computational clusters and the impact of this PR on them have not been fully evaluated, please comment on this quickly @sanderegg . If it doesnt impact them at all, even better.

@mrnicegyu11 not entirely sure here but from computational clusters end what is important is:

• a new AMI that can be configured to send docker logs to fluentd which I think you have prepared here https://git.speag.com/oSparc/osparc-infra/-/merge_requests/305 -> AMI must be created
• then the configuration of both autoscaling and clusters-keeper must be updated per deployment accordingly which seems to be there: https://git.speag.com/oSparc/osparc-ops-deployment-configuration/-/merge_requests/1511
  now it depends in what orders this gets deployed, but it is important that the AMI and configuration are in sync.

[sanderegg]

commented

[YuryHrytsuk]

What is the state of unchecked items? Could you add a note next to them what is the state?

Also do I get this correctly, that once this is merged and deployed to master, all logging workflows would be broken until machines are reprovisioned? The same with AMIs I presume. So, probably, once merged, DevOps must be on top of it.

All the infra changes would need to be applied on PROD, so we need clear and easy to do (difficult to make mistakes) instructions to ensure stag / prod release are done smoothly (if it gets to this stage ✊ ) @mrnicegyu11

Thanks for all the effort!

[YuryHrytsuk]

@mrnicegyu11 if is safer if it only gets to master and does not block future staging / prod releases (these stages will not be affected). Thanks 🙏

[mrnicegyu11]

• Host Port leads to non-zero-downtime, is it avoidable? It is not, as the docker daemon cannot send logs "directly into the swarm / into containers"
• Update-Policy is something to think about

[mrnicegyu11]

re Or to put it in other words, can it be only deployed in master or as long as we merge this PR, it will to all deployments with next staging / prod release?: It would propagate sadly, we can revert it but then changes to this repo, ops-config and osparc-infra also need reverting. AMIs need reverting as well. So it is not impossible but also not easy to change back @YuryHrytsuk

[mrnicegyu11]

re But loki does need volumes. Am I missing something?: If loki should never lose any log, one must give it a persistent volume. For now, i still consider graylog our main logging platform, and loki is to be tested, it is not clear if loki will be able to replace graylog fully or if it is only helpful in the context of grafana combined telemetry (logging, tracing, metrics)

[mrnicegyu11]

Required fix: bf6719e