T-8891: Compatibility with older Linux systems

[paweljw]

Vector fails to start when collector is ran via docker-compose on older Linux systems (e.g. Ubuntu 20.04 Focal, 5.4.x kernel line). The issue traces back to:

OS can't spawn worker thread: Operation not permitted (os error 1)

Having done some research, I'm of two minds as to what can be causing this:

• one option is caps; looking at effective caps on the collector container, 0xa80425fb misses both 0x18 and 0x17 (CAP_SYS_RESOURCE, CAP_SYS_NICE). This might result in operation not permitted on thread creation. 0x18 is a semi-nuclear cap though, so I'm keen to hear your thoughts on this.
• another option is seccomp. I found this moby issue which leads me to believe we may be trying to use clone3 which isn't explicitly whitelisted, and then Tokio for some reason doesn't fall back to clone. I propose adding an explicit seccomp profile here which is a copy of Docker's default with just clone3 added as an alternative to the completely-nuclear option of seccomp=undefined - to me the latter is not permissible in a production system.
• at the end of the day, this may also be some glibc incompatibility - in terms of what the host system has vs a potentially dynamically linked Vector binary. Looking at timberio Vector images, they use Vector via a .deb. We could explore building our own statically-linked Vector (rustflags = ["-C", "target-feature=+crt-static"] etc), but that gives us another moving piece to maintain. Static linking is currently used only for Windows packages it seems (no surprises there).

note to self: manual install docs will need updating if we merge with seccomp profile.

[paweljw]

@kessl more context: can't reproduce this on another Ubuntu 20.04 with 5.4.0 kernel, but a slightly newer Docker build (20.10.21 vs 20.10.6). Noting that clone3 was explicitly added to default seccomp in moby/moby@567c01f, tagged back to 20.10.10, but no earlier. Leads me to think more and more it's seccomp.

[jurajmasar]

Thank you!