/usr/bin/tar: invalid arguments

[maxnbk]

Due to limitations of living behind a corporate firewall, I'm trying to homeroll a concourse setup, simple to start, with just one DB host and 1 webnode+worker host.

The deployment appears to be all complete, albeit I have some unrelated trouble using systemctl stop on the worker service, probably due to handwritten systemd files. Something for later.
In any case, I can't get a simple hello_world task to execute, but don't know nearly enough about baggageclaim / garden / guardian to know where to look.

I'm on CentOS-7.6, concourse-6.0.0, with an OS kernel version which my engineering department assures me supports user namespaces, SELinux is disabled, and we have an internal docker registry I am configured to pull from, with docker installed on the host and able to pull images.

CONCOURSE_BAGGAGECLAIM_DRIVER is set to naive, but I experienced the same results with any other setting I knew to try, only landing on that after seeing it fixed other volume creation issues for CentOS users.

When I try to do an echo hello world type of job, with a :
fly -t staging-ny execute -c task_hello_world.yml, I get a simple 500, and this specific set of logs, where, near as I can tell, everything else is normal, but this appears not to be:

Apr 22 17:52:48 ny-stg-ci-worker01 concourse_worker.sh: {"timestamp":"2020-04-22T21:52:48.116831115Z","level":"info","source":"baggageclaim","message":"baggageclaim.api.volume-server.get-volume.get-volume.volume-not-found","data":{"session":"3.1.34.1","volume":"35754d71-b4d0-400c-4470-4692dd45bfa2"}}
Apr 22 17:52:48 ny-stg-ci-worker01 concourse_worker.sh: {"timestamp":"2020-04-22T21:52:48.116883051Z","level":"info","source":"baggageclaim","message":"baggageclaim.api.volume-server.get-volume.volume-not-found","data":{"session":"3.1.34","volume":"35754d71-b4d0-400c-4470-4692dd45bfa2"}}
Apr 22 17:52:48 ny-stg-ci-worker01 concourse_worker.sh: {"timestamp":"2020-04-22T21:52:48.131709440Z","level":"error","source":"baggageclaim","message":"baggageclaim.api.volume-server.stream-in.failed-to-stream-into-volume","data":{"error":"fork/exec /usr/bin/tar: invalid argument","session":"3.1.38","volume":"35754d71-b4d0-400c-4470-4692dd45bfa2"}}
Apr 22 17:52:48 ny-stg-ci-worker01 concourse_webnode.sh: {"timestamp":"2020-04-22T21:52:48.132374072Z","level":"error","source":"atc","message":"atc.create-artifact.failed-to-stream-volume-contents","data":{"error":"failed to stream in to volume","session":"1347"}}

Thank you for any help.

[cirocosta]

Hi @maxnbk , thanks for reporting!

I don't have a really good answer, but I've got some pointers for those who
might want to dig into this.

I was able to reproduce the issue on master (6.x,
f4d6874)
grabbing a Centos 7 machine on GCP (which runs on Linux 3.10).

At first, I thought that maybe the version of tar that they ship with was
very old, and then it could be not playing nice with arguments that we give it.
It turns out that while it's indeed older than the version of tar that the
concourse/concourse image ships with (1.26 vs 1.29), I was able to discard
that hypothesis by building the latest tar from source (1.32) and .. that
didn't bring any good results.

Attaching dlv to the process, I was able to gather what inputs it had used for
the execution of tar :

> github.com/concourse/baggageclaim/volume.(*tarZstdStreamer).In() /home/ubuntu/concourse/baggageclaim/volume/stream_in_out_linux.go:32 (PC: 0x17413e7)
Warning: debugging optimized function
    27:         tarCommand.Stdin = zstdDecompressedStream
    28:         tarCommand.Stdout = os.Stderr
    29:         tarCommand.Stderr = os.Stderr
    30:
    31:         err = tarCommand.Run()
=>  32:         if err != nil {
    33:                 zstdDecompressedStream.Close()
    34:
    35:                 if _, ok := err.(*exec.ExitError); ok {
    36:                         return true, err
    37:                 }

(dlv) p err
error(*os.PathError) *{
        Op: "fork/exec",
        Path: "/bin/tar",
        Err: error(syscall.Errno) EINVAL (22),}

(dlv) p tarCommand
*os/exec.Cmd {
        Path: "/bin/tar",
        Args: []string len: 5, cap: 5, [
                "tar",
                "-C",
                "/dev/fd/3",
                "-xf",
                "-",
        ],

which ... seem to be very reasonable, and I'm not sure why that'd fail like
this. Looking closely though, we can see that the problem is not coming from
tar complaining about flags or anything like that - instead, it's the execve
syscall failing with EINVAL:

       EINVAL An ELF executable had more than one PT_INTERP segment (i.e.,
              tried to name more than one interpreter).

which yeah, is quite weird

While on it, I was curious to see if containers would even run in a 3.10 kernel
(as that's quite old and below what we consider a minimum version - 3.19; see
https://concourse-ci.org/concourse-worker.html#worker-prerequisites), and
gdn couldn't create them:

run check step:
        run check step:
                find or create container:
                        runc run: exit status 1:
                                container_linux.go:346:
                                        starting container process caused "process_linux.go:319:
                                                getting the final child's pid from pipe caused \"EOF\""

[maxnbk]

Very qualitative analysis, Ciro, thanks for your investigation thus far.
So, I'm curious - What is different about what gdn needs to create containers compared to the ability to run docker containers in general?

I've encountered the difference between running containers via docker vs user-space-containers via podman, which requires a kernel which supports user namespaces as well, and that is supported/works.

I am also led to believe that redhat's kernel versioning scheme is not completely accurate with respect to its direct compatibility to features, as they essentially maintain and backport mainline kernel features into the older kernels, so a 3.10 redhat kernel may have significantly more features than a non-redhat one, but that isn't a guarantee of compatibility with those features either, technically, I imagine.

In any case, do you think it would be advisable to look into upgrading the kernel on these hosts and retrying? It's a bit prohibitive for me (need to convince Engineering why to bother), but I'm willing to give it a shot if it has a good chance of working.

[maxnbk]

Answered my own question.
I noticed that I had "kernel-ml" packages available, implying my Eng team has set up elrepo availability, so I installed the kernel-ml (mainline) version 5.5.8, set grubs default kernel to that version, rebooted, spun services up, and ran my same hello world test.

hello world
succeeded

says it all!
Thanks for digging in that deeply, I was really hoping I could avoid getting elbow deep with kernels, but this wasn't too bad.