How to properly cache podman containers on github actions?

[ssbarnea]

Using podman inside github actions is critical for some of us and in order to avoid never-ending job executions we need to make use of https://github.com/actions/cache support.

While apparently caching with the solution below works on MacOS, it seems that it does not on Linux.

      - name: Enable caching for podman-machine
        uses: actions/cache@v3
        with:
          path: |
            ~/.local/share/containers
            ~/.config/containers
          key: ${{ runner.os }}-${{ matrix.task-name }}-${{ hashFiles('package.json', 'yarn.lock', '.config/requirements.txt', '**/Taskfile.yml', 'tools/*.*') }}

Please note that we need to cache the contains on both linux and macos and that involves a very different strategy, as on macos caching containers means caching the entire podman machine image.

I opened this thread because I think that we should really produce and maintain a github reusable action for configuring podman, one that also covers caching containers.

What happens...

On Linux, you will see a pipeline warning such:

"Failed to save: "/usr/bin/tar" failed with error: The process '/usr/bin/tar' failed with exit code 2

Spotting where I comes from is tricky as all tasks report green but look for "Post " near the last tasks and expand it. you will see something like here:

Post job cleanup.
/usr/bin/tar --posix -cf cache.tzst --exclude cache.tzst -P -C /home/runner/work/vscode-ansible/vscode-ansible --files-from manifest.txt --use-compress-program zstdmt
/usr/bin/tar: ../../../.local/share/containers/storage/overlay/acc1a64e70538cbc8a7e867779a5ca667b00cd5c1e9a91ec240bf379e7ca6dcc/diff/etc/gshadow-: Cannot open: Permission denied
...(censored)
/usr/bin/tar: 
Warning: Failed to save: "/usr/bin/tar" failed with error: The process '/usr/bin/tar' failed with exit code 2

Basically tar fails to archive the containers due to permission denied. Not sure why this happens yet.

All github actions run a non-root user, but we can sudo if needed for script command, but stuff like caching will always run a 'runner' non-root user.

[vrothberg]

Thanks for reaching out, @ssbarnea!

@containers/podman-maintainers PTAL

Can you elaborate on which data you want to cache on Linux? Is the action run as a non-root user?

[ssbarnea]

I added extra info. Now I am not sure if that is caused by some leftover containers that might be still running at the end of the job. At least on macos, I did had to run podman machine stop to ensure that the VM can be archived.

[leleobhz]

My main issue concerning podman cache is because user usually cannot read/write other uids from cache (since some containers for example have UID 0 files inside it).

I fixed this inserting this step before caching:

      - name: Tar as root
        run: |
          sudo mv -fv /usr/bin/tar /usr/bin/tar.orig
          echo -e '#!/bin/sh\n\nsudo /usr/bin/tar.orig "$@"' | sudo tee -a /usr/bin/tar
          sudo chmod +x /usr/bin/tar

Less than beautiful, have a lot of concerns because tar runs as root but works for caching.

[TomerFi]

Worked for me - thanks!