Unable to run podman commands in podman container via azure pipeline

[abhidev8]

Issue Description

Hi, I am trying to automate container image creation using Azure pipelines, running Podman in Podman (PINP) and could not be able to run podman commands inside azure pipeline. Here's my repo structure:

Repo Name: BuildContainerPipeline
|==>azure-pipelines.yml
|==>ContainerImage.yml
|==>containerFile

• NOTE: The BuildContainerPipeline is executing on containerized azure agent which is created manually/from local machine fedora box based on the above containerFile

1. azure-pipelines.yml ==> this contains some basic checks and refers to template-containerimage.yml to build container image (refer to the below .yml script)

#Starter pipeline

trigger:
  batch: true
  branches:
    include:
      - main
      - feature/*

pool: DEV-AWS

variables:
- name: region
  value: "us-west-2"

jobs:
  - job: clean_workspace_job
    workspace:
      clean: all
    condition: always()
    steps:
      - checkout: none
      - task: CmdLine@2
        displayName: clean task
        inputs:
          script: |
            echo "Cleaning workspace"

  - deployment: Build_Container 
    displayName: Build Container
    strategy:
      runOnce:
        deploy:
          steps:
            - task: CmdLine@2
              displayName: List Agent Directory
              inputs:
                script: |
                  set -ex
                  ls -alh $AGENT_BUILDDIRECTORY
                  ls -alh $AGENT_BUILDDIRECTORY/s

            - template: ContainerImage.yml

3. ContainerImage.yml ==> this contains aws and podman commands that creates image and push to AWS ECR (refer to the below containerfile)

parameters:
  - name: region
    type: string
    default: "us-west-2"
  - name: displayName
    type: string
    default: "Build Agent Container"
  - name: imageName
    type: string
    default: "aws-ado-agent"
  - name: ECRName
    type: string
    default: "xxxxxxxx.dkr.ecr.us-west-2.amazonaws.com"
  - name: workingDir
    type: string
    default: ""
  - name: dockerFileDirPath
    type: string
    default: "$AGENT_BUILDDIRECTORY/s"
  - name: dockerFileName
    type: string
    default: dockerFile

steps:
  - task: CmdLine@2
    name: Build_Container
    displayName: ${{parameters.displayName}}
    inputs:
      script: |
        set -ex
        curl 169.254.170.2$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI
        echo "start Image build"
        aws ecr get-login-password --region ${{parameters. Region}} | podman login --username AWS --password-stdin ${{parameters.ECRName}}
        cd ${{parameters.dockerFileDirPath}}
        VERSION=$(Build.BuildNumber)
        IMAGE_NAME=${{parameters.imageName}}
        podman build --format docker -t ${IMAGE_NAME}:v$VERSION -f ${{parameters.dockerFileName}} .
        podman tag 12345 xxxxxxxx.dkr.ecr.us-west-2.amazonaws.com/${IMAGE_NAME}:v$VERSION
        podman tag 12345 xxxxxxxx.dkr.ecr.us-west-2.amazonaws.com/${IMAGE_NAME}
        podman push xxxxxxxx.dkr.ecr.us-west-2.amazonaws.com/${IMAGE_NAME}:v$VERSION
        podman push xxxxxxxx.dkr.ecr.us-west-2.amazonaws.com/${IMAGE_NAME}
        podman rmi xxxxxxxx.dkr.ecr.us-west-2.amazonaws.com/${IMAGE_NAME}:v$VERSION
        podman rmi xxxxxxxx.dkr.ecr.us-west-2.amazonaws.com/${IMAGE_NAME}
        podman image prune -a;
      workingDirectory: "$(Pipeline.Workspace)/${{parameters.workingDir}}"

3. containerFile ==> I have created this container file as mentioned here containers/podman

Describe the results you received

• When I run the pipeline, I get the error below at podman login command, I have tried running AWS ECR command separately and could not get any error, I am seeing errors when executing any podman commands in the pipeline.

Pipeline Log:

2023-06-20T08:22:23.0312499Z + podman login --username AWS --password-stdin xxxxxxxx.dkr.ecr.us-west-2.amazonaws.com
2023-06-20T08:22:23.1067413Z + aws ecr get-login-password --region us-west-2
2023-06-20T08:22:24.7063750Z cannot clone: Operation not permitted
2023-06-20T08:22:24.7067977Z Error: cannot re-exec process
2023-06-20T08:22:26.8903041Z Exception ignored in: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>
2023-06-20T08:22:26.8903581Z BrokenPipeError: [Errno 32] Broken pipe
2023-06-20T08:22:27.4149409Z ##[error]Bash exited with code '125'.

• I have also tried just running `podman info' command but ran in to same error

Pipeline Log:

2023-06-20T08:33:46.1083385Z + podman info
2023-06-20T08:33:46.1083639Z start Image build
2023-06-20T08:33:46.3498375Z cannot clone: Operation not permitted
2023-06-20T08:33:46.3499097Z Error: cannot re-exec process
2023-06-20T08:33:46.4126170Z ##[error]Bash exited with code '125'.

Describe the results you expected

• I should be able to run podman commands without any issues

podman info output (ran from my local fedora box)

host:
  arch: amd64
  buildahVersion: 1.28.0
  cgroupControllers: []
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: conmon-2.1.5-1.fc36.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.5, commit: '
  cpuUtilization:
    idlePercent: 99.23
    systemPercent: 0.46
    userPercent: 0.31
  cpus: 8
  distribution:
    distribution: fedora
    variant: container
    version: "36"
  eventLogger: file
  hostname: mypcname
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.10.16.3-microsoft-standard-WSL2
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 23933665280
  memTotal: 26699354112
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: crun-1.7.2-2.fc36.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.7.2
      commit: 0356bf4aff9a133d655dc13b1d9ac9424706cac4
      rundir: /tmp/podman-run-1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +WASM:wasmedge +YAJL
  os: linux
  remoteSocket:
    path: /tmp/podman-run-1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.0-0.2.beta.0.fc36.x86_64
    version: |-
      slirp4netns version 1.2.0-beta.0
      commit: 477db14a24ff1a3de3a705e51ca2c4c1fe3dda64
      libslirp: 4.6.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.3
  swapFree: 7516192768
  swapTotal: 7516192768
  uptime: 19h 10m 28.00s (Approximately 0.79 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /home/myuser/.config/containers/storage.conf
  containerStore:
    number: 6
    paused: 0
    running: 0
    stopped: 6
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/myuser/.local/share/containers/storage
  graphRootAllocated: 269490393088
  graphRootUsed: 12191260672
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 32
  runRoot: /tmp/podman-run-1000/containers
  volumePath: /home/myuser/.local/share/containers/storage/volumes
version:
  APIVersion: 4.3.1
  Built: 1668180253
  BuiltTime: Fri Nov 11 20:54:13 2022
  GitCommit: ""
  GoVersion: go1.18.7
  Os: linux
  OsArch: linux/amd64
  Version: 4.3.1

I could not get able to resolve this issue or find alternative solution for my use case. Request the community to help me on this issue.
Please let me know if you require more details on my use case. Thanks for your help/inputs
@sbidoul @cevich @rhatdan @Luap99

[cevich]

Full disclosure: I know basically nothing about Azure or it's pipelines. I also don't know much/anything about the YAML you posted.

The errors you posted are very typical for trying to run containers inside containers. Bottom-line: They normally typically require a --privileged environment - among other things. Perhaps there are pipeline settings/options for them?

In any case, you said "trying to automate container image creation". This actually can possibly be done using the buildah image and some secret-sauce: quay.io/buildah/stable:v1.30.0 (not 1.29.x - it has a bug). Okay, well, maybe the sauce isn't all that secret, but the runtime (pipeline) environment requirements are a bit lower with buildah. Maybe podman can be made to work for this...I just never had much luck.

Anyway, one key thing is the top-level (host) environment needs to provide the /dev/fuse device into the build container. Without that, this won't work at all. However, there aren't any other host-side requirements, and this can be run entirely inside a rootless container.

• Clone/checkout/download your repo. or directory containing the Containerfile and/or build context (other needed files) into /path/to/build/context/
• export BUILDAH_FORMAT=docker
• export BUILDAH_ISOLATION=chroot
• echo "$SECRET_REGISTRY_PASSWORD" | buildah login -u "$SECRET_REGISTRY_USER" --password-stdin $TARGET_REGISTRY (important for both the final push, and for layer-caching - can make a BIG difference in build-speed depending on how your Containerfile is laid out).
• I HIGHLY recommend using build cache because chroot builds are fairly "slow".
• buildah build --layers --cache-from $IMAGE_NAME --cache-to $IMAGE_NAME --cache-ttl=128h -t $IMAGE_NAME:$IMAGE_TAG /path/to/build/context/
• It's assumed that $TARGET_REGISTRY is the first part of $IMAGE_NAME.
• The cache doesn't have to go to $IMAGE_NAME (no tag) it can be directed at any other registry.server/namespace/image_name. The layers will be pushed/pulled as tags to whatever image-name choose.
• You should set the --cache-ttl to whatever is appropriate. Also might be a good idea to set the registry's cleanup/prune policy for this image to be slightly longer than the --cache-ttl.
• Layer cache is most effective if you setup your Containerfile such that the items which change the least-frequently are toward the top. Note: If you use build-args, they can invalidate cache depending on where/how they're used.

If you get an error like this, it's because the build container can't access /dev/fuse (from the host):

Error: mounting new container: mounting build container "blah": creating overlay mount to blah, mount_data="blah": using mount program /usr/bin/fuse-overlayfs: unknown argument ignored: lazytime
fuse: device not found, try 'modprobe fuse' first
fuse-overlayfs: cannot mount: No such file or directory

If you get an error during the image build regarding failure/permission-denied setting ownership on a file or directory: The build container's user-namespace isn't large enough. The typical default is 65536 ID's but some container images (debian/ubuntu specifically) try to use UID/GID 65535 (for apt?) which will fail. The only way to fix that that I'm aware of is to run the build container as root (ick!) or allocated more IDs to the user (on the host) in /etc/subuid//etc/subgid.

Hope that helps.

[abhidev8]

Thanks for the detailed response @cevich.

The errors you posted are very typical for trying to run containers inside containers. Bottom-line: They normally typically require a --privileged environment - among other things. Perhaps there are pipeline settings/options for them?

• [abhidev8] I will verify If there are any options/settings available in Microsoft Azure Pipelines to provide --privileged environment.

Anyway, one key thing is the top-level (host) environment needs to provide the /dev/fuse device into the build container. Without that, this won't work at all.
If you get an error like this, it's because the build container can't access /dev/fuse (from the host):

Error: mounting new container: mounting build container "blah": creating overlay mount to blah, mount_data="blah": using mount program /usr/bin/fuse-overlayfs: unknown argument ignored: lazytime
fuse: device not found, try 'modprobe fuse' first
fuse-overlayfs: cannot mount: No such file or directory

• [abhidev8] One quick question here, how do we provide access to /dev/fuse device? are there any commands that I can include while building my container image?
• I also faced above error while running podman pull command from my local fedora box (fedora box on WSL), I tried running modprobe fuse command ang got below error

modprobe: FATAL: Module fuse not found in directory /lib/modules/5.10.16.3-microsoft-standard-WSL2

• I have these subuid & subgid set in my container image

idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536

Am I missing something here while running from the fedora box? I appreciate your input!

[cevich]

[abhidev8] I will verify If there are any options/settings available in Microsoft Azure Pipelines to provide --privileged environment.

This is almost always the sticking point. Note: --privileged isn't needed for the buildah image (along with many of the other podman-image suggested options). That's why I suggested using buildah. At the same time my experience with pipeline-type orchestration tools, is they give you very little control over whats run (or available) on the host.

[abhidev8] One quick question here, how do we provide access to /dev/fuse device? are there any commands that I can include while building my container image?

In terms of the host, it would be the --device /dev/fuse option to the podman or docker run command. Assuming Azure pipeline even allows setting such things (they might not). Though...

modprobe: FATAL: Module fuse not found in directory /lib/modules/5.10.16.3-microsoft-standard-WSL2

...ugg, if that's coming from the host, it's probably a show-stopper. I could be wrong (I don't know this environment), it might be possible to install it (from somewhere) and modprobe it in w/o rebooting. Probably a long-shot though. Normally kernel modules are built and shipped along with the kernel itself. Not having fuse is a significant bad sign ☹️

dang.

I have these subuid & subgid set in my container image

That's not needed if you use the buildah-image. They're for running rootless podman inside the (also rootless) podman image. For the buildah image there's not much point, it's just extra complication for not much actual benefit.

In any case, assuming you can't use a different disk image for the host, you may be stuck. IMHO, this is why many/most cloud providers offer a dedicated container-image build service. It hides nearly all the container-in-container complications. The other folks you pinged may have an idea of how/if you can get around not having /dev/fuse. That's a requirement for both the podman and buildah images IIUC 😞

[abhidev8]

This is almost always the sticking point. Note: --privileged isn't needed for the buildah image (along with many of the other podman-image suggested options). That's why I suggested using buildah.

I must go back and discuss with my architect using buildah tool or any other tool for this purpose. Currently, we must stick to Azure pipelines as they are implemented and used throughout our organization.

At the same time my experience with pipeline-type orchestration tools, is they give you very little control over whats run (or available) on the host.

I totally agree with this point!

In terms of the host, it would be the --device /dev/fuse option to the podman or docker run command. Assuming Azure pipeline even allows setting such things (they might not). Though...

JFYI, we have different kind of setup/environment for running containers

• We have one pipeline that is used to create container image and push to AWS Elastic Container Registry (as mentioned in this blog)
• We have another pipeline that has Terraform scripts to create AWS Elastic Container Service (container management service) and corresponding infrastructure components which is used to run, stop, scale up/down containers

We do not execute podman run command in pipeline as containers are spin up through AWS ECS, I am just wondering if we have even such kind of settings to include --device /dev/fuse option in AWS ECS. let me check if this is possible in my use case.

Normally kernel modules are built and shipped along with the kernel itself. Not having fuse is a significant bad sign ☹️

That's what even my understanding is, but when I try to install fuse, I get message saying it is already installed!
Package fuse-2.9.9-14.fc36.x86_64 is already installed.

In any case, assuming you can't use a different disk image for the host, you may be stuck. IMHO, this is why many/most cloud providers offer a dedicated container-image build service. It hides nearly all the container-in-container complications.

Agreed! AWS provides EC2 Image Builder to automate container images, but I need to evaluate if it fits in my use case and also need to check on pricing for the same.

The other folks you pinged may have an idea of how/if you can get around not having /dev/fuse

Yeah, that would be great to know if that is possible.

[cevich]

Thanks for the background info./context.

My assumption with Azure pipelines is, you configure it to execute commands inside a specific container image. Then the service takes care of the container-execution and feeding it commands. If so, I would think it quite likely they don't allow changing container-execution options (like --privileged or --device /dev/fuse). But I could be wrong.

I must go back and discuss with my architect using buildah tool or any other tool for this purpose.

In case it helps, buildah is included in the podman binary (i.e. podman build is using a compiled-in buildah build). Anyway, it may not matter if you can't tell azure pipelines to pass devices into containers or execute them with extra privileges.

We do not execute podman run command in pipeline as containers are spin up through AWS ECS

Oh sorry for the confusion. I wasn't suggesting this. Only mentioned it as an example of how I tested this locally, running the build from the buildah image, started by podman (with --device).

I get message saying it is already installed!

Most likely that's just the user-space (command) component of fuse. It relies on communication with the kernel through the fuse module (which creates the /dev/fuse device).

AWS provides EC2 Image Builder to automate container images, but I need to evaluate if it fits in my use case and also need to check on pricing for the same.

Yep, gotcha. It's worth investigating IMHO. If they offer the build-capabilities you need, it'll likely save you money (in terms of execution-time) vs running a buildah container (in chroot mode) or kaniko or similar tool inside a container. Most of the build-time load is storage-bound. A build service will have (hopefully) optimized their storage performance, saving you oodles in build-time vs running in a container.

Anyway, if you have to run image builds inside a container, and cannot control the container's execution options (--device /dev/fuse), you're only "choice" is to use non-container based build tools like kaniko. They're "slow", but they do work and offer lots of control over caching and the build process in general.

[abhidev8]

My assumption with Azure pipelines is, you configure it to execute commands inside a specific container image. Then the service takes care of the container-execution and feeding it commands.

Yes, that's correct! @cevich

If so, I would think it quite likely they don't allow changing container-execution options (like --privileged or --device /dev/fuse).

Yes, that's what I am facing now

Yep, gotcha. It's worth investigating IMHO. If they offer the build-capabilities you need, it'll likely save you money (in terms of execution-time) vs running a buildah container (in chroot mode) or kaniko or similar tool inside a container. Most of the build-time load is storage-bound. A build service will have (hopefully) optimized their storage performance, saving you oodles in build-time vs running in a container.

Actually, we have other projects in our org. where we are running Docker IN Docker (DIND) and it is working fine, in our case my architect wants to go with Podman instead of Docker as it is more secure and robust in terms of container management service.

[cevich]

In the case of building in a containerized pipeline, the issue isn't so much the container technology as the way the outer container is being executed.

In case it helps: My background with building in containers comes from GitLab-CI where (similar to GitHub actions) you can host/provide a "runner" to service the pipeline. In providing a runner, it's possible to pass additional options to the container engine, such as --device /dev/fuse. That's how I was able to get pipeline-based builds working. It's also a popular architecture given you're in more control over the security aspects. Maybe Azure-pipelines offer a similar capability of hosting your own runner process?

Though I can totally understand not wanting to host "yet another tool" (and maintain it).

[cevich]

Just had a thought...with the aim of getting you building in this environment quickly. There are non-container-engine build tools available. For example, google's Kaniko is fairly easy to use. They even make a container image for this purpose - though you can also just install/run kaniko directly as well. There are other tools too, I'm just not familiar with them.

[abhidev8]

For podman build, there are a bunch of additional kernel-capabilities (provided by --privileged) (and sec-comp privileges?) needed to execute containers-in-containers - which is also needed for processing Dockerfile/Containerfile RUN command.

Infact, I am not able to run any podman related commands, tried with podman info and ran in to same permissions error that I encountered while executing podman build command

2023-06-20T08:33:46.1083385Z + podman info
2023-06-20T08:33:46.1083639Z start Image build
2023-06-20T08:33:46.3498375Z cannot clone: Operation not permitted
2023-06-20T08:33:46.3499097Z Error: cannot re-exec process
2023-06-20T08:33:46.4126170Z ##[error]Bash exited with code '125'.

[cevich]

yep, I'm pretty sure that's what you're hitting.

I got an answer from Nalin on the /dev/fuse question. He says you can use the VFS storage driver to get around this. However he cautioned that it's excruciatingly slow. I believe you can set this in storage.conf or via export STORAGE_DRIVER=vfs. I'll see if I can make this work locally...

[abhidev8]

That's some good news :) and one more thing, I have the below piece of commands set in my container file, if it helps!

# Copy & modify the defaults to provide reference if runtime changes needed.
# Changes here are required for running with fuse-overlay storage inside container.
RUN sed -e 's|^#mount_program|mount_program|g' \
           -e '/additionalimage.*/a "/var/lib/shared",' \
           -e 's|^mountopt[[:space:]]*=.*$|mountopt = "nodev,fsync=0"|g' \
           /usr/share/containers/storage.conf \
           > /etc/containers/storage.conf

As I mentioned in my post initially, I have taken code from this github containers/podman as base for my container file.

[cevich]

...okay, yep, this definitely works (thanks @nalind!).

Below I'm mimicking a pipeline environment through podman, so you can see I'm not granting any extra privileges or providing /dev/fuse. Just setting the required env. vars., and using the buildah image to execute the build command. I'm using the skopeo source because it's convenient, you don't actually need any volume mounts for this to work, just the Containerfile (and any other necessary content):

$ podman run -it --rm \
    -e BUILDAH_FORMAT=docker \
    -e BUILDAH_ISOLATION=chroot \
    -e STORAGE_DRIVER=vfs \
    -v $HOME/devel/skopeo/contrib/skopeoimage/stable/:/root/skopeo_stable:O \
    quay.io/buildah/stable:v1.30.0

[root@6380e731136f /]# buildah build -t skopeo:test /root/skopeo_stable/
STEP 1/7: FROM registry.fedoraproject.org/fedora:latest
Trying to pull registry.fedoraproject.org/fedora:latest...
Getting image source signatures
Copying blob 9c86172b66b3 done
Copying config 0e79e93fc5 done
Writing manifest to image destination
Storing signatures
STEP 2/7: RUN dnf -y update &&     rpm --setcaps shadow-utils 2>/dev/null &&     dnf -y install skopeo fuse-overlayfs         --exclude container-selinux &&     dnf clean all &&     rm -rf /var/cache /var/log/dnf* /var/log/yum.*
...cut...
STEP 7/7: ENTRYPOINT ["/usr/bin/skopeo"]
COMMIT skopeo:test
Getting image source signatures
Copying blob 3e6c72780d11 skipped: already exists
Copying blob 99416e1c1954 done
Copying config b9b264b4b8 done
Writing manifest to image destination
Storing signatures
--> b9b264b4b8c5
Successfully tagged localhost/skopeo:test
b9b264b4b8c5ab21109f017e5fea86f1e2fda8e1a511918af5c0a4e1c9b553ca

Yay!

[cevich]

As I mentioned in my post initially, I have taken code from this github containers/podman as base for my container file.

Gotcha. You'll find there is a similar setup in the buildah container image as well. Though note: the runtime env. vars. will override some settings in containers.conf within the image.

edit: storage.conf rather.

[abhidev8]

Thanks @cevich for exploring solutions for my use case :)

edit: storage.conf rather.

Correct me If I am wrong! I need to add instruction in my container file to append STORAGE_DRIVER=vfs in storage.conf file and If I look at my storage.conf file which is created as part of my containerfile, I see storage option is set like below:

[storage]

# Default Storage Driver, Must be set for proper operation.
driver = "overlay"

Do I need to change this to driver = "vfs" using RUN sed instruction in my container file? or I need to approach in any different manner to set the STORAGE_DRIVER=vfs option. Attached storage.conf file for reference
storage.conf.txt

[cevich]

A STORAGE_DRIVER=vfs env. var. at runtime will override a driver = <anything> at build-container creation time. I only mention using the env. vars. b/c it lets you use the stock buildah image. But if you need to make your own build-container image for other reasons, then yes, changing driver = "vfs" ahead of time is perfectly fine.

Note, both BUILDAH_FORMAT=docker and BUILDAH_ISOLATION=chroot are equivalent to buildah command-line options (--isolation and --format). If you prefer to pass them that way it's fine as well.

[cevich]

Oh, and once you get it all working. It might be worth checking out the --layers, --cache-from, --cache-to, and --cache-ttl options I mentioned at the beginning. With the vfs driver especially, you're main build-performance bottleneck is likely storage. For large, infrequently changing image layers, it can be faster to pull them from cache rather than building from scratch every time.

[cevich]

Oh good, okay so that means it can work with docker + podman's seccomp, or just native podman.

I see there is going to be major hit on performance end with vfs storage

Yeah, as I predicted. If you can't get the seccomp profile into ECS, I expect tools like Kaniko will have similarly abysmal performance. Leveraging some of the caching options can help, along with a carefully crafted Containerfile/Dockerfile. Otherwise, besides using a dedicated SaaS image-build service (which can have their own limitations), the only way to boost storage performance that I'm aware of is by using fuse-overlay (which you discovered isn't an option 😞) or some other more direct storage driver (also unlikely in this environment).

Given all that you've been through with this, it may be worth taking a step back and examining the overall architecture WRT image builds and associated automation. Perhaps a pivot to gihub-workflows or gitlab-ci are possibilities (I know both of them to work w/ a fuse-overlay setup)?

(BTW, I'm out on PTO for the next two weeks)

[abhidev8]

Thanks for your help and suggestions @cevich I will evaluate the options and will let you know which path I am going to take :)