Error running `podman build` with option `--build-context` on Hyper-V

[l0rd]

Issue Description

An image's build fails on Hyper-V when using additional build context. The same build works fine when using WSL as the machine provider.

copier: get: "/test": getting extended attributes for "/test": listing extended attributes of "/test": invalid argument

⚠️This error will be reproducible after we merge these PRs (1 and 2). Currently, podman build fails earlier with a different error message.
⚠️One way to address this problem would be to send the additional build contexts as tarfiles. However, investigating the root cause of this problem is still important as it can have other effects.

Steps to reproduce the issue

Steps to reproduce the issue

1. Create a folder context1 and add an empty file test in it
2. Create a folder context2 and add this Dockerfile:

FROM quay.io/libpod/alpine_nginx
COPY --from=test-context test /"

3. Run podman build --build-context test-context=.\context1 .\context2

Describe the results you received

  STEP 2/2: COPY --from=test-context test /
  Error: building at STEP "COPY --from=test-context test /": reading "/mnt/c/Users/mario/AppData/Local/Temp/ginkgo113287525/test": error during bulk transfer for copier.request{Request:"GET", Root:"/", preservedRoot:"/mnt/c/Users/mario/AppData/Local/Temp/ginkgo113287525", rootPrefix:"/mnt/c/Users/mario/AppData/Local/Temp/ginkgo113287525", Directory:"/", preservedDirectory:"/mnt/c/Users/mario/AppData/Local/Temp/ginkgo113287525", Globs:[]string{"/test"}, preservedGlobs:[]string{"/mnt/c/Users/mario/AppData/Local/Temp/ginkgo113287525/test"}, StatOptions:copier.StatOptions{CheckForArchives:false, Excludes:[]string(nil)}, GetOptions:copier.GetOptions{UIDMap:[]idtools.IDMap(nil), GIDMap:[]idtools.IDMap(nil), Excludes:[]string(nil), ExpandArchives:false, ChownDirs:(*idtools.IDPair)(nil), ChmodDirs:(*fs.FileMode)(nil), ChownFiles:(*idtools.IDPair)(nil), ChmodFiles:(*fs.FileMode)(nil), StripSetuidBit:false, StripSetgidBit:false, StripStickyBit:false, StripXattrs:false, KeepDirectoryNames:false, Rename:map[string]string(nil), NoDerefSymlinks:false, IgnoreUnreadable:false, NoCrossDevice:false}, PutOptions:copier.PutOptions{UIDMap:[]idtools.IDMap(nil), GIDMap:[]idtools.IDMap(nil), DefaultDirOwner:(*idtools.IDPair)(nil), DefaultDirMode:(*fs.FileMode)(nil), ChownDirs:(*idtools.IDPair)(nil), ChmodDirs:(*fs.FileMode)(nil), ChownFiles:(*idtools.IDPair)(nil), ChmodFiles:(*fs.FileMode)(nil), StripSetuidBit:false, StripSetgidBit:false, StripStickyBit:false, StripXattrs:false, IgnoreXattrErrors:false, IgnoreDevices:false, NoOverwriteDirNonDir:false, NoOverwriteNonDirDir:false, Rename:map[string]string(nil)}, MkdirOptions:copier.MkdirOptions{UIDMap:[]idtools.IDMap(nil), GIDMap:[]idtools.IDMap(nil), ChownNew:(*idtools.IDPair)(nil), ChmodNew:(*fs.FileMode)(nil)}, RemoveOptions:copier.RemoveOptions{All:false}}: copier: get: "/test": getting extended attributes for "/test": listing extended attributes of "/test": invalid argument

Describe the results you expected

The build should be successful

podman info output

host:
  arch: amd64
  buildahVersion: 1.36.0
  cgroupControllers:
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.10-1.fc40.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.10, commit: '
  cpuUtilization:
    idlePercent: 92.5
    systemPercent: 4.33
    userPercent: 3.17
  cpus: 4
  databaseBackend: sqlite
  distribution:
    distribution: fedora
    variant: coreos
    version: "40"
  eventLogger: journald
  freeLocks: 2048
  hostname: localhost.localdomain
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 1000000
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 1000000
  kernel: 6.8.9-300.fc40.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 180523008
  memTotal: 671866880
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.11.0-1.20240531102943328308.main.4.g6838c50.fc40.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.12.0-dev
    package: netavark-1.11.0-1.20240603091959994178.main.6.ge03b816.fc40.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.12.0-dev
  ociRuntime:
    name: crun
    package: crun-1.15-1.20240531133908971237.main.28.g05c4ea1.fc40.x86_64
    path: /usr/bin/crun
    version: |-
      crun version UNKNOWN
      commit: 6cd8ca4ff492428eced77d57e8e8ebe3821ff193
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20240510.g7288448-1.fc40.x86_64
    version: |
      pasta 0^20240510.g7288448-1.fc40.x86_64
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.2-2.fc40.x86_64
    version: |-
      slirp4netns version 1.2.2
      commit: 0ee2d87523e906518d34a6b423271e4826f71faf
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.3
  swapFree: 0
  swapTotal: 0
  uptime: 0h 1m 2.00s
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /var/home/core/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/home/core/.local/share/containers/storage
  graphRootAllocated: 106769133568
  graphRootUsed: 3944439808
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 0
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /var/home/core/.local/share/containers/storage/volumes
version:
  APIVersion: 5.2.0-dev-c51095982
  Built: 1717459200
  BuiltTime: Tue Jun  4 00:00:00 2024
  GitCommit: ""
  GoVersion: go1.22.3
  Os: linux
  OsArch: linux/amd64
  Version: 5.2.0-dev-c51095982g.

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

Additional environment details

Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[Honny1]

This issue is specific to the Podman Hyper-V provider on Windows and occurs when interacting with files on 9p-mounted volumes. While a recent change now transfers the build context as a tarball over the network, which bypasses this problem for builds, the underlying bug in direct volume mounts still exists. This behavior was identified in #27059.

Technical Background

The root cause is that the 9p filesystem implementation used by the Hyper-V provider does not correctly translate NTFS file attributes into a format understood by the Linux VM. As a result, when a copy tool (like cp -p) inside the VM attempts to read or preserve these attributes from a mounted Windows drive, it encounters an unrecognized format and throws an error.

By contrast, WSL avoids this problem by using drvfs, a specialized translation layer built on top of 9p. This layer correctly maps NTFS file attributes, allowing for seamless interoperability between the Windows host and the Linux guest environment.

How to Reproduce

This issue can be reproduced using a Podman machine running with the Hyper-V provider.

Method 1: Using podman build
This method demonstrates the failure when a Containerfile copies a file from the build context located on the host's C:// drive.

1. SSH into the Podman machine:

podman machine ssh

2. Create a Containerfile in a directory on your C:// drive (e.g., C:\build-context\Containerfile):

FROM scratch
COPY Containerfile /file

3. From inside the Podman machine, attempt to build the image from the mounted context:

podman build -t fail-test /mnt/c/build-context

Method 2: Using podman run with a Volume Mount
This command directly reproduces the attribute error by mounting a directory from the C:// drive and attempting to copy a file while preserving its attributes.

1. Create a directory C:\podman_share and place a file named test.txt inside it.
2. Run the following command, which will fail during the cp -p operation:

podman run --rm -v C:\podman_share\:/drive fedora sh -c "cp -p /drive/test.txt /test.txt"

Possible Solutions

1. Modify Copy Behavior: Adjust the file-copying logic to avoid preserving all source attributes when operating on an NTFS (9p) filesystem.
2. Use virtio-fs for Mounts: Investigate replacing 9p with virtio-fs. This modern protocol offers higher performance and may provide better attribute compatibility. (I am not sure if it's possible)
3. Use drvfs: Implement drvfs for the 9p mount. (Note: This is likely not feasible, as drvfs is a proprietary, deeply integrated component of WSL and is not available as a standalone driver for standard Hyper-V VMs.)