lchown : invalid argument error while doing podman run with rootless user id

[riddhiprj]

Problem statement - I am trying to create ansible execution environment , and I build the ee minimum image with ansible-builder command. I am using podman as container runtime. Creating image works fine.

when I run podman run -it -localhost/hcsc-ee-2.16-test bash , while logged in as user ansible , It works fine . But if I run this as my userid 455786 , it returns error - lchown /home/u455786/.local/share/containers/storage/volumes/###/_data: invalid argument

=================================
podman info

arch: amd64
buildahVersion: 1.31.5
cgroupControllers: []
cgroupManager: cgroupfs
cgroupVersion: v1
conmon:
package: conmon-2.1.8-1.module+el8.9.0+21697+6a5e98e7.x86_64
path: /usr/bin/conmon
version: 'conmon version 2.1.8, commit: 57ed23ee47beaf9a72b28f5666fab095a9ad4a38'
cpuUtilization:
idlePercent: 92.37
systemPercent: 4.51
userPercent: 3.13
cpus: 4
databaseBackend: boltdb
distribution:
distribution: '"rhel"'
version: "8.9"
eventLogger: file
freeLocks: 1996
hostname: tazrslanscn01
idMappings:
gidmap:
- container_id: 0
host_id: 455786
size: 1
uidmap:
- container_id: 0
host_id: 455786
size: 1
kernel: 4.18.0-513.24.1.el8_9.x86_64
linkmode: dynamic
logDriver: k8s-file
memFree: 9004441600
memTotal: 33425752064
networkBackend: cni
networkBackendInfo:
backend: cni
dns:
package: podman-plugins-4.6.1-9.module+el8.9.0+21697+6a5e98e7.x86_64
path: /usr/libexec/cni/dnsname
version: |-
CNI dnsname plugin
version: 1.3.1
commit: unknown
package: containernetworking-plugins-1.3.0-8.module+el8.9.0+21697+6a5e98e7.x86_64
path: /usr/libexec/cni
ociRuntime:
name: runc
package: runc-1.1.12-1.module+el8.9.0+21697+6a5e98e7.x86_64
path: /usr/bin/runc
version: |-
runc version 1.1.12
spec: 1.0.2-dev
go: go1.20.12
libseccomp: 2.5.2
os: linux
pasta:
executable: ""
package: ""
version: ""
remoteSocket:
exists: true
path: /run/user/455786/podman/podman.sock
security:
apparmorEnabled: false
capabilities: CAP_NET_RAW,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
rootless: true
seccompEnabled: true
seccompProfilePath: /usr/share/containers/seccomp.json
selinuxEnabled: false
serviceIsRemote: false
slirp4netns:
executable: /usr/bin/slirp4netns
package: slirp4netns-1.2.1-1.module+el8.9.0+21697+6a5e98e7.x86_64
version: |-
slirp4netns version 1.2.1
commit: 09e31e92fa3d2a1d3ca261adaeb012c8d75a8194
libslirp: 4.4.0
SLIRP_CONFIG_VERSION_MAX: 3
libseccomp: 2.5.2
swapFree: 0
swapTotal: 0
uptime: 1163h 2m 23.00s (Approximately 48.46 days)
plugins:
authorization: null
log:

• k8s-file
• none
• passthrough
• journald
  network:
• bridge
• macvlan
• ipvlan
  volume:
• local
  registries:
  search:
• registry.access.redhat.com
• registry.redhat.io
• docker.io
  store:
  configFile: /home/u455786/.config/containers/storage.conf
  containerStore:
  number: 1
  paused: 0
  running: 0
  stopped: 1
  graphDriverName: overlay
  graphOptions:
  overlay.ignore_chown_errors: "true"
  graphRoot: /home/u455786/.local/share/containers/storage
  graphRootAllocated: 115223822336
  graphRootUsed: 27731152896
  graphStatus:
  Backing Filesystem: xfs
  Native Overlay Diff: "true"
  Supports d_type: "true"
  Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
  number: 32
  runRoot: /run/user/455786/containers
  transientStore: false
  volumePath: /home/u455786/.local/share/containers/storage/volumes
  version:
  APIVersion: 4.6.1
  Built: 1713524958
  BuiltTime: Fri Apr 19 06:09:18 2024
  GitCommit: ""
  GoVersion: go1.20.12
  Os: linux
  OsArch: linux/amd64
  Version: 4.6.1

========================
/etc/subuid

ansible:100000:65536
bfadmin:165536:65536
u535525:231072:65536
u536377:296608:65536
i275943:362144:65536
u388072:427680:65536
455786:493216:65536

With the ansible user I can bash into the i