system test: Why podman kube generate - container test passes in CI environment?

[sstosh]

In my fedora 37 environment, podman kube generate - container test is always failed
because of warning message Kubernetes only allows 63 characters .

test log

not ok 2 podman kube generate - container
# (from function `assert' in file helpers.bash, line 670,
#  in test file 710-kube.bats, line 36)
#   `assert "$output" !~ "Kubernetes only allows 63 characters"' failed
# # podman rm -t 0 --all --force --ignore
# # podman ps --all --external --format {{.ID}} {{.Names}}
# # podman images --all --format {{.Repository}}:{{.Tag}} {{.ID}}
# quay.io/libpod/testimage:20221018 f5a99120db64
# # podman container create --cap-drop fowner --cap-drop setfcap --name cVE6D9OIkn5nTPPg quay.io/libpod/testimage:20221018 top
# d00b1b47374c20e71692a5deae513ff3426eb22bb774c81bd0dea3e337aab7c9
# # podman kube generate cVE6D9OIkn5nTPPg
# time="2023-03-24T09:42:03+09:00" level=warning msg="Truncation Annotation: \"sha256:36d93dbf1cf35346effba973a726cd4ba19a967b68c3ce03bde0c1127015f2f9\" to \"sha256:36d93dbf1cf35346effba973a726cd4ba19a967b68c3ce03bde0c112\": Kubernetes only allows 63 characters"
# # Save the output of this file and use kubectl create -f to import
# # it into Kubernetes.
# #
# # Created with podman-4.5.0-dev
# apiVersion: v1
# kind: Pod
# metadata:
#   annotations:
#     io.podman.annotations.ulimit: nofile=1048576:1048576
#     org.opencontainers.image.base.digest/cVE6D9OIkn5nTPPg: sha256:36d93dbf1cf35346effba973a726cd4ba19a967b68c3ce03bde0c112
#     org.opencontainers.image.base.name/cVE6D9OIkn5nTPPg: localhost/interim-image:latest
#   creationTimestamp: "2023-03-24T00:42:03Z"
#   labels:
#     app: cVE6D9OIkn5nTPPg-pod
#   name: cVE6D9OIkn5nTPPg-pod
# spec:
#   containers:
#   - command:
#     - top
#     env:
#     - name: http_proxy
#       value: <mask>
#     - name: https_proxy
#       value: <mask>
#     - name: no_proxy
#       value: 0.0.0.0,non-existent-registry.com,127.0.0.1,myrepository.example,localhost
#     - name: HTTP_PROXY
#       value: <mask>
#     - name: HTTPS_PROXY
#       value: <mask>
#     image: quay.io/libpod/testimage:20221018
#     name: cVE6D9OIkn5nTPPg
#     securityContext:
#       capabilities:
#         drop:
#         - CAP_FOWNER
#         - CAP_SETFCAP
# #/vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
# #|     FAIL: podman kube generate cVE6D9OIkn5nTPPg
# #| expected: !~ Kubernetes only allows 63 characters
# #|   actual:    time=\"2023-03-24T09:42:03+09:00\" level=warning msg=\"Truncation Annotation: \\\"sha256:36d93dbf1cf35346effba973a726cd4ba19a967b68c3ce03bde0c1127015f2f9\\\" to \\\"sha256:36d93dbf1cf35346effba973a726cd4ba19a967b68c3ce03bde0c112\\\": Kubernetes only allows 63 characters\"
# #|         >    \# Save the output of this file and use kubectl create -f to import
# #|         >    \# it into Kubernetes.
# #|         >    \#
# #|         >    \# Created with podman-4.5.0-dev
# #|         >    apiVersion: v1
# #|         >    kind: Pod
# #|         >    metadata:
# #|         >      annotations:
# #|         >        io.podman.annotations.ulimit: nofile=1048576:1048576
# #|         >        org.opencontainers.image.base.digest/cVE6D9OIkn5nTPPg: sha256:36d93dbf1cf35346effba973a726cd4ba19a967b68c3ce03bde0c112
# #|         >        org.opencontainers.image.base.name/cVE6D9OIkn5nTPPg: localhost/interim-image:latest
# #|         >      creationTimestamp: \"2023-03-24T00:42:03Z\"
# #|         >      labels:
# #|         >        app: cVE6D9OIkn5nTPPg-pod
# #|         >      name: cVE6D9OIkn5nTPPg-pod
# #|         >    spec:
# #|         >      containers:
# #|         >      - command:
# #|         >        - top
# #|         >        env:
# #|         >        - name: http_proxy
# #|         >          value: <mask>
# #|         >        - name: https_proxy
# #|         >          value: <mask>
# #|         >        - name: no_proxy
# #|         >          value: 0.0.0.0\,non-existent-registry.com\,127.0.0.1\,myrepository.example\,localhost
# #|         >        - name: HTTP_PROXY
# #|         >          value: <mask>
# #|         >        - name: HTTPS_PROXY
# #|         >          value: <mask>
# #|         >        image: quay.io/libpod/testimage:20221018
# #|         >        name: cVE6D9OIkn5nTPPg
# #|         >        securityContext:
# #|         >          capabilities:
# #|         >            drop:
# #|         >            - CAP_FOWNER
# #|         >            - CAP_SETFCAP
# #\^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
# # [teardown]
# # podman pod rm -t 0 --all --force --ignore
# # podman rm -t 0 --all --force --ignore
# d00b1b47374c20e71692a5deae513ff3426eb22bb774c81bd0dea3e337aab7c9
# # podman network prune --force
# # podman volume rm -a -f

This test is expected to not output this warning message.

podman/test/system/710-kube.bats

Line 36 in cb18a33

assert "$output" !~ "Kubernetes only allows 63 characters"

However, the warning message Kubernetes only allows 63 characters is always output.
I don't know why this warning message isn't output in CI environment.

# podman container create --cap-drop fowner --cap-drop setfcap --name test quay.io/libpod/testimage:20221018 top
0e3cb96bb0087b2cf4e58e1bd318f63d636b331d805708ce8e28f7233c5d863f

# podman kube generate test
WARN[0000] Truncation Annotation: "sha256:36d93dbf1cf35346effba973a726cd4ba19a967b68c3ce03bde0c1127015f2f9" to "sha256:36d93dbf1cf35346effba973a726cd4ba19a967b68c3ce03bde0c112": Kubernetes only allows 63 characters
# Save the output of this file and use kubectl create -f to import
# it into Kubernetes.
#
# Created with podman-4.5.0-dev
apiVersion: v1
kind: Pod

... snip ...

podman version

Client:       Podman Engine
Version:      4.5.0-dev
API Version:  4.5.0-dev
Go Version:   go1.19.5
Git Commit:   cb18a33f63de2a25d2352fc6a31e38908ac5b647
Built:        Fri Mar 24 09:38:57 2023
OS/Arch:      linux/amd64

podman info

host:
  arch: amd64
  buildahVersion: 1.30.0-dev
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - hugetlb
  - pids
  - misc
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.5-1.fc37.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.5, commit: '
  cpuUtilization:
    idlePercent: 92.76
    systemPercent: 0.01
    userPercent: 7.23
  cpus: 12
  databaseBackend: boltdb
  distribution:
    distribution: fedora
    variant: server
    version: "37"
  eventLogger: journald
  hostname: fedora37
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 6.0.7-301.fc37.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 4759842816
  memTotal: 8327143424
  networkBackend: cni
  ociRuntime:
    name: crun
    package: crun-1.8-1.fc37.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.8
      commit: 0356bf4aff9a133d655dc13b1d9ac9424706cac4
      rundir: /run/user/0/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.0-8.fc37.x86_64
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.3
  swapFree: 8326737920
  swapTotal: 8326737920
  uptime: 258h 20m 52.00s (Approximately 10.75 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /usr/share/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 0
    stopped: 1
  graphDriverName: overlay
  graphOptions:
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 106229137408
  graphRootUsed: 20633800704
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "true"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 1
  runRoot: /run/containers/storage
  transientStore: false
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 4.5.0-dev
  Built: 1679618337
  BuiltTime: Fri Mar 24 09:38:57 2023
  GitCommit: cb18a33f63de2a25d2352fc6a31e38908ac5b647
  GoVersion: go1.19.5
  Os: linux
  OsArch: linux/amd64
  Version: 4.5.0-dev

[Luap99]

I can confirm that it also fails for me locally on latest main (I checked with hack/bats 710:"podman kube generate - container")

Check was added in ef3f098
cc @rhatdan

[rhatdan]

@edsantiago or @vrothberg might know what is going on here. I thought we talked about dropping this warning down to info, since there is nothing users can do about it.

[edsantiago]

@rhatdan you're the one who added that !~ blah blah 63 test line? #17144

[edsantiago]

This is failing for me (f37 laptop) all the way back to v4.4.0 and beyond. To me that suggests that the problem is in non-podman: maybe containers.conf or crun or something else. I have no idea how to track this down.

[edsantiago]

Found the reason CI passes: podman load changes things such that the warning is not emitted.

$ bin/podman rmi -a
$ hack/bats --rootless 710:container 
FAILS!

$ bin/podman save -q quay.io/libpod/testimage:20221018 -o /tmp/foo.tar
$ bin/podman rmi -a
...
$ bin/podman load -q -i /tmp/foo.tar
Loaded image: quay.io/libpod/testimage:20221018
$ hack/bats --rootless 710:container
PASS!

$ bin/podman rmi -a
...
$ !hack
FAIL! Because it re-pulled the image

As for why podman save/load changes things, that's beyond my ken but I'm willing to bet a nickel that it has to do with #3761

[sstosh]

Thank you for your reply.
I understand.

When podman saves the container image with --format=docker-archive (default), Annotations fields are not saved.
Therefore, podman doesn't output the warning message and passes the test.

--format=docker-archive (default)

# podman pull -q quay.io/libpod/testimage:20221018
f5a99120db6452661930a1db3bf7390eec9b963f5f62c068fa32dc1d550afad3

# podman save -q quay.io/libpod/testimage:20221018 -o /tmp/test.tar
# podman rmi -a
Untagged: quay.io/libpod/testimage:20221018
Deleted: f5a99120db6452661930a1db3bf7390eec9b963f5f62c068fa32dc1d550afad3
# podman load -q -i /tmp/test.tar
Loaded image: quay.io/libpod/testimage:20221018

# podman inspect --format={{.Annotations}} quay.io/libpod/testimage:20221018
map[]

# podman container create --cap-drop fowner --cap-drop setfcap --name test quay.io/libpod/testimage:20221018 top
c122d4519b5775ca61a7edfbf8b3a7ae316832f6e45d884febdac710dd956ab6
# podman kube generate test
# Save the output of this file and use kubectl create -f to import
# it into Kubernetes.
#
# Created with podman-4.5.0-dev

... snip ...

--format=oci-archive

# podman pull -q quay.io/libpod/testimage:20221018
f5a99120db6452661930a1db3bf7390eec9b963f5f62c068fa32dc1d550afad3

# podman save -q --format=oci-archive quay.io/libpod/testimage:20221018 -o /tmp/oci.tar
# podman rmi -a
Untagged: quay.io/libpod/testimage:20221018
Deleted: f5a99120db6452661930a1db3bf7390eec9b963f5f62c068fa32dc1d550afad3
# podman load -q -i /tmp/oci.tar
Loaded image: quay.io/libpod/testimage:20221018

# podman inspect --format={{.Annotations}} quay.io/libpod/testimage:20221018
map[org.opencontainers.image.base.digest:sha256:36d93dbf1cf35346effba973a726cd4ba19a967b68c3ce03bde0c1127015f2f9 org.opencontainers.image.base.name:localhost/interim-image:latest]

# podman container create --cap-drop fowner --cap-drop setfcap --name test quay.io/libpod/testimage:20221018 top
211e064acc2d7ccdf676fcc3c2633c5e2c348beb960fc3a011dc21ac5095afe4
# podman kube generate test
WARN[0000] Truncation Annotation: "sha256:36d93dbf1cf35346effba973a726cd4ba19a967b68c3ce03bde0c1127015f2f9" to "sha256:36d93dbf1cf35346effba973a726cd4ba19a967b68c3ce03bde0c112": Kubernetes only allows 63 characters
# Save the output of this file and use kubectl create -f to import
# it into Kubernetes.
#
# Created with podman-4.5.0-dev

... snip ...

AFAICS, Annotations fields are removed in the following save command:

podman/test/system/120-load.bats

Line 171 in 905dc6d

run_podman save $iid -o $archive

podman/test/system/120-load.bats

Line 195 in 905dc6d

run_podman save $img_name -o $archive

podman/test/system/120-load.bats

Line 215 in 905dc6d

run_podman save $img_name -o $archive

Should we fix like save --format=oci-archive ?

[edsantiago]

Thank you for the explanation! Yes, we need to add tests for save --format. (later). I also, much more importantly, need to fix the 120-load test file so it never, ever leaves a reloaded image. So it always pulls a fresh one. But of course we can't do that until the Kubernetes-63 bug gets fixed.

[Luap99]

possibly related containers/image#1888
cc @mtrmac

[mtrmac]

No; containers/image#1888 is for annotations on layers, these annotations are on the top (per-arch) manifest level.

The cause is that docker-archive just doesn’t store anything close to the original version of a manifest, and there isn’t any annotations field defined.

[mtrmac]

# metadata:
#   annotations:
#     io.podman.annotations.ulimit: nofile=1048576:1048576
#     org.opencontainers.image.base.digest/cVE6D9OIkn5nTPPg: sha256:36d93dbf1cf35346effba973a726cd4ba19a967b68c3ce03bde0c112
#     org.opencontainers.image.base.name/cVE6D9OIkn5nTPPg: localhost/interim-image:latest

Um, why are annotations from the image’s manifest copied into the Pod’s annotations in the first place? Is there some specific reason to do that? The way I think about it, annotations are a general concept of an escape hatch to extend a format; that doesn’t mean that annotations that extend one format are a valid extension of another format — if anything, it means the opposite, and it seems to me copying the same annotations into multiple places is much more likely to confuse users (about which of these annotations actually makes any difference) than to help.

(From an extremely brief skim, it seems that the image annotations are all copied into the container runtime annotations (which I don’t understand, but, fine) during container creation; and then podman kube generate copies all of those annotations into the Pod annotations. Is that so? And if so, is that correct?

I’d expect the image-originated annotations to not be copied into the Kube ones, because they can be again retrieved from the image to pass to the container runtime, and just updating the image would naturally update those container runtime annotations.

But I’m very unfamiliar with the code and data flow.)

[edsantiago]

Looks like this was fixed in #18542, so I'm closing as resolved.

However, I haven't forgotten about the problem with podman load screwing up digests and annotations. I'm back to working on a way to mitigate that.

Thanks again for reporting this.