podman and docker produce slightly different output when using "history --format" and http://localhost/v1.24/images/${img}/history

[eriksjolund]

Is any of the differences between Podman and Docker shown below a bug?

---

I performed some tests on Fedora CoreOS next (fedora-coreos-37.20230303.1.0-qemu.aarch64.qcow2)

First some preparation steps:

1. sudo -i
2. useradd test1
3. usermod -aG docker test1
4. machinectl shell @test1
5. podman pull alpine:3.17.2
6. docker pull alpine:3.17.2
7. run podman images and detect the image id. Record the result in a shell variable
   img=d74e625d9115
8. systemctl --user start podman.socket

Test 1

[test1@localhost ~]$ podman history --format '{{.CreatedAt}}' alpine:3.17.2
2023-02-10 21:24:08 +0000 UTC
2023-02-10 21:24:08 +0000 UTC
[test1@localhost ~]$ docker history --format '{{.CreatedAt}}' alpine:3.17.2
2023-02-10T21:24:08Z
2023-02-10T21:24:08Z
[test1@localhost ~]$ 

Observation 1: Podman and Docker produce different output

Test 2

[test1@localhost ~]$ podman history --format '{{.Size}}' alpine:3.17.2
0 B
7.75 MB
[test1@localhost ~]$ docker history --format '{{.Size}}' alpine:3.17.2
0B
7.46MB
[test1@localhost ~]$ 

Observation 2: Podman inserts an extra space after the number

Observation 3:
7.46 != 7.75

(maybe the sizes are not expected to match?)

Test 3

[test1@localhost ~]$ curl -s --unix-socket $XDG_RUNTIME_DIR/podman/podman.sock http://localhost/v1.24/images/${img}/history | grep -o -P '.{0,7}missing.{0,7}'
"\u003cmissing\u003e"
[test1@localhost ~]$ curl -s --unix-socket /var/run/docker.sock http://localhost/v1.24/images/${img}/history | grep -o -P '.{0,7}missing.{0,7}'
"Id":"<missing>","Siz
[test1@localhost ~]$

[test1@localhost ~]$ curl -s --unix-socket $XDG_RUNTIME_DIR/podman/podman.sock  http://localhost/v1.24/images/${img}/history | jq .[1].Id
"<missing>"
[test1@localhost ~]$ curl -s --unix-socket /var/run/docker.sock http://localhost/v1.24/images/${img}/history |  jq .[1].Id
"<missing>"
[test1@localhost ~]$ 

Observation 4: Podman and Docker use different ways to encode the characters < and > when writing <missing>, but it seems that this difference does not influence the JSON interpretation (because the jq command prints the same result).

Test 4

[test1@localhost ~]$ curl -s --unix-socket $XDG_RUNTIME_DIR/podman/podman.sock  http://localhost/v1.24/images/${img}/history | jq .[1].Size
7745536
[test1@localhost ~]$ curl -s --unix-socket /var/run/docker.sock http://localhost/v1.24/images/${img}/history |  jq .[1].Size
7458929
[test1@localhost ~]$ 

Observation 5: The sizes do not match. (This observation is more or less the same as Observation 3).

Test 5

[test1@localhost ~]$ curl -s --unix-socket /var/run/docker.sock http://localhost/v1.24/images/${img}/history |  jq .[0].Tags
[
  "alpine:3.17.2"
]
[test1@localhost ~]$ curl -s --unix-socket $XDG_RUNTIME_DIR/podman/podman.sock  http://localhost/v1.24/images/${img}/history | jq .[0].Tags
null
[test1@localhost ~]$ 

Observation 6: Only docker has a tag.

Test 6

[test1@localhost ~]$ curl -s --unix-socket /var/run/docker.sock http://localhost/v1.24/images/${img}/history | jq .[0].Id
"sha256:d74e625d91152966d38fe8a62c60daadb96d4b94c1a366de01fab5f334806239"
[test1@localhost ~]$ curl -s --unix-socket $XDG_RUNTIME_DIR/podman/podman.sock  http://localhost/v1.24/images/${img}/history | jq .[0].Id
"d74e625d91152966d38fe8a62c60daadb96d4b94c1a366de01fab5f334806239"
[test1@localhost ~]$ 

Observation 7: Only docker prints sha256:

Extra info 1

For completeness here are the same queries but with the full result shown. I added --sort-keys to jq.

[test1@localhost ~]$ curl -s --unix-socket $XDG_RUNTIME_DIR/podman/podman.sock http://localhost/v1.24/images/${img}/history | jq --sort-keys . > /tmp/podman.txt
[test1@localhost ~]$ curl -s --unix-socket /var/run/docker.sock http://localhost/v1.24/images/${img}/history | jq --sort-keys . > /tmp/docker.txt

[test1@localhost ~]$ cat /tmp/podman.txt 
[
  {
    "Comment": "",
    "Created": 1676064248,
    "CreatedBy": "/bin/sh -c #(nop)  CMD [\"/bin/sh\"]",
    "Id": "d74e625d91152966d38fe8a62c60daadb96d4b94c1a366de01fab5f334806239",
    "Size": 0,
    "Tags": null
  },
  {
    "Comment": "",
    "Created": 1676064248,
    "CreatedBy": "/bin/sh -c #(nop) ADD file:9bd9ea42a9f3bdc769e80c6b8a4b117d65f73ae68e155a6172a1184e7ac8bcc1 in / ",
    "Id": "<missing>",
    "Size": 7745536,
    "Tags": null
  }
]

[test1@localhost ~]$ cat /tmp/docker.txt 
[
  {
    "Comment": "",
    "Created": 1676064248,
    "CreatedBy": "/bin/sh -c #(nop)  CMD [\"/bin/sh\"]",
    "Id": "sha256:d74e625d91152966d38fe8a62c60daadb96d4b94c1a366de01fab5f334806239",
    "Size": 0,
    "Tags": [
      "alpine:3.17.2"
    ]
  },
  {
    "Comment": "",
    "Created": 1676064248,
    "CreatedBy": "/bin/sh -c #(nop) ADD file:9bd9ea42a9f3bdc769e80c6b8a4b117d65f73ae68e155a6172a1184e7ac8bcc1 in / ",
    "Id": "<missing>",
    "Size": 7458929,
    "Tags": null
  }
]

--- /tmp/docker.txt	2023-03-11 09:44:26.596924156 +0000
+++ /tmp/podman.txt	2023-03-11 09:44:21.916530671 +0000
@@ -3,18 +3,16 @@
     "Comment": "",
     "Created": 1676064248,
     "CreatedBy": "/bin/sh -c #(nop)  CMD [\"/bin/sh\"]",
-    "Id": "sha256:d74e625d91152966d38fe8a62c60daadb96d4b94c1a366de01fab5f334806239",
+    "Id": "d74e625d91152966d38fe8a62c60daadb96d4b94c1a366de01fab5f334806239",
     "Size": 0,
-    "Tags": [
-      "alpine:3.17.2"
-    ]
+    "Tags": null
   },
   {
     "Comment": "",
     "Created": 1676064248,
     "CreatedBy": "/bin/sh -c #(nop) ADD file:9bd9ea42a9f3bdc769e80c6b8a4b117d65f73ae68e155a6172a1184e7ac8bcc1 in / ",
     "Id": "<missing>",
-    "Size": 7458929,
+    "Size": 7745536,
     "Tags": null
   }
 ]

Extra info 2

The commands above were run with podman 4.4.1 on Fedora CoreOS by using qemu on a macOS laptop (operating system: Ventura 13.2.1).

I have another computer (arch: amd64) running Fedora 37. I re-run the Podman commands there with podman 4.4.2.
The container image ID changed and the container image size changed but nothing else.

About the system:

Click me to show `docker version` and `podman version` and similar commands

[test1@localhost ~]$ docker version
Client:
 Version:           20.10.23
 API version:       1.41
 Go version:        go1.19.5
 Git commit:        %{shortcommit_cli}
 Built:             Sun Jan 29 17:38:04 2023
 OS/Arch:           linux/arm64
 Context:           default
 Experimental:      true

Server:
 Engine:
  Version:          20.10.23
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.19.5
  Git commit:       %{shortcommit_moby}
  Built:            Sun Jan 29 17:38:04 2023
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.6.15
  GitCommit:        
 runc:
  Version:          1.1.4
  GitCommit:        
 docker-init:
  Version:          0.19.0
  GitCommit:        

[test1@localhost ~]$ podman version
Client:       Podman Engine
Version:      4.4.1
API Version:  4.4.1
Go Version:   go1.19.5
Built:        Fri Feb 17 10:25:38 2023
OS/Arch:      linux/arm64

[test1@localhost ~]$ rpm -q podman 
podman-4.4.1-3.fc37.aarch64

[test1@localhost ~]$ podman info
host:
  arch: arm64
  buildahVersion: 1.29.0
  cgroupControllers:
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.6-3.fc37.aarch64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.6, commit: '
  cpuUtilization:
    idlePercent: 99.02
    systemPercent: 0.55
    userPercent: 0.43
  cpus: 1
  distribution:
    distribution: fedora
    variant: coreos
    version: "37"
  eventLogger: journald
  hostname: localhost.localdomain
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1001
      size: 1
    - container_id: 1
      host_id: 589824
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1001
      size: 1
    - container_id: 1
      host_id: 589824
      size: 65536
  kernel: 6.1.14-200.fc37.aarch64
  linkmode: dynamic
  logDriver: journald
  memFree: 1437827072
  memTotal: 2050260992
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: crun-1.8.1-1.fc37.aarch64
    path: /usr/bin/crun
    version: |-
      crun version 1.8.1
      commit: f8a096be060b22ccd3d5f3ebe44108517fbf6c30
      rundir: /run/user/1001/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/1001/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.0-8.fc37.aarch64
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.3
  swapFree: 0
  swapTotal: 0
  uptime: 0h 56m 19.00s
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /var/home/test1/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/home/test1/.local/share/containers/storage
  graphRootAllocated: 10132369408
  graphRootUsed: 1586069504
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 1
  runRoot: /run/user/1001/containers
  transientStore: false
  volumePath: /var/home/test1/.local/share/containers/storage/volumes
version:
  APIVersion: 4.4.1
  Built: 1676629538
  BuiltTime: Fri Feb 17 10:25:38 2023
  GitCommit: ""
  GoVersion: go1.19.5
  Os: linux
  OsArch: linux/arm64
  Version: 4.4.1

[rhatdan]

I would say open issues on all of these except for the size ones. No idea why the sizes differ, but I don't believe they are bugs. The others where the output is different, I believe are bugs, but the maintainers of that section may differ.
Thanks for doing this research.

[eriksjolund]

OK, I created these issues

Observation 1:

• podman history --format '{{.CreatedAt}}' IMG prints different output than docker #17768

Observation 2:

• podman history --format '{{.Size}}' IMG outputs an extra space (compared to docker) #17767

Observation 4:

• REST API: "\u003cmissing\u003e" instead of "<missing>" in returned Id attribute ( /v1.24/images/${img}/history ) #17769

Observation 6:

• REST API: missing tag from /v1.24/images/${img}/history #17763

Observation 7:

• REST API: missing string "sha256" in returned Id attribute ( /v1.24/images/${img}/history ) #17762