conmon / healthcheck / podman 4.6.0

[flyingfishflash]

Hi - ever since updating to podman 4.6.0, I'm seeing the following error in my journal:

Jul 28 20:24:46 nas conmon\[14584\]: conmon 6d051721c0cc878fdb72 <error>: Unable to send container stderr message to parent Broken pipe

This only occurs on containers that have healthchecks and occurs on each healthcheck, but doesn't seem otherwise negatively affect anything.

This is occurring on rootless containers.
The containers are generated via systemd / quadlets

Does anyone have any idea of where to investigate further. I don't anything else in the logs that seems meaningful.

[container-runner@nas podman]$ podman info
host:
  arch: amd64
  buildahVersion: 1.31.0
  cgroupControllers:
  - cpu
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: /usr/bin/conmon is owned by conmon 1:2.1.7-1
    path: /usr/bin/conmon
    version: 'conmon version 2.1.7, commit: f633919178f6c8ee4fb41b848a056ec33f8d707d'
  cpuUtilization:
    idlePercent: 99.21
    systemPercent: 0.33
    userPercent: 0.46
  cpus: 24
  databaseBackend: boltdb
  distribution:
    distribution: arch
    version: unknown
  eventLogger: journald
  freeLocks: 1714
  hostname: nas
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 4005
      size: 1
    - container_id: 1
      host_id: 427680
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 4005
      size: 1
    - container_id: 1
      host_id: 427680
      size: 65536
  kernel: 6.4.6-arch1-1
  linkmode: dynamic
  logDriver: journald
  memFree: 25959706624
  memTotal: 33447845888
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: /usr/lib/podman/aardvark-dns is owned by aardvark-dns 1.7.0-1
      path: /usr/lib/podman/aardvark-dns
      version: aardvark-dns 1.7.0
    package: /usr/lib/podman/netavark is owned by netavark 1.7.0-1
    path: /usr/lib/podman/netavark
    version: netavark 1.7.0
  ociRuntime:
    name: crun
    package: /usr/bin/crun is owned by crun 1.8.5-1
    path: /usr/bin/crun
    version: |-
      crun version 1.8.5
      commit: b6f80f766c9a89eb7b1440c0a70ab287434b17ed
      rundir: /run/user/4005/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  pasta:
    executable: ""
    package: ""
    version: ""
  remoteSocket:
    path: /run/user/4005/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /etc/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: /usr/bin/slirp4netns is owned by slirp4netns 1.2.0-1
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.4
  swapFree: 0
  swapTotal: 0
  uptime: 0h 19m 58.00s
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries: {}
store:
  configFile: /home/container-runner/.config/containers/storage.conf
  containerStore:
    number: 13
    paused: 0
    running: 13
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/container-runner/.local/share/containers/storage
  graphRootAllocated: 490304405504
  graphRootUsed: 74427219968
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 38
  runRoot: /run/user/4005/containers
  transientStore: false
  volumePath: /home/container-runner/.local/share/containers/storage/volumes
version:
  APIVersion: 4.6.0
  Built: 1689943108
  BuiltTime: Fri Jul 21 08:38:28 2023
  GitCommit: 38e6fab9664c6e59b66e73523b307a56130316ae-dirty
  GoVersion: go1.20.6
  Os: linux
  OsArch: linux/amd64
  Version: 4.6.0

[rhatdan]

@mheon PTAL

[mheon]

This should be re-filed as a bug report - seems like we broke something in Conmon re: healthchecks last release. I'll try and verify if it actually broke anything tomorrow, but my guess is that if it did it's only related to displaying error messages for healthchecks, and it might not even be that - could just be an extraneous warning.

[leleobhz]

I confirm this issue happening on Fedora 38 using systemd unit for non rootless container on same scenario: Container with healthcheck.

Infos:

Excerpt of journalctl --boot=0 | grep -E '(conmon|podman)':

ago 08 10:15:29 miriam systemd[1]: Started 009bd81fcec22595226dc9f04ef06d6aa551c3758501697c4bc3c05347ae739f.service - /usr/bin/podman healthcheck run 009bd81fcec22595226dc9f04ef06d6aa551c3758501697c4bc3c05347ae739f.
ago 08 10:15:29 miriam podman[50278]: 2023-08-08 10:15:29.410535446 -0300 -03 m=+0.101651773 container health_status 009bd81fcec22595226dc9f04ef06d6aa551c3758501697c4bc3c05347ae739f (image=quay.io/zenithtecnologia/zerotier-docker:dev, name=zerotier, health_status=healthy, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, description=Container with the openssl binary, giving ability to work with cryptographic keys and certificates needed for web servers., org.zerotier.version=, vcs-ref=20b10499f1a156388a02509bfc295a5abf80b71c, architecture=x86_64, quay.expires-after=never, vendor=Red Hat, Inc., PODMAN_SYSTEMD_UNIT=zerotier-one-podman.service, io.buildah.version=1.23.1, build-date=2023-07-27T14:12:52, vcs-type=git, version=9.2, distribution-scope=public, summary=ZeroTier - a smart programmable Ethernet switch for planet Earth., com.redhat.component=openssl-container, maintainer=Zenith Tecnologia <[email protected]>, release=10, io.k8s.display-name=zerotier, name=zerotier, url=https://github.com/ZenithTecnologia/zerotier-docker, io.k8s.description=This container runs Zerotier - a smart programmable Ethernet switch for planet Earth.)
ago 08 10:15:29 miriam conmon[50297]: conmon 009bd81fcec22595226d <error>: Unable to send container stderr message to parent Broken pipe
ago 08 10:15:29 miriam podman[50278]: 2023-08-08 10:15:29.433546714 -0300 -03 m=+0.124663011 container exec_died 009bd81fcec22595226dc9f04ef06d6aa551c3758501697c4bc3c05347ae739f (image=quay.io/zenithtecnologia/zerotier-docker:dev, name=zerotier, description=Container with the openssl binary, giving ability to work with cryptographic keys and certificates needed for web servers., version=9.2, distribution-scope=public, com.redhat.component=openssl-container, io.k8s.display-name=zerotier, org.zerotier.version=, PODMAN_SYSTEMD_UNIT=zerotier-one-podman.service, maintainer=Zenith Tecnologia <[email protected]>, vcs-ref=20b10499f1a156388a02509bfc295a5abf80b71c, vcs-type=git, vendor=Red Hat, Inc., io.buildah.version=1.23.1, url=https://github.com/ZenithTecnologia/zerotier-docker, build-date=2023-07-27T14:12:52, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, name=zerotier, release=10, summary=ZeroTier - a smart programmable Ethernet switch for planet Earth., architecture=x86_64, io.k8s.description=This container runs Zerotier - a smart programmable Ethernet switch for planet Earth., quay.expires-after=never)

podman info:

host:
  arch: amd64
  buildahVersion: 1.31.0
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - hugetlb
  - pids
  - rdma
  - misc
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.7-2.fc38.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.7, commit: '
  cpuUtilization:
    idlePercent: 73.78
    systemPercent: 3.91
    userPercent: 22.31
  cpus: 16
  databaseBackend: boltdb
  distribution:
    distribution: fedora
    variant: workstation
    version: "38"
  eventLogger: journald
  freeLocks: 2030
  hostname: miriam
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 6.4.7-200.fc38.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 13742452736
  memTotal: 33310380032
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.7.0-1.fc38.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.7.0
    package: netavark-1.7.0-1.fc38.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.7.0
  ociRuntime:
    name: crun
    package: crun-1.8.6-1.fc38.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.8.6
      commit: 73f759f4a39769f60990e7d225f561b4f4f06bcf
      rundir: /run/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20230625.g32660ce-1.fc38.x86_64
    version: |
      pasta 0^20230625.g32660ce-1.fc38.x86_64
      Copyright Red Hat
      GNU Affero GPL version 3 or later <https://www.gnu.org/licenses/agpl-3.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.0-12.fc38.x86_64
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.3
  swapFree: 8589930496
  swapTotal: 8589930496
  uptime: 1h 7m 1.00s (Approximately 0.04 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /usr/share/containers/storage.conf
  containerStore:
    number: 2
    paused: 0
    running: 1
    stopped: 1
  graphDriverName: overlay
  graphOptions:
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 118329704448
  graphRootUsed: 70370422784
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "true"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 3
  runRoot: /run/containers/storage
  transientStore: false
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 4.6.0
  Built: 1689942206
  BuiltTime: Fri Jul 21 09:23:26 2023
  GitCommit: ""
  GoVersion: go1.20.6
  Os: linux
  OsArch: linux/amd64
  Version: 4.6.0

screenfetch -d 'distro;+cpu;+kernel;+gpu;+mem':

           /:-------------:\          OS: Fedora 
        :-------------------::        Kernel: x86_64 Linux 6.4.7-200.fc38.x86_64
      :-----------/shhOHbmp---:\      CPU: Intel Core i7-7820X @ 16x 4.3GHz [31.0°C]
    /-----------omMMMNNNMMD  ---:     GPU: NVIDIA GeForce RTX 2060
   :-----------sMMMMNMNMP.    ---:    RAM: 6917MiB / 31767MiB
  :-----------:MMMdP-------    ---\  
 ,------------:MMMd--------    ---:  
 :------------:MMMd-------    .---:  
 :----    oNMMMMMMMMMNho     .----:  
 :--     .+shhhMMMmhhy++   .------/  
 :-    -------:MMMd--------------:   
 :-   --------/MMMd-------------;    
 :-    ------/hMMMy------------:     
 :-- :dMNdhhdNMMNo------------;      
 :---:sdNMMMMNds:------------:       
 :------:://:-------------::         
 :---------------------://           

[duritong]

I do see the same message, when I run a systemd-timer, that executes a command in a running rootless container:

Relevant part of the script:

[...]
SYNAPSE_ROOMS_TO_COMPRESS=$(podman exec -it postgres psql -P pager=off -U synapse synapse -t -A -c 'SELECT array_to_json(array_agg(row_to_json (r))) FROM (SELECT room_id, count(*) AS count FROM state_groups_state GROUP BY room_id HAVING count(*) > 50000 ORDER BY count DESC) r;' |  jq -r 'sort_by(.count) | .[].room_id')
[...]

journal has:

Aug 10 11:29:57 example con-example-matrix-cron-cleanup-daily[1619777]: ++ podman exec -it postgres psql -P pager=off -U synapse synapse -t -A -c 'SELECT array_to_json(array_agg(row_to_json (r))) FROM (SELECT room_id, count(*) AS count FROM state_groups_state GROUP BY room_id HAVING count(*) > 50000 ORDER BY count DESC) r;'
Aug 10 11:29:57 example con-example-matrix-cron-cleanup-daily[1619778]: ++ jq -r 'sort_by(.count) | .[].room_id'
Aug 10 11:30:20 example conmon[1619798]: conmon 399e6a5d601d63ce9323 <nwarn>: stdio_input read failed Input/output error
Aug 10 11:30:20 example conmon[1619798]: conmon 399e6a5d601d63ce9323 <error>: Unable to send container stderr message to parent Broken pipe

But the command itself runs fine.

[Beastie71]

Did this get put in as a bug? If so I can provide additional info if needed as I am seeing it fill my journal.

[leleobhz]

Did this get put in as a bug? If so I can provide additional info if needed as I am seeing it fill my journal.

Hello @Beastie71

I filled it at containers/conmon#438

[HerrFrutti]

conmon version 2.1.7
podman version 4.6.1

and this fills my logs:

conmon[11328]: conmon 62de48d99d5858ab629f <error>: Unable to send container stderr message to parent Broken pipe