Podman healthchecks fail -- When executed by systemd, pass when ran manually?

[Aldenar]

Hello,

I am completely new to Podman, trying to migrate to it over from Docker, for its native support of running under unprivileged users.

Right now, I am having issue with its health check mechanism - I already figured out that periodic health checks are implemented through transient systemd timers manually running the podman healthcheck run *container-id* command.

All of these automatic health checks fail. However, if I run the very same command there is in the transient unit manually, the health check passes fine!

I'm using the following compose file to start up a sample container to debug the issue with:

version: '2.4'
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.10.1
    volumes:
      - ./data/elasticsearch:/usr/share/elasticsearch/data
    environment:
      - "ES_JAVA_OPTS=-Dlog4j2.formatMsgNoLookups=true"
      - "bootstrap.memory_lock=true"
      - "discovery.type=single-node"
      - "logger.level=INFO"
      - "xpack.security.enabled=true"
      - "ELASTIC_PASSWORD=foobar"
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536
        hard: 65536
    healthcheck:
      test: ["CMD", "curl", "-s" ,"-f", "http://elastic:foobar@localhost:9200/_cat/health"]
      interval: 5s
    restart: always

System info:

host:
  arch: amd64
  buildahVersion: 1.30.0
  cgroupControllers:
  - cpu
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon_2:2.1.7-0debian9999+obs15.124_amd64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.7, commit: '
  cpuUtilization:
    idlePercent: 94.13
    systemPercent: 0.96
    userPercent: 4.9
  cpus: 4
  databaseBackend: boltdb
  distribution:
    codename: bookworm
    distribution: debian
    version: "12"
  eventLogger: journald
  hostname: myHostname
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 6.1.0-10-amd64
  linkmode: dynamic
  logDriver: journald
  memFree: 2701340672
  memTotal: 8326729728
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: crun_101:1.8.6-0debian9999+obs57.8_amd64
    path: /usr/bin/crun
    version: |-
      crun version 1.8.6
      commit: 73f759f4a39769f60990e7d225f561b4f4f06bcf
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns_1.2.0-1_amd64
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.4
  swapFree: 913903616
  swapTotal: 1070571520
  uptime: 48h 10m 26.00s (Approximately 2.00 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /home/rootless/.config/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 1
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/rootless/.local/share/containers/storage
  graphRootAllocated: 52723159040
  graphRootUsed: 863649792
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 1
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /home/rootless/.local/share/containers/storage/volumes
version:
  APIVersion: 4.5.1
  Built: 0
  BuiltTime: Thu Jan  1 01:00:00 1970
  GitCommit: ""
  GoVersion: go1.19.8
  Os: linux
  OsArch: linux/amd64
  Version: 4.5.1

Other edits made to podman config:

• Changed default network backend to netavark for native container name resolve (/etc/containers/containers.conf)
• Added docker.io to the list of unqualified-search-registries

[Luap99]

What does fail mean? Can you provide error messages?

[Aldenar]

That's the thing!

Running the podman healthcheck run manually outputs nothing. Just returns (with 0)

The sysd automated healthchecks return 1, and the string "unhealthy". Nothing more.