Fix startup health check command behavior by arsenalzp · Pull Request #27857 · podman-container-tools/podman

arsenalzp · 2026-01-03T20:23:25Z

This PR fixes #27724.
The issue is that health check command is ignored when options --health-start-period, --health-startup-interval are set, but --health-startup-cmd is not present.
In contrast to Docker which uses health check command for the startup health check.

Checklist

Ensure you have completed the following checklist for your pull request to be reviewed:

Certify you wrote the patch or otherwise have the right to pass it on as an open-source patch by signing all
commits. (git commit -s). (If needed, use git commit -s --amend). The author email must match
the sign-off email address. See CONTRIBUTING.md
for more information.
Referenced issues using Fixes: #00000 in commit message (if applicable)
Tests have been added/updated (or no tests are needed)
Documentation has been updated (or no documentation changes are needed)
All commits pass make validatepr (format/lint checks)
Release note entered in the section below (or None if no user-facing changes)

Does this PR introduce a user-facing change?

packit-as-a-service · 2026-01-04T16:24:24Z

[NON-BLOCKING] Packit jobs failed. @containers/packit-build please check. Everyone else, feel free to ignore.

TomSweeneyRedHat · 2026-01-15T22:37:54Z

I've restarted what appears to be flakey tests. The changes LGTM, but I definitely want a head nod from @mheon

arsenalzp · 2026-01-23T15:19:06Z

Hello @mheon,
Could you please be so kind to review this PR?

mheon · 2026-01-25T05:55:32Z

I think you need to add an alias for health-start-interval to health-startup-interval for Docker compatibility, and I don't see handling for the new field in image health check configs?

arsenalzp · 2026-01-26T09:21:53Z

I think you need to add an alias for health-start-interval to health-startup-interval for Docker compatibility, and I don't see handling for the new field in image health check configs?

Hello,
I agree with creating alias for parameters.
Regarding the image health check config - do you have any suggestion how it would be implemented in the good way? I will do that.
Thank you!

TheRealBecks · 2026-02-09T10:03:01Z

@mheon Any idea how to proceed here?

Honny1 · 2026-03-11T12:22:57Z

@arsenalzp I think Matt meant that the --start-period flag is set from the image (HEALTHCHECK command). Here is the code that performs the override of image hc: https://github.com/containers/podman/blob/f7aefa46bbe5f0d8b1cfb0ff7b9fcd5fee2a9545/pkg/specgen/generate/container.go#L65

Honny1 · 2026-04-29T08:20:45Z

@arsenalzp Any updates on this?

arsenalzp · 2026-04-30T14:27:31Z

@arsenalzp Any updates on this?

Hello,
Yes, I remember about this PR.
I will take a look at it soon!

arsenalzp · 2026-08-11T09:57:40Z

Hello,
It this PR still valid? Can I proceed work on it?

Honny1 · 2026-08-11T10:03:44Z

Yes

arsenalzp · 2026-08-11T10:13:49Z

It was a quite time ago so, as far as I understand, we need to implement overriding of an image health check value by one from CLI.

Signed-off-by: Oleksandr Krutko <alexander.krutko@gmail.com>

Honny1

I did review. My memory is rusty but based on converstion is needed:

adding --health-start-interval as an alias for --health-startup-interval for Docker compatibility.
the --start-period and other flags can be seted from the image (HEALTHCHECK command). Here is the code that performs the override of image hc: https://github.com/containers/podman/blob/f7aefa46bbe5f0d8b1cfb0ff7b9fcd5fee2a9545/pkg/specgen/generate/container.go#L65

Note for me: Next review check whole hc logic.

packit-as-a-service · 2026-08-22T12:09:07Z

[NON-BLOCKING] Packit jobs failed. @podman-container-tools/packit-jobs please check. Everyone else, feel free to ignore.

Signed-off-by: Oleksandr Krutko <alexander.krutko@gmail.com> remove unnecessary conversion in FillOutSpecGen func Signed-off-by: Oleksandr Krutko <alexander.krutko@gmail.com>

Honny1 · 2026-08-25T12:00:50Z

Well Docker's --health-start-interval is not a separate startup healthcheck. It simply runs the same health command at a different (shorter) interval during the start period. Podman's startup healthcheck (--health-startup-cmd) is a fundamentally different concept: an entirely separate command that must succeed before the main HC begins.

This PR mixes StartupHC and HC by creating a StartupHealthConfig using the regular --health-cmd when --health-startup-interval is set. This means the container will run the Podman StartupHC machinery (which has its own success/failure counters, transitions, etc.) rather than simply running the health command at a shorter interval during the start period.

I would not mix --health-start-period and --health-startup-interval.

arsenalzp · 2026-08-25T12:17:07Z

Hello, no this PR doesn't.
As we discussed in the linked issue, the main purpose of this PR was to use health check command for start-up health check purposes in case when start-up command wasn't set. So, if you set start-up command then it is used, otherwise common health check command is used.

The second issue the PR fixes - it is startup interval, which doesn't work. You can check corresponding issue link, there are plenty snippets which show how does Docker work and Podman doesn't.

Let's summarize:

health check command is used when star-up health check command wasn't set - done;
start-up health check interval in the start-up health check period is now working as expected;

Here is the observation

arsenalzp · 2026-08-25T20:26:13Z

@Honny1 let's take a look on the fix was done by my PR:

$ bin/podman run -d --name test --health-cmd 'stat /tmp/true' --health-start-period 12s --health-startup-interval 4s --stop-signal SIGTERM quay.io/libpod/testimage:20241011 sh -c 'sleep 8 && touch /tmp/true && sleep infinity'

and now check the health:

$ bin/podman inspect -f '{{json .State.Health}}' test | jq
{
  "Status": "starting",
  "FailingStreak": 0,
  "Log": [
    {
      "Start": "2026-08-25T22:47:38.734941826+03:00",
      "End": "2026-08-25T22:47:38.796719315+03:00",
      "ExitCode": 1,
      "Output": "stat: can't stat '/tmp/true': No such file or directory\n"
    },
    {
      "Start": "2026-08-25T22:47:43.378029581+03:00",
      "End": "2026-08-25T22:47:43.426019666+03:00",
      "ExitCode": 1,
      "Output": "stat: can't stat '/tmp/true': No such file or directory\n"
    }
  ]
}

and bit later:

$ bin/podman inspect -f '{{json .State.Health}}' test | jq
{
  "Status": "healthy",
  "FailingStreak": 0,
  "Log": [
    {
      "Start": "2026-08-25T22:47:38.734941826+03:00",
      "End": "2026-08-25T22:47:38.796719315+03:00",
      "ExitCode": 1,
      "Output": "stat: can't stat '/tmp/true': No such file or directory\n"
    },
    {
      "Start": "2026-08-25T22:47:43.378029581+03:00",
      "End": "2026-08-25T22:47:43.426019666+03:00",
      "ExitCode": 1,
      "Output": "stat: can't stat '/tmp/true': No such file or directory\n"
    },
    {
      "Start": "2026-08-25T22:47:48.791204551+03:00",
      "End": "2026-08-25T22:47:48.812022621+03:00",
      "ExitCode": 0,
      "Output": "  File: /tmp/true\n  Size: 0         \tBlocks: 0          IO Block: 4096   regular empty file\nDevice: 1000ach/1048748d\tInode: 2097359     Links: 1\nAccess: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)\nAccess: 2026-08-25 19:47:46.440184471 +0000\nModify: 2026-08-25 19:47:46.440184471 +0000\nChange: 2026-08-25 19:47:46.440184471 +0000\n"
    }
  ]
}

So, the container's start-up health is checked every 4 sec, then the container was ready in 8 sec (the file was created) and the start-up health check has been transited to common health check algorithm which runs every 30 sec:

$ bin/podman inspect -f '{{json .State.Health}}' test | jq
{
  "Status": "healthy",
  "FailingStreak": 0,
  "Log": [
    {
      "Start": "2026-08-25T22:51:25.65021046+03:00",
      "End": "2026-08-25T22:51:25.675402972+03:00",
      "ExitCode": 0,
      "Output": "  File: /tmp/true\n  Size: 0         \tBlocks: 0          IO Block: 4096   regular empty file\nDevice: 1000ach/1048748d\tInode: 2097359     Links: 1\nAccess: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)\nAccess: 2026-08-25 19:47:46.440184471 +0000\nModify: 2026-08-25 19:47:46.440184471 +0000\nChange: 2026-08-25 19:47:46.440184471 +0000\n"
    },
    {
      "Start": "2026-08-25T22:51:56.534258393+03:00",
      "End": "2026-08-25T22:51:56.584108198+03:00",
      "ExitCode": 0,
      "Output": "  File: /tmp/true\n  Size: 0         \tBlocks: 0          IO Block: 4096   regular empty file\nDevice: 1000ach/1048748d\tInode: 2097359     Links: 1\nAccess: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)\nAccess: 2026-08-25 19:47:46.440184471 +0000\nModify: 2026-08-25 19:47:46.440184471 +0000\nChange: 2026-08-25 19:47:46.440184471 +0000\n"
    },
    {
      "Start": "2026-08-25T22:52:27.685199688+03:00",
      "End": "2026-08-25T22:52:27.748304981+03:00",
      "ExitCode": 0,
      "Output": "  File: /tmp/true\n  Size: 0         \tBlocks: 0          IO Block: 4096   regular empty file\nDevice: 1000ach/1048748d\tInode: 2097359     Links: 1\nAccess: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)\nAccess: 2026-08-25 19:47:46.440184471 +0000\nModify: 2026-08-25 19:47:46.440184471 +0000\nChange: 2026-08-25 19:47:46.440184471 +0000\n"
    },
    {
      "Start": "2026-08-25T22:52:58.647056082+03:00",
      "End": "2026-08-25T22:52:58.694411579+03:00",
      "ExitCode": 0,
      "Output": "  File: /tmp/true\n  Size: 0         \tBlocks: 0          IO Block: 4096   regular empty file\nDevice: 1000ach/1048748d\tInode: 2097359     Links: 1\nAccess: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)\nAccess: 2026-08-25 19:47:46.440184471 +0000\nModify: 2026-08-25 19:47:46.440184471 +0000\nChange: 2026-08-25 19:47:46.440184471 +0000\n"
    },
    {
      "Start": "2026-08-25T22:53:29.720210717+03:00",
      "End": "2026-08-25T22:53:29.738156295+03:00",
      "ExitCode": 0,
      "Output": "  File: /tmp/true\n  Size: 0         \tBlocks: 0          IO Block: 4096   regular empty file\nDevice: 1000ach/1048748d\tInode: 2097359     Links: 1\nAccess: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)\nAccess: 2026-08-25 19:47:46.440184471 +0000\nModify: 2026-08-25 19:47:46.440184471 +0000\nChange: 2026-08-25 19:47:46.440184471 +0000\n"
    }
  ]
}

Now, let's set the start-up health command:

bin/podman run -d --name test --health-startup-cmd 'stat /tmp/true' --health-start-period 12s --health-startup-interval 4s --health-cmd 'echo Im healthy' --stop-signal SIGTERM quay.io/libpod/testimage:20241011 sh -c 'sleep 9 && touch /tmp/true && sleep infinity'

and then check the health:

$ bin/podman inspect -f '{{json .State.Health}}' test | jq
{
  "Status": "healthy",
  "FailingStreak": 0,
  "Log": [
    {
      "Start": "2026-08-25T23:06:22.813961771+03:00",
      "End": "2026-08-25T23:06:22.857790506+03:00",
      "ExitCode": 1,
      "Output": "stat: can't stat '/tmp/true': No such file or directory\n"
    },
    {
      "Start": "2026-08-25T23:06:27.383670962+03:00",
      "End": "2026-08-25T23:06:27.40040345+03:00",
      "ExitCode": 1,
      "Output": "stat: can't stat '/tmp/true': No such file or directory\n"
    },
    {
      "Start": "2026-08-25T23:06:32.659081492+03:00",
      "End": "2026-08-25T23:06:32.712440158+03:00",
      "ExitCode": 0,
      "Output": "Im healthy\n"
    },
    {
      "Start": "2026-08-25T23:07:03.388737924+03:00",
      "End": "2026-08-25T23:07:03.44015703+03:00",
      "ExitCode": 0,
      "Output": "Im healthy\n"
    }
  ]
}

That's exactly what was expected!

And now, let's take a look how does it work with the current algorithm (podman Version: 6.2.0-dev):

$ bin/podman run -d --name test --health-cmd 'stat /tmp/true' --health-start-period 12s --health-startup-interval 4s --stop-signal SIGTERM quay.io/libpod/testimage:20241011 sh -c 'sleep 8 && touch /tmp/true && sleep infinity'

let's check the health:

$ bin/podman inspect -f '{{json .State.Health}}' test | jq
{
  "Status": "healthy",
  "FailingStreak": 0,
  "Log": [
    {
      "Start": "2026-08-25T23:14:31.452404124+03:00",
      "End": "2026-08-25T23:14:31.494189713+03:00",
      "ExitCode": 1,
      "Output": "stat: can't stat '/tmp/true': No such file or directory\n"
    },
    {
      "Start": "2026-08-25T23:15:02.341189475+03:00",
      "End": "2026-08-25T23:15:02.392869195+03:00",
      "ExitCode": 0,
      "Output": "  File: /tmp/true\n  Size: 0         \tBlocks: 0          IO Block: 4096   regular empty file\nDevice: 1000ach/1048748d\tInode: 2097359     Links: 1\nAccess: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)\nAccess: 2026-08-25 20:14:39.322574021 +0000\nModify: 2026-08-25 20:14:39.322574021 +0000\nChange: 2026-08-25 20:14:39.322574021 +0000\n"
    }
  ]
}

As you may see the command runs every 30 sec, despite we set the start-up health interval to 4 sec.

Another discrepancy:

bin/podman run -d --name test --health-cmd /bin/false --health-start-period 12s --health-startup-interval 4s --stop-signal SIGTERM quay.io/libpod/testimage:20241011 sh -c 'sleep infinity'

let's check the health in 10 sec:

$ bin/podman inspect -f '{{json .State.Health}}' test | jq
{
  "Status": "starting",
  "FailingStreak": 2,
  "Log": [
    {
      "Start": "2026-08-25T23:21:47.457388804+03:00",
      "End": "2026-08-25T23:21:47.605835246+03:00",
      "ExitCode": 1,
      "Output": ""
    },
    {
      "Start": "2026-08-25T23:22:18.2958529+03:00",
      "End": "2026-08-25T23:22:18.387919946+03:00",
      "ExitCode": 1,
      "Output": ""
    },
    {
      "Start": "2026-08-25T23:22:48.907862571+03:00",
      "End": "2026-08-25T23:22:48.985084484+03:00",
      "ExitCode": 1,
      "Output": ""
    }
  ]
}

Hmm, the container is still starting, despite of the start-up period was set to 8 sec. The start-up health interval was to 4 sec, however the health check runs every 30 sec (default).

arsenalzp force-pushed the podman60_27724 branch 4 times, most recently from daf5321 to 07f6c53 Compare January 4, 2026 15:49

arsenalzp force-pushed the podman60_27724 branch from 07f6c53 to 609505e Compare January 4, 2026 18:17

baude reviewed Jan 8, 2026

View reviewed changes

Comment thread pkg/specgenutil/specgen.go

arsenalzp force-pushed the podman60_27724 branch 2 times, most recently from 1c2a037 to 1786799 Compare January 13, 2026 20:02

mheon reviewed Jan 25, 2026

View reviewed changes

Comment thread pkg/specgenutil/specgen.go

fix health-startup-cmd when it wasn't set

f791617

Signed-off-by: Oleksandr Krutko <alexander.krutko@gmail.com>

arsenalzp force-pushed the podman60_27724 branch from 1786799 to f791617 Compare August 12, 2026 13:30

Honny1 reviewed Aug 13, 2026

View reviewed changes

Comment thread pkg/specgenutil/specgen.go

Comment thread libpod/define/healthchecks.go

Comment thread libpod/healthcheck.go

add alias for health-startup-interval, add image healthcheck overrides

9f883f9

Signed-off-by: Oleksandr Krutko <alexander.krutko@gmail.com> remove unnecessary conversion in FillOutSpecGen func Signed-off-by: Oleksandr Krutko <alexander.krutko@gmail.com>

arsenalzp force-pushed the podman60_27724 branch from 5ebd6e3 to 9f883f9 Compare August 22, 2026 12:10

Uh oh!

Conversation

arsenalzp commented Jan 3, 2026

Checklist

Does this PR introduce a user-facing change?

Uh oh!

packit-as-a-service Bot commented Jan 4, 2026

Uh oh!

Uh oh!

TomSweeneyRedHat commented Jan 15, 2026

Uh oh!

arsenalzp commented Jan 23, 2026

Uh oh!

mheon commented Jan 25, 2026

Uh oh!

Uh oh!

arsenalzp commented Jan 26, 2026

Uh oh!

TheRealBecks commented Feb 9, 2026

Uh oh!

Honny1 commented Mar 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Honny1 commented Apr 29, 2026

Uh oh!

arsenalzp commented Apr 30, 2026

Uh oh!

arsenalzp commented Aug 11, 2026

Uh oh!

Honny1 commented Aug 11, 2026

Uh oh!

arsenalzp commented Aug 11, 2026

Uh oh!

Honny1 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

packit-as-a-service Bot commented Aug 22, 2026

Uh oh!

Honny1 commented Aug 25, 2026

Uh oh!

arsenalzp commented Aug 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

arsenalzp commented Aug 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Honny1 commented Mar 11, 2026 •

edited

Loading

arsenalzp commented Aug 25, 2026 •

edited

Loading

arsenalzp commented Aug 25, 2026 •

edited

Loading