System testing on FreeBSD

[dfr]

For a while now, I have been working on enabling podman system tests on FreeBSD as a background project. At this point, I have large parts of the test suite working with good coverage for tests of basic functionality. This has helped find bugs and feature gaps both in the podman port to FreeBSD and in my supporting OCI runtime ocijail.

I would like to start merging some of this into podman but I'm not sure what the best approach is for the the test image used for most of the tests. I could use the existing linux image - FreeBSD's built in Linux emulation works well here but I would actually prefer a FreeBSD image. My current work-in-progress uses a FreeBSD port of busybox (quay.io/dougrabson/busybox:1.34.1) which is small and works well with the test suite which assumes an alpine image which itself is based on busybox.

Including FreeBSD images in the testimage build would work but would also make that process more complex. Would it be reasonable to have a separate image for FreeBSD? Perhaps quay.io/libpod/testimage-freebsd (or quay.io/dougrabson/testimage-freebsd since I have no access to libpod)? This would add a little logic to test/system/helpers.bash to set a different default image on FreeBSD.

[Luap99]

The testimage is just a manifest list, technically the only thing required is for us to add your bsd image to that manifest list then a podman pull on freebsd would automatically pull the correct image without any changes to the test code.

The bigger problem would be to rebuild the image when we want update that, right now we cross compile for other linux architectures not sure how this workflow can work with bsd.
cc @edsantiago

[dfr]

I haven't tried it but in theory at least, it would be possible to build all the linux and freebsd images for the manifest on a FreeBSD host using Linux emulation but that might be a step too far. I'm looking forward to @edsantiago's thoughts on this.

[edsantiago]

Thanks for all your ongoing work, @dfr!

I'm not sure what the goal is here, or even what the question is, so I'd like a quick moment to establish common understanding:

• Where/when will these tests run? Is it your intention to make them part of the c/podman CI setup? Is this something you will run on your end, either manually or automatically? Something else?
  • Corollary: what kind of maintenance will be required to track (infrequent) testimage changes? Who do you expect to maintain the FreeBSD testimage?
• May we assume you've looked at https://github.com/containers/podman/blob/main/test/system/build-testimage , and that is the source of your "make that process more complex" comment?
  • Corollary: I think we can all agree that "building the official testimage on FreeBSD" is a nonstarter

For one specific set of answers to the above questions, the solution to your problem might be as simple as:

$ env PODMAN_TEST_IMAGE_USER=dougrabson hack/bats ....

Your responses will help fine-tune the discussion.

[dfr]

A medium term goal (for me at least) is to make them part of the podman CI, probably non-blocking at least to start with. Before that, I have some changes to the test suite (mostly skipping things like cgroups and systemd) that I would like to get reviewed and merged to main - this would allow manual testing without juggling branches locally.

For the testimage, I'm happy to maintain it whenever it needs changing. In general, I'm always happy to work on fixing freebsd-related build problems and want to keep any friction with the linux build and CI as low as possible.

I have looked at build-testimage and in my prototype, I have a version which conditionally builds either FreeBSD or Linux images. Thats kind of messy so perhaps adding a build-freebsd-testimage would be better. For running tests, my helpers.bash uses something like this:

if is_freebsd; then
    PODMAN_TEST_IMAGE_USER=${PODMAN_TEST_IMAGE_USER:-"dougrabson"}
    PODMAN_TEST_IMAGE_TAG=${PODMAN_TEST_IMAGE_TAG:-"20230706"}
else
    PODMAN_TEST_IMAGE_USER=${PODMAN_TEST_IMAGE_USER:-"libpod"}
    PODMAN_TEST_IMAGE_TAG=${PODMAN_TEST_IMAGE_TAG:-"20221018"}
fi

We could decouple freebsd builds and just add them to the main manifest in build-testimage, using something like:

$PODMAN manifest add $testimg $($PODMAN pull --platform freebsd/amd64 quay.io/libpod/testimage-freebsd:latest)

[edsantiago]

Hi again. Just checking in. I took some time today to pull and inspect quay.io/dougrabson/testimage:20230725, and to see if qemu offers any hope for emulating freebsd on linux. As best I can tell, there isn't.

I have a moderate (8 out of 10) aversion to adding if is_freebsd ... use-a-different-image-from-another-repo. And I have a strong preference (9 out of 10) for keeping our image build process in-house. That makes it really hard to come to a compromise. Have you had any further insights on this since last month?

[dfr]

I tried to tidy up my changes to build-testimage to adapt it to building for both FreeBSD and Linux - you can see the current iteration here.

I'm not sure what the best long term solution is. One possibility is to separate building FreeBSD images, building that whenever necessary and pushing to some agreed place using a 'latest' tag. This can be 'appended' to the Linux manifest using podman manifest add --all quay.io/somewhere/freebsd-testimage:latest as a final step in build-testimage. This would have no impact on running Linux system tests since the test host will never pull the FreeBSD image from the manifest.