Performance Issue with Container Startup Due to docker-uid-gid-setup.sh on Mechanical Hard Drives

[H-Dynamite]

Description:
For a long time, I have been experiencing performance issues with the docker-uid-gid-setup.sh script. Since my Malcolm instance runs on a mechanical hard drive, restarting the system causes all containers to take approximately 40 minutes to start up. The containers with the slowest startup times include zeek-live, logstash, dashboards, and netbox.

After investigating the integrated tasks inside the containers, I found that most of the time is spent on permission assignment tasks, such as:

usermod --non-unique --uid ${PUID:-${DEFAULT_UID}} ${PUSER}

I am not very familiar with Docker, but I noticed that each container has a root account and its own dedicated user (e.g., zeeker, suricata, logstash, opensearch-dashboards, etc.). Malcolm also runs under a non-root user.

I am considering whether it is possible to sacrifice some security to improve startup speed. For example, could I change the users of key containers (zeek-live, logstash, dashboards, etc.) to root? However, I am unsure of the potential implications. Is it feasible to run Malcolm this way?

Any insights or suggestions would be greatly appreciated!

[mmguero]

This is a known issue for how containerization and user/group namespaces work, and it's really exacerbated by HDDs.

The "optionally run as root" idea is interesting, but there are a few containers that will flat-out refuse to run if it thinks the user is root. But there may be some other options that could help you:

The main one is, what user ID is Malcolm running under on your host? In other words, at the command line logged in at the account under which you start Malcolm, if you run id -u and id -g what is returned? My guess is it's probably not the default one that the Malcolm containers were built with, which is 1000 and 1000, respectively. This is "fine" in that it works, that's what the usermod and chown and chmod operations you see are, but in a system like yours where you've got mechanical drives it goes from taking maybe two or three minutes (I'm running UID 1001 on my dev machine here, so mine goes through the same operations) to 40 minutes.

If you are running Malcolm under a UID/GID other than 1000/1000, you may be able to avoid the costly operations by rebuilding all of the container images to use your expected UID/GID as the default instead. This could be done a couple of ways:

• having GitHub build the images for you
  • fork the Malcolm repo on GitHub
  • in your fork, change ALL of the instances of UID and GID 1000 in the files under ./Dockerfiles in the source tree (search and replace in your text editor, sed, whatever)
  • commit your changes on your fork
  • use GitHub to build the images for you
  • use the convenience script to pull and tag your images

OR

• build the images yourself, locally
  • check out the Malcolm code (either the official fork, or your own, doesn't matter)
  • make the changes to the Dockerfiles as described above
  • ./scripts/build.sh (answer Y to force full rebuild)
  • it will build the images and tag them appropriately

The build might take a long time (probably... an hour-ish? maybe longer if the HDDs are a limiting factor for the build as well) but then after that once you're running with these custom images the startup should be faster because it will look at your UID/GID and see they already match (I think this is the really slow one, and if they match it gets skipped) and should start up faster.

The other "solution" for this is for my team to go through the containers and be a little bit more surgical in determining exactly which files need their ownership changed and which ones are ok to leave owned by root during build. But I think that your best bet is probably building custom containers as I've described with your custom UID/GID values baked-in.

If I've misunderstood something about the premise here (like, you are actually running as 1000/1000?) then let me know and we can talk through it.

[H-Dynamite]

I saw the Dockerfile with the following content:

ARG DEFAULT_UID=1000
ARG DEFAULT_GID=1000
ENV DEFAULT_UID $DEFAULT_UID
ENV DEFAULT_GID $DEFAULT_GID
ENV PUSER "suricata"
ENV PGROUP "suricata"

Here, DEFAULT_UID is indeed set to 1000. I had assumed that PUID=1001 from the config/process.env file would ultimately take effect. Currently, UID 1000 on my host machine is not the same as the one running the Malcolm instance—the Malcolm instance is actually running under UID 1001.

I will now modify PUID=1001 so that Malcolm runs under the user with DEFAULT_UID=1000 and test the performance.

[mmguero]

The stuff from config/process.env does ultimately take effect: that's exactly what it's doing during start up, is changing the ownership of stuff at build time (1000) to what you've set it to at runtime (1001). But it's not a free operation.

If you rebuild the images after changing those values, build time and runtime match, so you don't have to pay for it with the time of the change.

[mmguero]

I've logged #867 to look for potential optimizations in this process, as well.

[H-Dynamite]

I've logged #867 to look for potential optimizations in this process, as well.

Thank you for taking this issue seriously. I realized I installed Malcolm using this guide. During the installation, the user with UID=1000 already existed on my system. Since I have multiple users, I opted to use a different one and did not utilize the user with UID=1000. At the time, I didn't fully grasp the meaning of the prompt: "Malcolm processes will run as UID 1000 and GID 1000. Is this OK? (Y / n)". Now I understand.

Perhaps we could add some notes to this documentation to prevent new users from making similar mistakes. Alternatively, we could consider specifying DEFAULT_GID as 2000 and have users create this user themselves beforehand, rather than dynamically modifying permissions at runtime—which indeed comes at a high cost. The system performance may be acceptable initially after installation, but after prolonged operation, accumulating logs and files could significantly slow down the permission assignment process.

[mmguero]

The system performance may be acceptable initially after installation, but after prolonged operation, accumulating logs and files could significantly slow down the permission assignment process.

It's a fixed up-front cost that depends only on the files baked-in to the container images at build time. Additional files created during runtime are created with the right UID going forward and do not pay this penalty.

I'll take your suggestion for improving the documentation under consideration, thanks.