Current stow implementation in udi8/9 will break under certain conditions

[isuftin]

Describe the bug

In the UDI8/9 dev images, there is a stow command that runs: https://github.com/devfile/developer-images/blob/main/universal/ubi9/entrypoint.sh#L8-L24

I am running into an issue using OpenShift DevSpaces 3.17.0 when using the UDI9 image linked above (and our own derivative builds). The issue happens when we enable fuse-overlayfs, persistUserHome and disable the init container.

In this senario, DevSpaces will start the workspace and write to /home/user/.config/containers/storage.conf as part of the overlayfs configuration. It will also attempt to run /entrypoint.sh from the base developer image. When that runs, it calls stow . -t /home/user/ -d /home/tooling/ --no-folding -v 2 because at this point stow has not been run. The stow command fails with the following

* cannot stow ../tooling/.config/containers/storage.conf over existing target .config/containers/storage.conf since neither a link nor a directory and --adopt not specified
All operations aborted.

The workspace then goes into crashloop.

This works fine if we use ephemeral user home because the if block that stow finds itself in will guard against stow running in that case.

This also works fine if we keep the init container enabled because the init-container runs before the workspace container starts. At that point /home/user/.config/containers/storage.conf does not yet exist, so the init container finishes, writes /home/user/.stow_completed and by the time the workspace image's /entrypoint.sh runs, it will skip running the stow command because /home/user/.stow_completed exists.

Che version

7.95@latest

Steps to reproduce

1. Define the CheCluster to use OverlayFS
2. Enable persistent home
3. Disable the init-container
4. Start an empty workspace using the workspace samples

Expected behavior

Stow should run and not fail when it is unable to perform a stow on any one file.

Runtime

OpenShift

Screenshots

No response

Installation method

OperatorHub

Environment

Linux

Eclipse Che Logs

No response

Additional context

While I've discovered this bug in this specific case, this bug will happen any time that a non-dir, non-link file exists in /home/user/ but also exists in /home/tooling/ and stow runs such as in the case of a persistent user volume.

[isuftin]

A possible fix might be for the init portion of entrypoint to find all of the non-dir, non-linked files in /home/user that also appear in /home/tooling and add them to the .stow-ignore list before running stow.

Bad, untested code follows:

find /home/user -type f -xtype f -print  | sort | sed 's|/home/user||g' > user.txt
find /home/tooling -print | sort | sed 's|/home/tooling||g' > tooling.txt
IGNORE_FILES="$(comm -12 user.txt tooling.txt)"
rm user.txt
rm tooling.txt
for f in $IGNORE_FILES; do echo "${f}" >> /home/tooling/.stow-ignore;done
stow . -t /home/user/ -d /home/tooling/ --no-folding -v 2

[ibuziuk]

cc: @dkwon17 @AObuchow FYI please, review
@isuftin thank you for reporting, in case you would like to provide a PR - contributions from the Community are most welcome ;-)

[isuftin]

@ibuziuk - The fix probably needs to happen in https://github.com/devfile/developer-images/ since that's where the images are created. But the fix would also have to make it up to RedHat's UDI8. I wonder if I should close this issue here and open it @ https://github.com/devfile/developer-images/

[ibuziuk]

@isuftin thank you for the contribution. Having the issue here should be fine 👍