Issue with symlinks

[BeebBenjamin]

Something can happen where on linux the symlinks become invalid. As a result trying to retrieve previous viewers breaks and results in the server throwing: ERR_EMPTY_RESPONSE

[BeebBenjamin]

Something is causing an infinite loop on the iiif/manifest, iiif/image and mirador directories...

[BeebBenjamin]

I think the way symlinks are created is wrong...

[BeebBenjamin]

If you log into the container and run find -L image in app/data/iiif/image you get find: File system loop detected; ‘image/image’ is part of the same file system loop as ‘image’.

[BeebBenjamin]

Here's another clue. Viewers from previous sessions do not appear in the symlinked folders, but viewers created in a new session do... we need to retain the viewers from previous sessions...

[jcanofuentes]

Thanks, I'll look into it. I'll start by trying to replicate the behaviour and I will provide an update on my progress by the end of the week.

[BeebBenjamin]

OK, I did a test. Symlinks work, but you still get the @image or @manifest or @mirador appearing in the file listing (could probably filter this out) because clicking on these leads to an infinite loop.

Stopping the container keeps all the links working.

However, if I use down (destructive) command the machine is destroyed, but some of the old files stay in the data directory and get listed in a new machine that is provisioned.

If you try to navigate to these previously generated resources ERR_EMPTY_RESPONSE is thrown. My guess is some of the files created during the generation process are not present in the new machine, so the old generated viewers, images, and manifests are broken when you attempt to view them in a new machine.

I assume that generated files get put into app/iiif which is a docker container folder that isn't kept when the machine is wiped.

I think we should put generated files on persistant storage so that one can keep previously generated content but provision a new machine.

[jcanofuentes]

You were right. The symlinks pointed the wrong way (/app/data/iiif/manifest → /app/iiif/manifest), causing loops and leaving content in paths that get deleted on down.

Fixed in branch fix-symlinks-use-copy by flipping the symlinks:

• /app/iiif → /app/data/iiif
• /app/image → /app/data/image

Now everything generates directly in the ./data volume. After docker compose down and up, old viewers still work

Tested:

1. Created viewer
2. docker compose down
3. docker compose up
4. Old viewer still accessible

[BeebBenjamin]

Cool. I can test your fix. I also have one, I've decided to combine the python terminal data generation location with the docker machine. I've made some changes to server.py and other things to rationalise it!

[jcanofuentes]

Hey! Any update on testing the fix-symlinks-use-copy branch? Ready to create the PR when you give the ok

[BeebBenjamin]

I am rationalising the whole storage. When I am ready I will put in a PR for it, it superceeds some of the work in your fix branch. But I will need help on my PR making sure docker doesn't create resources that need root access. We need to find a way to get docker to use the current user to create the resources instead of root. So that we do not end up with resources with mixed access privileges. If that makes any sense to you?