Gallery Watcher - Feature building

[alextran1502]

This discussion thread discusses implementing the most-wanted and missing feature of Immich #7 .

Gallery watcher

Below are some of my initial thoughts.

There will be an additional WATCH_LOCATION in the .env file, pointing it to the gallery folder.
The user can opt-in to use this feature or leave it commented out not to trigger the watch functionality of the server.

User story

What

• The admin user has an existing gallery
• The admin user is asked to manage other tenant galleries by having access to the tenant's gallery or by a shared drive

How

• Setup an Immich instance
• Create an admin user and sign in
• [Optional] Create additional user
• Bring down the containers
• Put the gallery in the respective user's email as the folder name
• Point the gallery folder to WATCH_LOCATION
• Bring up the containers

MVP

1. Initial scan

• Check if the folder structure conforms to the rules for multi-user support
• Check if the user exists in the database
• Start scanning and adding file info similar to an asset uploaded.

2. Subsequent start-up and scan

• Find the new and existing files since the last scan - based on a unique file path.
• Maybe needed a flag as isWatchedFolderAsset for ease of querying in the database to distinguished the uploaded asset vs the gallery's asset.
• Maybe a performance issue if the folder has 500_000 photos, so doing this in a background task would be preferred.

3. Handling of symlinks

• unsure
• Does this matter?

4. Actions when new files are added

• Similar to a new asset is uploaded

5. Actions when existing files are removed

• Remove the file from the database

6. Actions when existing files are moved

• Must know the asset's previous path
• Check if the new path is different from the previous path
• Find the ID with the previous path
• Update the path location with the new path

7. Multi-user support based on their email address in the watched folder

├── top-level-folder
│   ├── [email protected]
│   │   ├── Album_1
│   │   ├── Album_2
│   ├── [email protected]
│   │   ├── Album_A
│   │   ├── Album_B

Technology

• Use Docker's mount-volume to mount the desired folder and map it directly to the server and microservice container
• Use Chokidar to watch for file event
• Maybe there is a better programming language/technology to scan and keep track of file changes on disk. If so, maybe another container is needed to implement that system. It will then communicate with the server/microservice container to perform database level mechanism

[spupuz]

Symlink in my opinion should not be used,
move should be managed as a new asset uploaded and the previous item (or path)should be deleted from the db and reinserted with new path as new pic maybe.

[fyfrey]

I think symlinks are really important. With symlinks, users can create the folder structure that "conforms to the rules for multi-user support" without actually moving their files/folders around

[fyfrey]

the move action is just fine: Update the path location with the new path. Done. No need to delete and reinsert an asset. Thumbnails can also be kept.

[JM-Lemmi]

What would be the downside of accepting symlinks?

[bo0tzz]

Symlinks do not play well with docker mounts, see this stackoverflow question for some more info. I don't think we need to explicitly -reject- symlinks, but IMO it will be up to the user to use them in a way that does not break.

[ravensorb]

Wouldn't this also be possible via docker volume mount?

[fyfrey]

We should differentiate between UPLOAD_LOCATION, WATCH_LOCATION and another folder e.g. THUMB_LOCATION
The WATCH_LOCATION should usually be mounted read-only in docker to guard against immich accidentally deleting stuff.
The thumbnails and encoded videos from WATCH_LOCATION could be stored under UPLOAD_LOCATION but it's cleaner to save them in a clearly separate location like THUMB_LOCATION.

[bo0tzz]

There is another open issue for a separate thumbs storage folder (#418). I agree having it separate from the originals folder would be a good thing. Would we want to tie it to this change though?

[fyfrey]

I guess we better set up an independent thumbs storage folder before tackling this large feature

[oddlama]

I also think a separate thumbs folder might be necessary. Thumbs are also something that are very specific to immich, so they could always be stored in a separate folder. No need to associate them to a specific user or storage location. I believe a mapping fom internal image uuid -> thumbnail storage would suffice for all future purposes.

[olehkrupko]

guard against immich accidentally deleting stuff

On the other hand, if I want to delete an image, it would be nice if it was possible from both Immich and watch folder side

I'm horrified how problematic this set of features can become :D

[ravensorb]

Not sure if it should be forced to be read-only. Personally, I'd like to be able to remove any pictures that I decided I don't need (duplicates, out of focus, etc). Maybe there is a option in the User interface to allow delete operations?

[godismyjudge95]

In regards to this:

5. Actions when existing files are removed
   Remove the file from the database

Does that mean that the files must remain in the watched folder structure? Based on my understanding, I thought Immich would make a full copy of all the images and effectively import the images at original quality - negating the need for the original files to stay in the watched folder.

I understand why people might want to keep the files in the watched folder, but for me personally, when Immich is ready I am moving all my images underneath it - I won't want to bother with traditional file structures anymore.

It would be awesome if there was an option to instead use the watched folder as a sort of queue for importing into Immich and once an image is imported it would be deleted from the watched folder (indicating it has been imported). Or if that is too scary, at least allow one to delete the images manually from the watched folder to prevent too much space being used up.

Otherwise, this sounds like an awesome feature. Looking forward to it!

[oddlama]

I have a similar feeling about this. I tried to provide a summary about that in a separate answer, maybe that aligns with your views.

[Hukuma1]

I think Immich "making a full copy" is precisely what some people wish to avoid, hence the watch folder(s). The idea isn't to completely migrate to some new software, it's to integrate existing images to a new software. It's basically a question of: do you want Immich to conform to your setup, or require you to conform to Immich's setup?

[JM-Lemmi]

I thought Immich would make a full copy of all the images and effectively import the images at original quality - negating the need for the original files to stay in the watched folder.

This is already possible with the current Immich from my understanding? You can use the cli tool to upload your existing images to Immich and then just delete them.

From my understanding this feature is for using existing Data and basically having Immich as a frontend without changing the original Data. I am one of the people that would use this, because in the end, the display options come and go, but my Image Folders have stayed the same for the last 15 years.

Having to make a copy basically doubles the storage needs for my gallery.

[briandipalma]

Yes, you can upload and delete folders of images already using the CLI. I do this as in practice the Immich structure is the same as what I'd have anyway apart from one extra user ID folder.

[oddlama]

Here are some of my thoughts about this, in no particular order:

First of all, I noticed that the main thing users expect from #7 is a way to drop files into a folder and have them automatically uploaded to immich as their user. I see that there are two main ways to approach this.

1. Immich acts purely as a "view" into a user provided directory. The user can manipulate the directory structure to their liking and immich will always reflect that.
2. Immich just consumes media from a given directory, but images are stored only in immich.

(1) A filesystem view

I would personally advise against permanent asset storage in the user-directory as described in (1). I can see that this will become a very complex part of immich and may create a whole myriad of problems in the future. Here are some thoughs about that:

• This will cause immich to basically have several distributed databases that you need to keep in sync.
• Changes to the filesystem (move, delete, add, folders for albums) will have to be processed immediately to keep an up-to-date view.
  • What if a user renames a whole album? What happens if a copy of an image is added under a different name? What if the user undo's a filesystem operation? ...
• Changes to the filesystem that happen while immich is stopped also need to be processed immediately on startup. This requires permanent re-evaluation of the whole structure, as anything could change while immich is stopped: What if a file is renamed while immich is not running? Is this a delete and re-add? Should the move be detected? ...
• You'd basically create a second API for immich that operates on filesystem-level instead of using HTTP, but a lot of operations would be the same.
• How to handle thumbnail storage?
• What if a user deletes a photo by accident? The current approach would immediately delete it from immich too, and SMB shares usually don't have a "Trash" from where the photo can be restored.

(2) A consume-only directory

All of these problems can be avoided by instead opting for a simpler approach that still solves #7: A consume directory (2). The user can dump all files that should be added to immich into a specific directory, and they will be processed by immich automatically.

• Consume operations are atomic due to the filesystem being inherently atomic. This means that immich only needs to support a single operation: Adding images. When a image is added successfully, immich deletes the corresponding file from the directory. There is no possibility for files being missed, and there can't be any issues due to renaming/deleting files while processing (as the file descriptor wouldn't change as soon as immich opens them).
• Filesystem watching would become trivial: Any change in the consume directory just has to notify immich to work on the directory queue until all images are gone (deleted by immich due to completion).
• Parallelization (for the future maybe) would be a non-issue, at least on the filesystem-watcher's side. File descriptors are unique and no file operations could cause asset confusion.
• Direct adding to albums can be supported by exposing existing albums as subdirectories. Dropping files into the main directory adds them to your account, adding them to a specific album directory will add them to your account and also to the album. Immich can automatically create the necessary album folders. (Possibly creating a new folder would allow creating a new album).
• Duplicates can easily be detected by calculating and comparing it's checksum (or any other means e.g. fuzzy duplicates), and moving the file to a duplicates folder (atomic).
• The most significant disadvantage is that users will not be able to access their library outside of immich (but there's a solution to that, see below).
• Accidental deletion is also not possible and must instead be done via the web-interface.

A popular self-hosted document management system called paperless-ngx pretty much does the same thing already, and they have huge success in doing so. They also provide a solution to the problem that users cannot access their library when only using a consume directory, that is called "storage paths" in paperless.

Their idea is to allow configuring a separate storage path for each user (and maybe each album), that defines where immich will place the stored assets. But this directory is read-only to the user, and modifying the content is not allowed. This guarantees that immich doesn't need to implement the complex scanning and database resynchronization methods, while still allowing users to view their assets via e.g. SMB. This would require only minimal changes to what immich does today, because the database can stay the same. Just the filesystem path for uploaded files would have to be set dynamically.

NOTE: To allow user's to have good browsing experience outside of immich, I would advise to not store images with their uuid as the name, but instead with a generated name that is naturally sortable like{asset_date}_{short_id}_{original_name} => e.g. YYYYMMDD-HHMMss_9a45cf1d_IMG-5542.jpg. Also remember setting the modified time of the file.

Multi-user support

Instead of forcing a specific file-structure, I would apply the same "storage paths" principle introduced before. There is a global storage
directory for immich where images are stored by default, like it exists now. But the admin can configure specific storage paths for users and maybe for albums. These paths can be stored in a separate table.

[email protected]:

• consume = /media/smb/share_for_user1/images/consume
• archive = /media/smb/share_for_user1/images/archive

[email protected]:

• consume = /media/smb/share_for_user2/images/consume
• archive = /media/smb/share_for_user2/images/archive

By using the "consume" approach, the images folder would then look like the following:

/media/smb/share_for_user1/images
├ consume/
│ ├ 2010 Rome Vacation_1afc451a             # <-- {album_1_name}_{album_1_short_id}
│ ├ 2013 New Zealand_5b29fa72               # <-- {album_2_name}_{album_2_short_id}
│ │ └ IMG_14.jpg (copied here by user, not yet consumed)
│ ├ ...
│ ├ IMG_1234.jpg (copied here by user, not yet consumed)
│ └ IMG_1235.jpg (copied here by user, not yet consumed)
├ archive/
│ ├ 2010 Rome Vacation_1afc451a             # <-- {album_1_name}_{album_1_short_id}
│ │ └ ... (lightweight symlinks to stored images)
│ ├ 2013 New Zealand_5b29fa72               # <-- {album_2_name}_{album_2_short_id}
│ │ └ ... (lightweight symlinks to stored images)
│ └ ... (all stored images)
├ .trash/                                                   # <-- Immich could support moving files to a trash folder when they are deleted in the web-interface (this folder is a regular unwatched folder and the user can then delete them permanently if wanted, or after X days automatically)
└ .duplicates/                                           # <-- Directory where to place duplicates (and maybe allow a setting to just delete duplicates)

3. Handling of symlinks

I would not handle symlinks encountered in consume directories in any special way. They should never occur. If you instead opt for the view-approach, symlink handling must be supported and could become more complex (detect original? what if original isn't in immich's view folder?).

When providing configure asset storage paths, symlinks could be managed by immich to provide up-to-date views into albums even on file-system level.

---

These are just my opinions and initial thoughts about the matter. I hope they will be useful in your decision process, but as always: Just do your thing. I'm always happy to provide feedback.

[bo0tzz]

This is a good description of the situation. I think two issues are being conflated in @alextran1502's original post: #7 ("Support watching the filesystem"), and #34 ("Support user defined storage location/structure"). Those map to points 2 and 1 in your post respectively. I think it's important that we very clearly define the separation between those two, and describe exactly what we do and don't want to implement.

I think storing assets to a custom path structure (which would sort-of fit onto #34) might still be viable as a separate feature, but without allowing the user to make changes to that file tree - otherwise we would run into the issues that you describe.

Edit: #402 is another issue in this space. There's a few that have been intermixed and confused for eachother to some degree.

[oddlama]

Absolutely, thanks for the clarification and the links. By planning a little ahead, I am sure we can find an optimal path to upgrade immich step-by-step to avoiding having to do a bunch of unnecessarily complex changes at once and to maximize feature reuse.

I'm not sure if this is actually the way Alex chooses to pursue, but here would be my attempt at summarizing a possible upgrade-path utilizing smaller incremental changes to immich:

1. Implement basic consume behavior

   • Goal: One configurable folder per user. No album related features, just a simple consume-and-delete folder.
   • DB: Requires a new table for mapping user_id -> consume_folder_path
   • Duplicates would be ignored for now (i.e. just delete them, or just don't process the files)

2. Separate thumbnail storage

   • Goal: Completely separate thumbnails from assets, as thumbnails are basically an implementation detail of immich.
   • DB: Requires a new global database setting for the thumbnail storage path. Otherwise I believe something similar to asset_id -> thumbnail_path should already exist, this would have to be removed, as thumbnails will have then have a predicatable name and path.
   • I do not think this path should be allowed for templating like path/{date}/{asset_id}.webp, as no other application would really need access to the thumbnails. This is something I would deem immich-specific. So just a separate configurable path would be all that is needed. This also frees immich from having to check this path for any external changes.

3. Refactor internal asset storage

   • Goal: Internal restructuring to support dynamic asset paths.
   • DB: Requires a new table for mapping user_id -> asset_storage_path, plus a global default storage path that is used when this mapping is not present for a user.
   • All internal functions should use a wrapper (asset manager) to retrieve/access assets paths, so there's a single module that has to deal with asset paths.
   • Allows templating this with some predefined tokens. (Needs more thought)
     • {original_filename} (full original name)
     • {basename} (=filename without extension)
     • {extension} (=normalized lowercase extension)
     • {incremental_id}, {full_asset_id}
     • {year}, {month}, {day}, {hour}, {minute}, {second}
     • {date} (full naturally sortable date e.g. YYYYMMDD-HHMMss)
     • {iso_date} (UTC ISO8601 date YYYY-MM-DDTHH:MM:ssZ, could cause issues with windows due to use of :, but some people prefer this format as it is a standard, uses UTC and is easily parsed by other software)
   • On migration, this path is set to the currently used location, so it stays compatible with existing installations: /path/to/current/location/{full_asset_id}.{extension}
   • This would then allow versatile storage schemes: /smb/my_images/{year}/{month}/{date}-{basename}.{extension}
   • We should require the use of either {full_asset_id} or {incremental_id}, to avoid having to deal with potential file name clashes. A global incremental_id would just exist for convenience, to avoid long paths with a uuid4 in the filename. There would be no guarantees that this incremental id actually refers to an asset, but would just be a more aesthetically pleasing disambiguator.
   • Make sure that files retain their correct modification date when moved from a consume folder (maybe actually move the file)

4. Expose asset storage paths in web interface

   • Goal: Provide a UI to allow the admin user to customize the asset storage location and format. This is a privileged operation!
   • I have no idea how to design a proper interface for this, but I imagine a simple configuration site where the admin user can change the storage paths for a selected user, or revert them to the default path. The default path is only configurable via environment variables.
   • Unsolved issue: How would you allow moving old installations to a new path? The metadata is available at this point, but there needs to be a tool to move that over to a new location (maybe a simple offline script)

5. Album importing

   • Goal: Allow users to add a whole lot of files that are automatically added to a new/existing album.
   • When immich is started or creates a new album, a corresponding folder is created in the consume directory for every user with access to that album
   • Placing new files in such a folder, imports the assets and adds them to the album.
   • A new folder with at least one image that is added on the same directory level can automatically create a new album (with the folder name). Empty folder will not create new albums straight ahead to allow users to rename the folder before adding assets.
   • A new album creation from a new folder will immediately create a second folder for that album. This is OK. Until the initial folder is empty, it will act similarly and import assets to the new album. The user-created folder is then deleted 10 seconds after it has been emptied if nothing else is added in the meantime.

6. (Optional) Album linking

   • Goal: Assets that are stored in a user-defined storage path should be viewable using SMB alone.
   • DB: It should be configurable whether this feature is enabled, on a per-user basis. Probably by adding a new storage path user_id -> album_link_collection_path.
   • This new path is not templated.
   • The path must be reachable relative to the user-defined storage_path. More precisely, there needs to be a common prefix between the configured user_storage_path and this album_link_collection_path, and they need to be on a common mount point.
     • OK: /smb/user1/images/{year}/{month}/{date}-{incremental_id}-{original_name} and /smb/user1/images/albums, when /smb/user1/ is a mount point
     • invalid: a link from the album path would cross a mountpoint to get to the storage path.
     • For each album, immich can now populate a new directory {album_link_collection_path}/{album_name}, where any associated image is stored using {date}-{original_filename}. This could maybe be made configurable, but I would figure that albums should specifically be flat directories. Not sure on that one.

7. (Optional) Trash directory

   • Goal: Allow users to specify what to do with deleted assets.
   • DB: Introduce another storage path mapping user_id -> trash_path.
   • If unset, files are deleted permanently.
   • If set, files are instead moved to the given directory. Should support the same templating as regular storage paths.
   • A good default to suggest might be /path/to/images/.trash/{date}-{basename}-{full_asset_id}.{extension}

8. (Optional) Versatile duplicate handling

   • Goal: Allow users to specify what to do with duplicated assets that are consumed.
   • DB: Introduce another storage path mapping user_id -> duplicate_path.
   • Moving duplicates to a separate folder might be a desired thing in the future, when more sophisticated duplicate detection algorithms
     are used. Then it would be undesirable to just delete duplicates, because it might be a false-positive duplicate. (Maybe we also need a force-import directory at that point).

The order of these changes could definitely be changed, but someone with more knowledge of the codebase than I would be better at assessing that. Let me know what you think about this.

[fyfrey]

@bo0tzz agreed, there are quite a number of issues/aspects conflated and we should clearly define what problems we want to actually solve

@oddlama thanks for the incredible detailed write-up! I agree, (1) a filesystem view is certainly more complex than (2) a consume-only directory even if combined with path templates. I really like the idea of path templates - for configuring the storage of assets backed up via immich app/web. However, for importing an existing library that is going to be used concurrently with other software, path templates are not enough. It needs the (1) filesystem view for compatibility. I'll do some more elaborate description in a separate comment.

[ScuttleSE]

In my opinion, anything that can't handle a read-only volume with an arbitrary folder structure is out. There should never be any need for the application to write to the folders of the original images. Also, forcing a user to adhere to a specific folder-structure is an absolute no-go.

[oddlama]

I'd see it a little more nuanced than that but I agree with the general sentiment. A defined folder structure may be necessary at some level to provide certain features (e.g. map import folders to specific albums, without requiring privileged configuration each time an album is added). But I agree that the default should not be invasive and to keep required privileges at the absolute minimum.

[davidrudlstorfer]

Two very important points if this is implemented.

[MrSuicideParrot]

I was one of the guys that was waiting for having the ability to import existing photos. However, I was hoping to be able to import my existing photos without a second copy and maintaining my current structure of files (or making same adaptations, but without compromising my ability to stop using immich and going back to samba ahahah). The reason is simple, no one wants to be locked to a software.

So my opinion will be directed to my use case of course.

(1) A filesystem view
I don't believe that monitoring the filesystem is such a burden. Linux kernel has the inotify system that enables you to monitor files in a disk. In conjunction with regular scans (one a day, can be configurable by the user or on demand), it should be enough to have the files available in immich. When the server restarts, it's simple, the immich needs to make a filesystem scan. An example of this kind of behavior is the plex server. Regardless, even if there is delays or momentary incongruities, I think it's a compromise that the user that wants to use this features should accept.

(2) Consume-only directory with files deleted after import
I think this can be easily done with another microservice, with the existent api of immich and inotify to monitor the folder.

Therefore, I think these are two distinct features. The only thing they have in common is the need for monitoring the filesystem.
Regardless, the suggestion of @oddlama may provide a middle ground between the two options, if there is an archive and consume folder for individual albuns.

[bish0polis]

While I love the idea of a pristine source directory - which probably works best with docker - I'm looking forward to a non-container installation. So while I can't comment on the given workflow, this is a feature I'm also interested in.

I'd also accept a processing folder which does copy files in and potentially duplicates them in an out-folder, because I'm hoping for a facility that recreates the pic with a few embedded thumbs, updated data and a new filename (based on templatable form) to enforce a standard -- and it has to be geriatric-friendly for the users I'm thinking, so a copy-here workflow works best there. So I have big hopes, but the current workflow isn't included. ;-) Good luck!

[natedawg]

It would be amazing if immich allowed multiple read only photo libraries. I have multiple locations with different types of photos. I have my own photos, family photos, internet photos as well as professional photos. I tried making multiple accounts to manage this but that doubles the storage requirements and there is no easy way to switch accounts. I think the way Plex manages this would fit immich perfectly. It allows you to create multiple libraries of photos with different folder locations and you can choose to individually share libraries with other users.

Some thoughts...

For the frontend

• Each user would have a settings page where you create a library and choose one or multiple locations to scan/import.
• You would be able to create as many libraries as you want as well as add as many locations as you need for each library.
• Upon library creation you would be able to check or uncheck if you want the library to show up on the front page timeline. (I don't really need my internet memes and wallpapers mixed with my family photos but I would like to view/share them sometime.)
• The library page would allow for a manual scan/import to be started as well as an option to share the library with specific users.
• The immich side bar would list your personal libraries and libraries shared with you.

For the backend

• Thumbnails would be saved to a different location that can be specified in docker-compose.
• When scanning an additional flag should be added to differentiate between local read only assets and user/mobile uploaded assets to allow for filtering/searching.

[JM-Lemmi]

The additional feature of multiple seperated libraries you bring up here may be beyond what this feature is supposed to become, but I'd greatly appreciate it too. Especially having not all of them show up on the main page would be a great feature.

[ravensorb]

There is something to the comment around Plex. Could there be something to the idea of a watch for change + a periodic scan of the filesystem (or a subset)? I'd be ok if I sometimes needed to trigger a full scan. This could also allow for things like auto album creation via subfolders and then these albums would be within the library that owns a specific root folder. It would also allow for setting a read-only flag at a Library level (along with other options like scan rate, include in timeline, detection, etc).

As an added thought -- this could in the future allow for supporting the import of "connection" to other services that host images (OneDrive, GoogleDrive, etc).

[rhuneai]

I potentially would use this feature for my existing photos, and non-mobile photos (e.g. taken on DSLR). How is the "moved" detection meant to work (i.e. what uniquely identifies a particular photo)?

It sounds like the filename would be what is used, but for my camera it often reuses filenames/numbering (e.g. after formatting the memory card; or every 1000 photos it starts a new folder and the numbering starts again).

[alextran1502]

On Linux, there is the concept of inode, basically an unique id for the file

[spupuz]

I potentially would use this feature for my existing photos, and non-mobile photos (e.g. taken on DSLR). How is the "moved" detection meant to work (i.e. what uniquely identifies a particular photo)?

It sounds like the filename would be what is used, but for my camera it often reuses filenames/numbering (e.g. after formatting the memory card; or every 1000 photos it starts a new folder and the numbering starts again).

Would use hash probably

[latetedemelon]

I have my photos stored with a cloud service provider currently. Being able to consume read-only directories would enable me to migrate to immich with a higher "family acceptance" factor by allowing them to use it without messing with our old system. Further it would allow parties to keep their own photos seperate if they wanted to remove their photos in the future.

It's a large ask but I'd like to see the ability to "hot-swap" sources with the following use case:

1. Try out the product using read-only directories
2. Decide to use Immich going forward
3. Migrate / copy the existing read only directories to a new location.
4. Add new source and remove old while still respecting the current metadata

Ideally when scanning the new directory the system would recognize that an image has been seen before, has existing metadata (albums, tags etc.) and apply those even though the image is at a new location (new location could be either R/O or R/W)

[fyfrey]

I wrote this based on the great input and ideas in this discussion (especially thanks to @oddlama for the in-depth descriptions!)
Further, I've added another solution applied in other software I have used myself.

"Feature Wishlist" sorted from easy to more difficult:

1. Import existing library (duplicate file & double storage requirements are ok)

This is possible right now: Simply import the library with the CLI. Re-run once new assets are added to external library. Already imported assets are not added twice because immich does not important duplicates. Although this is certainly not an optimal from an efficiency point-of-view, a nightly cron-based batch import of the external library should be good enough.

Biggest downside is already mentioned in the headline: This approach creates duplicates and thus uses twice the storage unless the user's filesystem does deduplication. Another issue is that changes to files are not nicely reflected in immich: If the user changes an asset (edit an image, store as new file, remove old) and it is re-imported, immich now shows two images (old and edited).

related to #7

2. Consume-only directory

Automatically add new assets found in the specified dir to immich. This allows users to e.g. automatically import DSLR images stored on their server to immich.
This could be added to the CLI or the server. The implementation of watching a consume directory is straightforward and immich only needs to add a single operation that is triggered by changes: "import that new file as an asset like it was send from app/web". See the extensive post by @oddlama #1006 (comment)

Downsides if the files are copied into the consume dir (the user has a second copy in another folder): Similar to 1. Files are stored twice and edited versions produce near-duplicates.

Downsides if the files are moved into the consume dir (user has no second copy): User has no control what is done to the consume files. They might just be deleted due to a bug in immich and gone forever (unless the user has proper, functional backups in place). Even if the worst case does not happen, the user has no nice way to get the images out of immich should it die at some point :-(

related to #7

3. User defined storage

Let the user control how immich stores the assets backed up via app/web, so that there is no vendor lock-in and users can use other software to have a meaningful read-only access the assets stored by immich.
All assets imported into immich (from app/web/CLI or "2. Consume-only directory" implemented in the server) are stored in a folder structure that the admin can configure (e.g. via environment variables, better in the server settings on the web). To customize, path templates like $year/$month/$originalFilename-$immichUUID.$ext can be used. For additional info, again see the extensive post by @oddlama #1006 (comment)

An example of another selfhosted project that provides this feature is paperless-ngx (awesome document management for a single user): All documents added to paperless are stored in a configurable folder structure so that you can meaningfully browse the files that are managed by paperless. If the configured path changes, all files are moved/renamed by paperless. Caveat: It's a read-only access - you must not modify or move any document files!

Limitations: The user / other software is not allowed to modify files under immich's control!

related to #34 and #402

4. Take control of existing library

This basically combines 2. and 3, thus immich provides the single source of truth, i.e. has control over all assets.

All existing assets are placed in immich's consume dir and re-structured into the user defined folder structure. Depending on the manually created structure (e.g. year/month/filename), this could actually produce an identical folder layout. If folders were used as albums in the manually created library, producing the same result after the import becomes really tricky since immich supports an asset to be in multiple albums. For more info and possible implementation steps, see the post and comment by @oddlama #1006 (comment)

Another selfhosted project that provides this feature is paperless-ngx (see 3. for more info, advantages and limitations).

This approach solves some downsides of 2. If the user lets immich consume an existing library without keeping additional copies, the user can import an existing library without taking twice the storage space - yet, the user still retains read access to the all the assets in a meaningful way.
However, the limitations of 3. still apply.

Downsides: User must trust immich to not accidentally delete/destroy assets or have a reliable backup.
All assets are controlled by immich, the user or other software only gets a read-only view. This is actually a huge limitation. Imagine you have multiple programs choosing this approach to "work with existing libraries", it turns out you can not even use two of such programs together! Both require to be in control, have write access etc. Thus, you can always only have exactly one program in your selfhosted world that takes this approach. I would not like immich to be that one program :-) many other programs and selfhosted services dealing with photos (Lightroom, Plex, Jellyfin, ...) have found another solution enabling you to use all of them together: See 5.

5. Use existing library together with other services/programs

This is more or less the approach @alextran1502 has described in his initial post.

Immich scans an existing folder structure and adds all assets to its database. No duplicates are made, immich only needs read access to the files.
This approach allows Immich to be used together with many other services/programs that operate on the same library. You can still use the image editing software of your choice to make changes to your photos. You can still manually organize your assets in absolutely any structure you like (symlinks could be used to build the structure required for immich to understand which assets belong to which user or album etc.). Immich cannot accidentally delete assets from the existing library as it only needs reed access.

In theory, it does not even need watching for changes: Similar to 1. run a nightly cron job to re-index the external library. Delete all assets marked as external that are no longer found, add all that are not yet contained in the DB. Of course, filewatching as an "online scan" is a very valuable addition. But it could be done as a separate feature after the "offline scan" is implemented.

Granted, this approach is more difficult than 4., but it is possible to get it right: See e.g. Lightroom, Jellyfin, Kodi, ...
I use the same SMB Share on my NAS to provide multiple PCs access to my DSLR raw images. Lightroom can be used on multiple PCs, stores all settings in an internal DB, notices if I move folders etc. Kodi re-indexes everything at startup. At the same time, I can serve these images via Jellyfin.

For potential issues, see again the post by @oddlama #1006 (comment)
I will try to answer them here:

This will cause immich to basically have several distributed databases that you need to keep in sync.

It's more like several APIs: A HTTP API and a filesystem API. However, the filesystem is both API and storage at the same time.

Changes to the filesystem (move, delete, add, folders for albums) will have to be processed immediately to keep an up-to-date view.

It's not mandatory to reflect changes immediately (but of course, the sooner, the better): From database terminology: We do not need ACID guarantees, eventually consistent works well enough, i.e. it is okay if changes are only visible with some delay.

What if a user renames a whole album? What happens if a copy of an image is added under a different name? What if the user undo's a filesystem operation? ...

If a file watcher is used, all the operations are queued and executed in order within immich.

Changes to the filesystem that happen while immich is stopped also need to be processed immediately on startup. This requires permanent re-evaluation of the whole structure, as anything could change while immich is stopped: What if a file is renamed while immich is not running? Is this a delete and re-add? Should the move be detected? ...

Yes, on server start a full re-indexing is required. Renames can be detected via hashing (first doing all add operations, then delete all assets from DB no longer found on the filesystem)

You'd basically create a second API for immich that operates on filesystem-level instead of using HTTP, but a lot of operations would be the same.

Agreed. Luckily, the immich server already has a good structure where HTTP Controller and FileWatcher could use the same underlying services (e.g. AssetService, AlbumService etc.)

How to handle thumbnail storage?

Use a separate user configurable folder for storing all thumbs (both from internal assets and external assets)

What if a user deletes a photo by accident? The current approach would immediately delete it from immich too, and SMB shares usually don't have a "Trash" from where the photo can be restored.

Deleting the DB entry and be done. The user will need to restore that photo from trash (or backup). This is how it always has been before throwing immich into the mix :-)

This approach resolves the limitations of 4.

Downsides: Implementation complexity.

related to #7 #34

[bnigh]

Thank you @zoodyy & @oddlama for the detailed threads.

As someone who has only watched immich progress so far (haven't migrated my current setup), these threads help give a clearer insight to features.

I have a few questions / clarifications...

Questions / Clarifications

• @zoodyy - In your post, what is the difference between 3. User defined storage and 4. Take control of existing library?
  • It seems like these are describing the same scenario / wish list
• @zoodyy - Does below Characterizations of Use Cases / Features align with what you were describing?
• For users that desire to manage source of truth themselves (ie: they manage their own directory structure where immich has read only access), what is the desired management pattern / structure for how they will utilize uploads (ie $WATCH_LOCATION vs $UPLOAD_LOCATION)?

Characterization of Use Cases / Features

At a high level, I would say there would be the following patterns for source of truth:

• (1) Immich manages source of truth
• (2) User manages source of truth

(1) Immich manages source of truth

• this would mean that all data would be persisted and managed by immich
• Photos can be uploaded via app, web, or cli and will be persisted at $UPLOAD_LOCATION
• Per my understanding, this is currently supported by immich as mentioned in above post's 1. Import existing library (duplicate file & double storage requirements are ok) section
• When uploading from filesystem via cli, users could decide how to manage original data (whether to delete / keep / move somewhere else), but immich would only care about the data imported

The following feature / enhancements / issues would give greater flexibility:

• (A) Upload Directory (Consume-Only Directory)
  • This feature would aim to have immich manage the existing ability to upload via cli by exposing an upload (consume only / import) directory path (ie: $IMPORT_LOCATION), where users could cd or mv data into which would be uploaded (consumed / imported) into immich, where immich would then persist and manage the data within $UPLOAD_LOCATION
  • Per my understanding, this matches the feature described in above post's 2. Consume-only directory section
  • Per my understanding, this is captured in part by Support watching the file system #7 and Support user defined storage location/structure #34
• (B) Customize file structure
  • This feature would aim to allow customizing the file structure within $UPLOAD_LOCATION
  • Per my understanding, this matches the feature described in above post's 3. User defined storage section
  • Per my understanding, this is captured in part by [Feature]: Option to use a Photoprism-like directory structure for storing images - i.e Multiple level directory vs flat directory. #402
• (C) Customize thumb directory
  • This feature would aim to allow customizing the path and/or structure of the thumbnail directory
  • Per my understanding, this is captured by [Feature]: Separate settings for Thumbnail and Original photo storage locations in .env file #418

(2) User manages source of truth

• this would mean that the user would manage a directory (ie: $WATCH_LOCATION) that immich would essentially have read only view into
• Per my understanding, this is the use case described in the original thread
• Per my understanding, this matches the feature described in above post's 5. Use existing library together with other services/programs section
• Per my understanding, this is captured in part by Support watching the file system #7 and Support user defined storage location/structure #34
• Note
  • It may be impossible for user to manage source of truth completely without losing upload (backup) features. If user wants to use upload from app, web, or cli, these would likely still need to be uploaded to $UPLOAD_LOCATION. immich would need
    • read access to $WATCH_LOCATION
    • write access to $UPLOAD_LOCATION

The following feature / enhancements / issues would be required for this approach:

• (C) Customize thumb directory
  • See above description of (C) Customize thumb directory
  • If supporting read only of $WATCH_LOCATION, this may be a requirement
• (D) Support Gallery Watching (ie: $WATCH_LOCATION)
  • This feature is not currently supported, so would require an or multiple iterative enhancements

[bo0tzz]

what is the difference between 3. User defined storage and 4. Take control of existing library?

'User defined storage' corresponds to your scenario 1B, and 'take control of existing library' corresponds to your scenario 2, to my understanding.

[fyfrey]

@bnigh my point 4. Take control of existing library includes 3. User defined storage. The crucial difference is that 3. does only apply the folder structure to assets added to Immich via app/web/CLI. 4. on the other hand, imports an existing library and also reorganizes all these previously "external" assets according to the user defined storage.

@bnigh @bo0tzz my scenarios 1-4. all fall under the category (1) Immich manages source of truth since Immich owns the assets and can to whatever it wants with it (with 3./4. the user at least gets the chance of read-only view on their assets)
My point 5. Use existing library together with other services/programs aligns with (2) User manages source of truth as Immich only needs read access to the external library and the user / other programs can organize the library freely.

[alextran1502]

You can vote in #7, this is a more detailed focus discussion thread

[kkasberg]

with machine learning/object detection, are there any plans to implement auto-identifying individuals? I would love to have it learn my family members' faces so that I can easily search for images with them in them.

[Peda1996]

Would it also be possible to automatically move favorited images to a subfolder, as this might be quite useful for users who want to at first sort out images, and then only share these fotos with other people.

I have another good idea for human readable folders. It would be nice, if a Reverse Geolocation Information could be added automatically to the automatically generated folder structure. Like:
2022 - 12 Rome

Also I agree, that users should be able to change the directories.

[ollioddi]

Awesome thread with a lot of great ideas. Also really excited if this can get implemented. I would like to give kudos for the current implementation, i think that the primary benefit of Immich is that it is at very good backup tool and that it can put files in a directory as expected with templates.

An issue for me however is owning a "proper" camera, which creates very large files (Raw 45~ MB) compared to my phone. I usually use my camera for family gatherings, travel etc. While im happy with how my phone is backed up, for me it is essential to have my camera photos in separate folders, like an "album" for easy uploading and maintaining huge files which are grouped together. I would not like this to be simply "consumed".

This has another benefit, which is that i can separate my personal photos with the ones i would like to share with my family. I think (if possible), that a good solution would be to give the admin user the possibility to have "shared" directories with other users. Like sharing an album which is actually just its own directory folder. This would also make it easier to do external backups, and again separating personal with shared files.

[atlas-shrugged08]

Heyya... @alextran1502 I was wondering if there's any chance this feature is going to get implemented sometime soon or if you can share your current thoughts on timeline?
Yours is the most promising app as far as I have seen but I have like a 100k pics/videos in an existing folder and would prefer to use that folder directly.

[titro]

Fully agree! For backup purposes it is the best choice cause it also covers the app part and dev progress is really promising. Thanks to Alex and the Team!

But I do not want to use one app or gallery solution for my Smartphone pics and another one for all other existing pics or new pics which are not made with the Smartphone. So this feature is the missing Part to rule them all😉

[skaza2125]

What stops people from importing to Immich then pointing their other apps towards the Immich Photo structure? If all those apps just need a read only directory then it shouldn't be that big of an issue other than the one time hassle of copying photos to Immich.

[atlas-shrugged08]

Point taken.
It's 1.2 TB of data with a carefully crafted folder structure and backup procedures going into multiple locations. So yes, its a hassle for me to reconstruct all of that.

[ollioddi]

Another point is that even if you point other apps at Immich's library, it will not watch for any changes. Say i delete a file from Windows, Immich does not get this update. Also other things such as photo editing, or just moving around for a good spring cleaning would not be picked up.

Backup is also very valid, which you of course can with immich, but restoring might be difficult if everythings not the same. Right now i messed my own library up, and lost all imported assets (yay). This means i have to use the CLI to import everything again.

[etnoy]

the feature is coming

[flozi00]

Is there any way to sponsor especially this Feature ?

[skaza2125]

The devs have decided that this isn't a main priority at the moment, so even if you sponsor it, I would be hesitant to say it will be implemented soon. Immich isn't a for-profit app. Its goal is to replicate google photos.

[etnoy]

I am currently working on this feature, stay tuned

[update-freak]

did I understand it correctly that when this feature is implemented I can use the pictures in \volume1\pictures as main path for the albums, e.g. \volume1\pictures\2023 Bodensee, \volume1\pictures\2023 Frankreich? For me it's important that I did not have to upload the pictures again in Immich (having all the files double). Then a periodic scan for new files in that directory whould be good (e.g. at 4 am in the night). Another important thing is that I can deactive the deleting of pictures in Immich, so only via stystem it is possible to delete pictures (that I did not delete pictures when I did not want to delete ones).

[davidrudlstorfer]

Another thing that should be mentioned and I wasn't able to find in the issue or in this discussion is the following:

Currently I organize my photos (DSLR and mobile) in the following way:

--> 2021
    --> ALBUM 1
        - 001.jpg
        - 002.jpg
    --> ALBUM 2
--> 2022
    -->  ALBUM 3
        --> RAW
            - 001.arw
            - 002.arw
        --> EDITED
            - 001.jpg
            - 002.jpg
        --> MISCELLANEOUS
            - random_file.txt

When the read only mode of Immich is implemented it would be nice to add a ignore file (for example like git) to ignore certain subdirectories or even file types from importing. In my special use case I would only import my edited photos to Immich but not the original RAW files.

Great project and thanks in advance!

[Spectre5]

This would be great!

[akshara-tg]

Ability to exclude certain folder is a great feature. By the way, I hope you mean indexing in your statement @davidrudlstorfer "In my special use case I would only import my edited photos to Immich"

[davidrudlstorfer]

Good catch @akshara-tg! Indexing would have been the better choice 😆

[titro]

we all agree that this is a super cool functionality. so let's hope the devs / team change their current decision and give that gallery watcher feature more attention :-)

[etnoy]

I'm working on this exact feature, stay tuned

[Yacubane]

Hey, it's the most requested feature, are there any plans or roadmap for implementing this feature? ☺️
I'm aware that read-only gallery feature does exists, but at current state it's too restricted.

[alextran1502]

Improvements are being worked on. They are all in experimentation state because of the complexity of the feature :)

[titro]

The current experimental version works as it it described. Looking really forward for the development of this feature!
Cause this thread is in the meantime very long: Is there a feature list how the gallery watcher should work at the end or is the summary in post no 1 the current version? I already gave this a try and want to give some input for functionalities (if it is appreciated)

[etnoy]

Things are currently happening to completely re-implement the existing photos import

[etnoy]

For those who want to track my progress in implementing this awesome feature, see this PR: #3124

[etnoy]

Fixed in #3124