[Feature] Add a job detecting missing assets and removing broken database entries

[boobin]

The feature

I used a tool czkawka to detect and remove duplicate or very similar photos directly from the filesystem. Then i made a script to get the list of assets with path from the database, validate all the paths and listing all asset ids pointing to deleted paths, and sending the id list to DELETE asset api endpoint.

I worked as intended AFAIK, but it could be added as a maintenance command on the server.

Platform

• Server
• Web
• Mobile

[AngelaDMerkel]

This is a very similar request to #1968 because the goal is the same. Introducing a deduplication function into Immich directly would mitigate the need to detect missing assets and remove broken database entries.

[boobin]

Sure, but a cleanup job allows to use existing tools as well. Having both would be best.

Direct deletion on the disk can be easier for duplicates or similar photos, or any other use case already handled by dedicated tools.

[feerlessleadr]

Mind sharing the script that you ran to detect asset IDs pointing to deleted paths? I'm looking to do something similar.

Thanks

[boobin]

I didn't do a full automated script. I used an "adminer" container to get a CSV extract of assets id+path ( SELECT assets."id", "originalPath" FROM "assets" ), then i listed missing ids with:

#!/usr/bin/env bash

basePath=/mnt/nas/storage/immich-data

while IFS="," read -r asset_id file_path
do
  prefix_removed_path=${file_path/#upload\/}
  full_path=$basePath/$prefix_removed_path
  if [[ ! -f $full_path ]]; then
    echo "\"$asset_id\""
  fi
done < <(tail -n +2 ~/Téléchargements/asset_extract.csv)

Then i put this list in a proper JSON array to send to the DELETE endpoint.

It would be possible to do that fully automated but would probably need to have a DB port open to run the query where you can access the assets as well.

[feerlessleadr]

Thanks - seems a bit out of my league.

[dima-ser]

This would be very useful feature. My workflow is manually moving files from Immich upload directory into another, organized directory which I then use as external library in Immich. When I do that, Immich shows duplicate images on the timeline and no way to remove the duplicate thumbnail/orphan database entry. Being able to run a job that removes all missing photos from Immich would be great.