[Feature] Auto Image Rotation Detection

[Tyree]

I have searched the existing feature requests to make sure this is not a duplicate request.

• Yes

The feature

Certain apps* have a feature that detects rotated images and curates them for the user to choose to rotate them as suggested or not. Similar to how Immich now detects duplicates, I'd love to see it be able to automagically find these rotated images!
Thanks!

#1695 is similar, but it's for a manual image rotation tool and not for auto detection. Perhaps it could just be extended to include an auto-detect functionality.

Platform

• Server
• Web
• Mobile

[NicholasFlamy]

A message with a suggestion for this, including a link to an existing AI model for this:
https://discord.com/channels/979116623879368755/1129456657995403294/1246841616191324160

Link to existing AI model:
https://github.com/pidahbus/deep-image-orientation-angle-detection

[hexxone]

+1 for the idea
Thanks for reposting this. Would be very cool to see!

[Chuckame]

If we don't want to touch the original photo, we could generate related xmp sidecar using the tiff:Orientation property coming from Adobe xmp specs

[NicholasFlamy]

Pretty sure external libraries can already write sidecar XMP files with image editing and rotating if you have read-only disabled.

[dionorgua]

It looks like Immich doesn't support tiff:Orientation.
It doesn't work for me:
#15549
#15550

[parsapoorsh]

i had this problem, but i couldn't find a lightweight model to do rotation detection, so i glued one together:
https://github.com/parsapoorsh/resnet-ixion

it's size is about ~220 MB, and it has a great performance for images that have an object or face in them (like the images in demo instance)

but it can't detect rotation of scanned papers or an empty wall and stuff like that

[parsapoorsh]

i'm downloading ImageNet 2017 to train a better version. the current version generally has good accuracy, but it struggles with edge cases.

[hexxone]

fyi, I couldnt wait and wanted to try your model as well, so I forked your repo and added a CLI tool for batch processing:
https://github.com/hexxone/resnet-ixion

Disclaimer: I'm not really confident in my python and AI knowledge, but I ran this tool on my external library of photos I was planning to import into immich and it seems to yield good results:

Completed: 100% (10681/10681)
Total time: 7m 25s
==== batch processing complete ====
Processed: 10681 images
Had Errors: 24 images
Inconsistent: 644 images
Corrected 515 images

the errors are mainly caused by video files, because I didnt bother to exclude them yet.
the inconsistent amount could probably be brought down further as well, I just didnt know the best way/math for doing so..
Maybe this helps in some way :) Cheers

[parsapoorsh]

hi, sorry for the delay and not being around, im currently not in a location with access to my PC and my Laptop isn't a gaming one, so i can't continue doing the AI stuff. i will resume soon.

Disclaimer: I'm not really confident in my python and AI knowledge

im not an AI expert ether. im new to this, but doing this stuff has become easier then ever and there are great resources around to learn from.

added a CLI tool for batch processing

i made a GUI for batch processing (gui.py), which shows the images that it's not confident enough (score lower then 98%), to be manually decided.

through this, i found out the model isn't reliable enough to be a part of immich yet, most of the time it performs really good, but for some images it flops badly.

edge cases i found where model is not reliable:

1. if there aren't enough obvious objects
2. low brightness
3. the image is near a 45 deg angle
4. text documents

possible solutions:

1. im thinking to solve the "not enough obvious objects" problem, instead of fine-tunning resnet152, train efficientnet b3 on ImageNet 2017 + MS COCO, but it's gonna take like a week per epoch on my GTX 1660 super

2.3. for the low brightness and bad angles, i can randomly rotate the images 11.25 deg, make them dark and stuff like that (data augmentation), which needs more then 1 epoch for the training to have the optimal results (time-consuming)

4. text documents are a bit tricky, we need the model to detect a text's angle, (not OCR, just angle detection).
   but i have no idea how make it to do so.
   maybe it's possible by overlaying some random text on training images, but i think that going to harm the main normal image rotation detection.
   also the text needs to be in multiple visually different writing-languages, like english, russian, arabic, hindi, etc, and in different fonts, which just complicate things for document angle detection.
   because of all of these complications, i believe documents rotation detection and normal image detection should have different models.

video rotation detection:

the errors are mainly caused by video files

doing this for videos is easy, for speed reasons just extract the keyframes (related to video compression), exclude the visually similar ones, and run the model on reaming ones, average the scores.

basically this ffmpeg command:

ffmpeg -skip_frame nokey -i input.mp4 -vf "mpdecimate,select='eq(pict_type,I)*gt(scene,0.35)'" -vsync vfr output_%04d.jpg

here is the result on Bad Apple (3m40s video, 78 pictures output):

later i will write a video-rotation-detection script using this.

[hexxone]

Hey, thanks for your continued efforts!

I also think it will take a while to get this feature ready for immich to cover all the edge cases.
Fortunately for me, I had mostly very obvious photos that were wrongly rotated by an older digital camera, and for me, just leaving out the "inconsistent" images is enough for now. I just couldnt be bothered to manually check those >500 images ^^ hence the CLI tool...

Training a better model to cover the other edge-cases seems like a good idea, I will gladly provide some training time on my 3090 if that helps this awesome community :) I probably just need some help/instructions with downloading & setting up the datasets correctly ^^

Just hit me up when you get to the point 👍

[AllTaken]

Sounds a lot like you are letting the "perfect" get in the way of of the good.
Having something in Immich that would suggest rotations for images with a decent accuracy, would already be a very useful addition.
As long as there in general are more accurate suggestions than false positives, I would argue that it is a very useful feature.
I think in general people have thousands of images. If there's a suggestion to rotate for maybe tens of images, and even 3 of them are correct, it would still be useful, since it basically takes no time to go through tens of images as a human, whereas thousands is basically unfeasible.
Later increments in accuracy would just improve the solution.

[sildur]

DigiKam has recently added AI autorotation functionality. The details are in this commit:
https://invent.kde.org/graphics/digikam/-/commit/46002a51837355644ac2f6c0addb6ea4b3a43750

They are using deep-image-orientation-angle-detection.onnx, which can be found here:
https://huggingface.co/Chuckame/deep-image-orientation-angle-detection/tree/main

More details about the model can be found here:
https://github.com/pidahbus/deep-image-orientation-angle-detection

[Chuckame]

Amazing! I've started in the past, trying to include the model and corresponding apis in immich, but the python part feels really complex to modify...

[abanck]

Seems part of implementing this (and manual rotation tool) is supporting reading image orientation from an XMP sidecar file. I looked at several related issues and PR's and in some this is referenced but could not find an issue/PR directly for reading the XMP file orientation. Is there a related open issue/PR for this, or any plans for this?

Like several other tools exiftool supports creating XMP sidecar files with the Orientation specification so reading this in Immich would help integration of image rotation detection greatly.