Duplicate image detection

[manthey]

I'm adding this here just to surface the content better.

As part of the ImageDePHI project (https://github.com/DigitalSlideArchive/ImageDePHI), we wanted to detect duplicate images, possibly with different metadata. For instance, if a WSI file had been deidentified, the metadata will be different but the image is the same. We made a small gist to show off how you could compare images within IFDs in TIFF here (https://github.com/DigitalSlideArchive/large-image-utilities/blob/main/ifdhash.py). This takes the sha512 of the combined compressed tiles in order for each ifd. If the sha512 hash matches, the image is identical and identically compressed (within the collision space of sha512).

This could equally be applied to other formats that do not alter tiles (e.g., converting the same image to DICOM).