Yet Another Immich Migration Guide

[bitsondatadev]

Yet Another Immich Migration Guide

One migration guide to rule them all. 💍

Overview

The goal of this guide is to be a central guide where the community can update and consolidate the dispersion of different workflows and scripting towards a more centralized approach to migration to Immich.

Why?

This will become more important as the community becomes less technically savvy and people support their families performing our favorite task...data migrations on all their data (godspeed). As the community builds towards a more common list of well tested and understood steps, we can then work with the core team to simplify those steps, enable customization of migration from multiple sources and for various preprocessing tasks to verify most paths and one off issues are handled.

Getting the migration portion of this story ironed out is so important and commonly overlooked since most developers feel a sense of accomplishment by writing their cool custom scripts that only make sense to them. This is good because it's how projects like Immich get started in the first place. But this technology should eventually lower the assumptions that these tasks are handled by software developers or hackers. That said, we can't and shouldn't just jump from where Immich is today to over-engineers GUI features that over-obfuscate the internals. In this way it's valuable to take the slow journey as a community to gather what inputs different migration steps need today and in the future as Immich adoption soars and people gain autonomy over their data.

Guide Goals

1. Community driven migration process development. (more)

   • Encourage everyone to point to this guide versus create a new guide or find old ones.
   • If there is new information to be added, comment below and tag me and I will update this guide. If want to have write access to update your portion of the docs, then provide a link and tag me so I can link to your docs.
   • We should add pointers to alternative ways while carving out a [recommended] happy path.

2. Updated to use latest Immich features. (more)

   • The project is quickly adding better features to handle issues with migration.
   • If the core maintainers could kindly inform us when if they see something that could improve/impact this guide that would be valuable.
   • Initially minimize information relevant to older versions in an info box.

3. Organize features to simplify migration steps. (more)

   • Assist with backlog maintenance for open discussions/issues/PRs related to this migration guide's workflow, and remove those requests so that other migration issues surface faster.
   • Work with the Immich maintainers to help prioritize adding features to make simplify the migration guide to fewer and simpler steps that still enable custome workflows.

Enough rambling, here's the dang guide already!

Guide

Prerequisites

1. Install Podman (Recommended) or Docker.
2. [Optional]: Install exiftool if needed for the exif inspection or manual fixing errors.
3. Verify space requirement on your system.
   Image and video libraries tend to exceed hundreds of gigabytes so they are exported in compressed formats that make it faster and easier to transfer, but you'll take up space for both the compressed file and decompressed files after you extract the data.

Phase 1: Download and Extract the data

1. Follow the download instructions from your given source:
   • Google Photos
     • [Recommended] When asked about the size for the Google photos export, use the largest file setting (e.g. 50 GB).
   • Apple Photos
2. Extract the data:
   • Google Photos will export using zip files chunked into smaller sets of archived files based on the option you chose during the download.
     • Zip Files Having all the zip files in the same directory, run the following command to unzip the files in the same directories. There shouldn't be any overlap in files, just directories of those files, which will either map to or create the shared unzipped directory:

          unzip 'takeout-YYYYMMDDTHHmmssZ*.zip'
       

       ⚠️ IMPORTANT
       Make sure to change the date fillers to your takeout export date., the '' characters are needed for multiple zip files, and unzip has only been tested on Ubuntu Linux.

   • 🗒️ TODO
     Apple Photos instructions

Phase 2: Process image metadata

The primary metadata that exists on images that comes in handy during imports are the EXIF data, Album data, geolocation, and other data possibly stored by the source provider like face tagging. Since there is no current state-of-the-art tool for fixing these issues, we're going to use the Docker install of the garzj/google-photos-migrate project. If you prefer npm, feel free to take that route and skip this section. To keep consistent with using Docker, we will follow these steps:

1. Download the google-photos-migrate repository, unzip it and move into the google-photos-migrate directory. You can optionally use git clone if you prefer.

   # build the image
   docker build -f Dockerfile -t localhost/google-photos-migrate:latest .
   

2. Create an output and error directory in the root directory of the google-photos-migrate repository.

3. Assuming your unzipped Google Takeout folder is located at ~/Pictures/Takeout/, run the following command with the takeout directory updated according to your location.

   docker run --rm -it --security-opt=label=disable \
       -v "$(readlink -e ~/Pictures/Takeout/):/takeout" \
       -v "$(readlink -e ./output):/output" \
       -v "$(readlink -e ./error):/error" \
      localhost/google-photos-migrate:latest \
        full '/takeout' '/output' '/error' --timeout=60000
   

4. (Optional) if there are files in the error folder, you can use exiftool to attempt to fix those issues manually, or just ignore them (the files without metadata are there already). See this tutorial for some ideas on how to use exiftool.

   • For the lazy, consider at least running a separate upload of these files to a different Immich album so that they will ideally be found as duplicates and added to a central album like fixme-album. One day you'll get to them...I have faith in you.

5. The resulting images are in the output folder.

🗒️ TODO
Apple Photos steps, geo location, face tagging data?

ℹ️ NOTE
Hopefully a lot of these editing features will be added into immich-cli proper and handle various edge cases in a streamlined way.

Phase 3: Upload images to Immich

This phase assumes you've created the user you intend to upload data for. As done in previous sections, we will also use the Docker command to upload the

1. If you don't already have one for the user you're uploading data for, create a new Immich API key from their account.

2. Run the upload command.

   I assume you'll want to do recursive and album options if you have to enable copying the album names automatically based on the directory structure. You may also want to add --dry-run to verify nothing blows up. Here's a list of all the options for the upload command.

   docker run -it -v "$(pwd)":/import:ro \
     -e IMMICH_INSTANCE_URL=http://localhost:2283/api \
     -e IMMICH_API_KEY=MY_IMMICH_KEY_ABC123 \
     ghcr.io/immich-app/immich-cli:latest upload . --recursive --album
   

3. Verify things are looking good on the web browser.

   • See Appendix A below for common issues.
   • See Appendix B for features and discussions aimed to improve migration.

Appendix

Appendix A: Common issues

• Shared photos that you didn't own show up as blank photos, or blank copies of photos.

• 🗒️ TODO
  Fill this out

Appendix B: Migration-related features and discussions

• Transfer photos from user A to user B
• Automatic creation of albums in external libraries
• iOS original livephotos video uploaded vs edited one

• 🗒️ TODO
  Fill this out

Appendix C: Migration tools

• immich-go

  • ✅ 🔥 badass cli interface
  • ✅ reduces OS/Framework dependencies needed to run
  • 😐 Fast beta features that aim to improve upon the supported cli but not far ahead of cli
  • ❌ not tested well outside of Google Photos
  • ❌ not maintained by core maintainers
  • opinion: since the cli has kept up closely with the go features and fixed original limitations from when immich-go was created, having both immich-go and immich-cli tends to confuse new users. Ideally immich-go maintainer can merge their efforts with core immich team, potentially providing Docker/Go binary). This guide will aim to recommend immich supported options where they exist and just using Docker/Podman to generalize the toolset.

• mattwilson1024/google-photos-exif

  • ✅ set the foundations for a bunch of other great tools like
  • ❌ last updated Oct 2021
  • ❌ as the creator stated, this tool was specific to their setup

• CrcCheckCopy

  • ❌ Immich now has this feature 💪🏻
  • ❌ Only works for Windows/Mac no Linux

• delete_photos.js

  • ❌ Immich now has this feature 💪🏻

• gphotos-sync Python tool

  • ❌ Uses Googles API vs export 🙅🏻

Appendix D: Existing guides

• gphotos import + albums migration
  • ✅ basics of google photos migrations
  • ✅ preserves exif data
  • ✅ the exiftool step to fix the errored data.
  • ✅ preserves albums
  • ✅ has a bunch of gems in the comments
    • nice comparison of google takeout helpers
    • there may be some plans to migrate pre-processing features into immich-cli which would remove one set of toolsets but depends when immich folks get bandwidth
  • ❌ lacks some handholding (dev only)
  • ❌ lacks Apple and other sources
  • ❌ last updated Jan 2023
  • ❌ not using newer Immich features
• google-photos-takeout-scripts
  • ✅ basics of google photos migrations
  • ✅ excellent overview of exiftool operations
  • ❌ uses shell scripts for optimum pain an 0 interoperability
  • ❌ lacks handholding (dev only)
  • ❌ lacks Apple and other sources

[tenbeh5]

Hi @bitsondatadev, thank you for such a great documentation for photos migration for immich.
Do you have any update on Apple Photos instructions?

[jac4e]

Some of the steps and scripts from this repo could be useful for migrating from iCloud: https://github.com/platelminto/icloud-to-nextcloud
Specifically the bulk download from iCloud and the Photos steps 1,2,3,4, and 5b would get you into a position where you have all your iCloud photos in album based directories with restored EXIF data.

[jac4e]

Some of my photo metadata csv's were empty which broke the merge csv script. Just had to rename them to continue.

[jac4e]

By default, the merge csv script should be ran with your CWD being the Photos_All folder.
Afterwards the generated csv should be moved from its child folder in the Photos_All folder to the Photos_All folder.
Change your CWD to the parent dir of Photos_All before running the add exif data script.

[tenbeh5]

Thank you @jac4e

[jac4e]

I am in the process of doing this right now, just waiting for downloaded zips to unzip. Hopefully I can give some good news on if it worked in a few hours

[tenbeh5]

I am actually testing from-icloud sub command using immich-go from the documentation https://github.com/simulot/immich-go?tab=readme-ov-file#from-icloud-sub-command

[Nykri]

Hi @bitsondatadev ! I mainly agree with what you said : the immich-cli is getting good and can perform most migrations. But it lacks a few options and some user-friendliness. Maybe a companion GUI in the administration page could resolve both issues.
It would of course allow basic users to perform their migration without using the cli. But it would also enable much more customization to fit migration to everyone's needs.

For example my original library has dates in folder name so I modified immich-cli to clean up folder names before creating albums. This works for me but this is quite hard to keep customizable from cli, you would need to pass regex as arguments (like here). Imo it would rapidly turn to a nightmare with too much options to pass to immich upload. This is a case where a GUI would prove useful with options to tick and a live preview of what album names will be.

I also agree with the fact that efforts should be centralized in a single tool and not hundreds of user specific scripts (even if they are really useful to do things quick and dirty). Maybe a companion GUI should be initiated and progress hand-in-hand with immich-cli as option are added.