IOS application stops upload

[german-aksenov]

The bug

fresh installation of immich. I'm trying to upload images from mobile phone (iOS 16.6) to server.

2269 images was transferred and progress is stoped (still 247 in queue)
in the mobile application log there is

Ignoring duplicate assets on device:
{
"id": "N/A",
"remoteId": "N/A",
"localId": "DA49F295-C7C7-4778-A546-9BCD06840EEF/L0/001",
"checksum": "w1C5UxYBQV0FhqQ0Fzopbw5yp9A=",
"ownerId": 7102608187667092359,
"livePhotoVideoId": "N/A",
"stackCount": "0",
"stackParentId": "N/A",
"fileCreatedAt": "2019-11-10 12:39:20.000",
"fileModifiedAt": "2023-08-12 11:56:59.000",
"updatedAt": "2023-08-12 11:56:59.000",
"durationInSeconds": 0,
"type": "AssetType.image",
"fileName": "",
"isFavorite": false,
"isRemote: false,
"storage": "AssetState.local",
"width": 568,
"height": 1152,
"isArchived": false,
"isTrashed": false,
}
{
"id": "N/A",
"remoteId": "N/A",
"localId": "3549205F-9AE9-4F3B-92DF-F85A2F9866A4/L0/001",
"checksum": "w1C5UxYBQV0FhqQ0Fzopbw5yp9A=",
"ownerId": 7102608187667092359,
"livePhotoVideoId": "N/A",
"stackCount": "0",
"stackParentId": "N/A",
"fileCreatedAt": "2019-11-10 12:39:20.000",
"fileModifiedAt": "2023-08-12 11:56:59.000",
"updatedAt": "2023-08-12 11:56:59.000",
"durationInSeconds": 0,
"type": "AssetType.image",
"fileName": "",
"isFavorite": false,
"isRemote: false,
"storage": "AssetState.local",
"width": 568,
"height": 1152,
"isArchived": false,
"isTrashed": false,
}

looks like very old images unable to be uploaded.

The OS that Immich Server is running on

Docker compose + debian

Version of Immich Server

1.84.0

Version of Immich Mobile App

1.84.0 build.124

Platform with the issue

• Server
• Web
• Mobile

Your docker-compose.yml content

version: "3.8"

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    command: ["start.sh", "immich"]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env
    depends_on:
      - redis
      - database
      - typesense
    restart: always

  immich-microservices:
    container_name: immich_microservices
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    extends:
       file: hwaccel.yml
       service: hwaccel
    command: ["start.sh", "microservices"]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env
    depends_on:
      - redis
      - database
      - typesense
    restart: always

  immich-machine-learning:
    container_name: immich_machine_learning
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    volumes:
      - model-cache:/cache
    env_file:
      - .env
    restart: always

  immich-web:
    container_name: immich_web
    image: ghcr.io/immich-app/immich-web:${IMMICH_VERSION:-release}
    env_file:
      - .env
    restart: always

  typesense:
    container_name: immich_typesense
    image: typesense/typesense:0.24.1@sha256:9bcff2b829f12074426ca044b56160ca9d777a0c488303469143dd9f8259d4dd
    environment:
      - TYPESENSE_API_KEY=${TYPESENSE_API_KEY}
      - TYPESENSE_DATA_DIR=/data
      # remove this to get debug messages
      - GLOG_minloglevel=1
    volumes:
      - tsdata:/data
    restart: always

  redis:
    container_name: immich_redis
    image: redis
    restart: always

  database:
    container_name: immich_postgres
    image: postgres:14-alpine@sha256:28407a9961e76f2d285dc6991e8e48893503cc3836a4755bbc2d40bcc272a441
    env_file:
      - .env
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
    volumes:
      - pgdata:/var/lib/postgresql/data
    restart: on-failure:5

  immich-proxy:
    container_name: immich_proxy
    image: ghcr.io/immich-app/immich-proxy:${IMMICH_VERSION:-release}
    ports:
      - 2283:8080
    depends_on:
      - immich-server
      - immich-web
    restart: always

volumes:
  pgdata:
  model-cache:
  tsdata:

Your .env content

# You can find documentation for all the supported env variables at https://immich.app/docs/install/environment-variables

# The location where your uploaded files are stored
UPLOAD_LOCATION=/immich_lib

# The Immich version to use. You can pin this to a specific version like "v1.71.0"
IMMICH_VERSION=release

# Connection secrets for postgres and typesense. You should change these to random passwords
TYPESENSE_API_KEY=hidden
DB_PASSWORD=hidden

# The values below this line do not need to be changed
###################################################################################
DB_HOSTNAME=immich_postgres
DB_USERNAME=postgres
DB_DATABASE_NAME=immich

REDIS_HOSTNAME=immich_redis

LOG_LEVEL=debug
PUBLIC_LOGIN_PAGE_MESSAGE="My Family Photos and Videos"

Reproduction steps

1. fress install immich ios app and server
2 login via mobile app 
3 manually start syncing

Additional information

No response

[alextran1502]

Are you using DNS or local IP?

[german-aksenov]

DNS

[alextran1502]

You will need to add max_client_body_size to your proxy otherwise you cannot upload large files.

https://immich.app/docs/administration/reverse-proxy#nginx-example-config

[german-aksenov]

i have
client_max_body_size 0;

[alextran1502]

Can you help capturing a video of the issue?

[agross]

@alextran1502 I think you misunderstood. My point was that uploads hang while the app is in the foreground, but resume immediately when switching away and back to the app. See the end of the recording (0:35-0:45) where nothing happens even though the app is foregrounded. I could have submitted a 5-minute video of "no uploads proceeding".

[alextran1502]

Hmm, I would like to see a log for when it gets stuck. I think I might have put in a fix for this in the next release already. Let's try again

[agross]

@alextran1502 If you are referring to ffc31f0, then yes the log contains lines related to your commit (I think).

2023-12-18 23:49:09.884686,LogLevel.SEVERE,"ImmichErrorLogger","Catch all error: PathNotFoundException: Cannot open file, path = '/private/var/mobile/Containers/Data/Application/BF4D5A11-611C-4F0E-9451-25D2380B6F0A/tmp/.video/5DBAF376-B585-4682-9603-65030327FC4B_L0_001_1700329609.261448_o_copy_E9A120C7-CDBF-4E41-B8C5-417A6438C3FC.mov' (OS Error: No such file or directory, errno = 2) - PathNotFoundException: Cannot open file, path = '/private/var/mobile/Containers/Data/Application/BF4D5A11-611C-4F0E-9451-25D2380B6F0A/tmp/.video/5DBAF376-B585-4682-9603-65030327FC4B_L0_001_1700329609.261448_o_copy_E9A120C7-CDBF-4E41-B8C5-417A6438C3FC.mov' (OS Error: No such file or directory, errno = 2)","#0      _checkForErrorResponse (dart:io/common.dart:55)
#1      _File.open.<anonymous closure> (dart:io/file_impl.dart:381)
<asynchronous suspension>
#2      _FileStream._start.onOpenFile (dart:io/file_impl.dart:153)
<asynchronous suspension>
"
2023-12-18 23:49:08.101801,LogLevel.SEVERE,"ImmichErrorLogger","Catch all error: PathNotFoundException: Cannot delete file, path = '/private/var/mobile/Containers/Data/Application/BF4D5A11-611C-4F0E-9451-25D2380B6F0A/tmp/.video/F51C0510-C162-4199-BAD6-321CE49686EB_L0_001_1661061176.937932_o_IMG_3460.MOV' (OS Error: No such file or directory, errno = 2) - PathNotFoundException: Cannot delete file, path = '/private/var/mobile/Containers/Data/Application/BF4D5A11-611C-4F0E-9451-25D2380B6F0A/tmp/.video/F51C0510-C162-4199-BAD6-321CE49686EB_L0_001_1661061176.937932_o_IMG_3460.MOV' (OS Error: No such file or directory, errno = 2)","#0      _File.throwIfError (dart:io/file_impl.dart:675)
#1      _File._deleteSync (dart:io/file_impl.dart:329)
#2      FileSystemEntity.deleteSync (dart:io/file_system_entity.dart:424)
#3      BackupService.backupAsset (package:immich_mobile/modules/backup/services/backup.service.dart:398)
<asynchronous suspension>
#4      BackupNotifier.startBackupProcess (package:immich_mobile/modules/backup/providers/backup.provider.dart:461)
<asynchronous suspension>
#5      BackupNotifier._resumeBackup (package:immich_mobile/modules/backup/providers/backup.provider.dart:592)
<asynchronous suspension>
"
2023-12-18 23:49:04.801180,LogLevel.SEVERE,"ImmichErrorLogger","Catch all error: PathNotFoundException: Cannot open file, path = '/private/var/mobile/Containers/Data/Application/BF4D5A11-611C-4F0E-9451-25D2380B6F0A/tmp/.video/5DBAF376-B585-4682-9603-65030327FC4B_L0_001_1700329609.261448_o_copy_E9A120C7-CDBF-4E41-B8C5-417A6438C3FC.mov' (OS Error: No such file or directory, errno = 2) - PathNotFoundException: Cannot open file, path = '/private/var/mobile/Containers/Data/Application/BF4D5A11-611C-4F0E-9451-25D2380B6F0A/tmp/.video/5DBAF376-B585-4682-9603-65030327FC4B_L0_001_1700329609.261448_o_copy_E9A120C7-CDBF-4E41-B8C5-417A6438C3FC.mov' (OS Error: No such file or directory, errno = 2)","#0      _checkForErrorResponse (dart:io/common.dart:55)
#1      _File.open.<anonymous closure> (dart:io/file_impl.dart:381)
<asynchronous suspension>
#2      _FileStream._start.onOpenFile (dart:io/file_impl.dart:153)
<asynchronous suspension>
"
2023-12-18 23:49:03.688593,LogLevel.INFO,"BackupNotifier","_getBackupAlbumsInfo: Found 29 available albums",""
2023-12-18 23:49:03.453509,LogLevel.INFO,"SyncService","Syncing a new local album to DB: Recents",""
2023-12-18 23:49:03.305650,LogLevel.INFO,"AlbumService","Found 29 device albums",""
2023-12-18 23:49:03.061425,LogLevel.INFO,"AssetNotifier","Load partner assets: 142ms",""
2023-12-18 23:49:03.060580,LogLevel.INFO,"SyncService","Upserted 2 assets into the DB",""
2023-12-18 23:49:02.946712,LogLevel.INFO,"BackupNotifier","Found 45 local albums",""
2023-12-18 23:49:02.928338,LogLevel.INFO,"BackupNotifier","[_resumeBackup] Start back up",""
2023-12-18 23:48:53.680862,LogLevel.SEVERE,"ImmichErrorLogger","Catch all error: PathNotFoundException: Cannot open file, path = '/private/var/mobile/Containers/Data/Application/BF4D5A11-611C-4F0E-9451-25D2380B6F0A/tmp/.video/5DBAF376-B585-4682-9603-65030327FC4B_L0_001_1700329609.261448_o_copy_E9A120C7-CDBF-4E41-B8C5-417A6438C3FC.mov' (OS Error: No such file or directory, errno = 2) - PathNotFoundException: Cannot open file, path = '/private/var/mobile/Containers/Data/Application/BF4D5A11-611C-4F0E-9451-25D2380B6F0A/tmp/.video/5DBAF376-B585-4682-9603-65030327FC4B_L0_001_1700329609.261448_o_copy_E9A120C7-CDBF-4E41-B8C5-417A6438C3FC.mov' (OS Error: No such file or directory, errno = 2)","#0      _checkForErrorResponse (dart:io/common.dart:55)
#1      _File.open.<anonymous closure> (dart:io/file_impl.dart:381)
<asynchronous suspension>
#2      _FileStream._start.onOpenFile (dart:io/file_impl.dart:153)
<asynchronous suspension>
"

[agross]

I'm happy to report that with the above fix for the client all remaining 5k assets were uploaded today!

[alextran1502]

@agross Amazing! Thank you for reporting back!

[german-aksenov]

video5346315744634222561.mp4

[german-aksenov]

I just manually mark all assets on the phone not backed up.. and push them to the server. All loaded, but in backup screen nothing changed ;( the automatic upload still doesn’t work. The Remainder count still show the same items to do

[alextran1502]

Can you get the logs from the mobile app as well as the logs on server. Also the screenshot of the upload page on the mobile app?

[german-aksenov]

Immich_log_2023-11-04T21_08_17.920082.csv
immich_server_logs.txt

[german-aksenov]

found some errors in postgree. can it be related to the issue?
immich_postgres_logs.txt

[german-aksenov]

Immich_log_2023-11-04T22_32_50.295557.csv

more detailed log while upload stuck

[german-aksenov]

reinstall the app. the issue still exist (direct connection via lan to local immich). the upload freeze on this photo. May be it can helps

[alextran1502]

Can you temporarily remove that photo and see if the app will continue to upload?

[german-aksenov]

No. it doesn't help. moreover the Remainder value did not changed.
then i made screenshot it was sent to server....

[german-aksenov]

1.85.0 problem still here

[aviv926]

Do you have enough server storage?

[german-aksenov]

Yes.
/dev/mapper/beelink--vg-immich 102626232 21547240 75819728 23% /immich_lib

[aviv926]

You tried to upload photos from another device, it is recommended to try it in order to verify that there is no problem on the server side.
Since even after deleting the "problematic" image the problem still continues, I would recommend performing such an experiment

[german-aksenov]

hi, thanks for the support. try on the another iphone (xr) - it is ok. all assets are uploaded without errors.
created new account for my iphone - the error is here. upload stops at the same point 245 remainder.
made anothe experiment: turn on sync for albums one by one (removing Recent). some have remaining asstes.
for example: i have 1 photo in album Instagram (visible on iphone foto gallary) and 1 photo remaining for backup for this album (but immich do not recognize this foto



)

[german-aksenov]

found some more information. this problematic foto on iphone. then i press edit buttom in the iphone photo gallery there is an error. Looks like some problem with photo and this do not allow to upload. may be immich should just skip this

?

[alextran1502]

Can you somehow get the photo and share it with us so we can use it for troubleshooting?

[german-aksenov]

the iphone do not allow to share this photo :( any action to copy file from iphone (Save to Files, send by mail, etc ) show error:
"Unable to Share There was an error while preparing to share. Please try again later". Looks like this photo was stored in icloud and only preview on the iphone.... and somehow the photo is missed in icloud now. any way... i install thirdparty app to extract photo. here it is
0001 IMG_1411.zip

[pgebheim]

For what it's worth I'm having this happen as well and it has nothing to do with a specific photo.

In my case I see logs in the app saying it is unable to delete some photo from a local cache and the process stalls. This has happened across a variety of photos -- stopping and starting the sync manually gets it unstuck.

I believe this is related to the process the phone goes through when you access a photo which is actually on iCloud.

[Pummelfisch]

Bug was also reported here #9870 recently. There are two reports with the exact same date 4th October 2020 without a filename. It looks really similar to the original report of OP #4837, since that one also got stuck on a file from the 4th October 2020.

[eligao]

Me too always got backups stuck on a file from the 4th October 2020, was that a special timestamp?

[alextran1502]

Yep it is a place holder. The next release will probably fix the issue of file getting stuck and have more info in the log of which files cause the stuck

[Pummelfisch]

Got fixed for me with the current update of the iOS-App.

[RaZer0r]

i still have the issue. 11 items remain to be uploaded. Which logs should i look into to find out which files the problem could be?

[eligao]

Disabling "ignore iCloud photos" seemed to have fixed my problem, even though the progress still seem off, as it continues uploading a few items after the remainder count reaches 0 and backup count > total count.
Anyways the upload logic just got refactored in #18726, hope the legacy issues are gone in the next release.