[BUG] Connection terminated due to connection timeout when re-running "Extract Metadata" job

[florihupf]

The bug

Just upgraded to 1.83.0 but this also happened with 1.82.1: When running the "Extract Metadata" job I get a lot of connection errors (every 1-2s).

If I let the job continue I end up with missing thumbnails.

Example logs:

[Nest] 7  - 10/29/2023, 5:56:59 AM   ERROR [ExceptionsHandler] Connection terminated due to connection timeout
Error: Connection terminated due to connection timeout
    at Connection.<anonymous> (/usr/src/app/node_modules/pg/lib/client.js:132:73)
    at Object.onceWrapper (node:events:628:28)
    at Connection.emit (node:events:514:28)
    at Socket.<anonymous> (/usr/src/app/node_modules/pg/lib/connection.js:63:12)
    at Socket.emit (node:events:514:28)
    at TCP.<anonymous> (node:net:337:12)
[Nest] 7  - 10/29/2023, 5:56:59 AM   ERROR [ExceptionsHandler] Connection terminated due to connection timeout
Error: Connection terminated due to connection timeout
    at Connection.<anonymous> (/usr/src/app/node_modules/pg/lib/client.js:132:73)
    at Object.onceWrapper (node:events:628:28)
    at Connection.emit (node:events:514:28)
    at Socket.<anonymous> (/usr/src/app/node_modules/pg/lib/connection.js:63:12)
    at Socket.emit (node:events:514:28)
    at TCP.<anonymous> (node:net:337:12)
[Nest] 7  - 10/29/2023, 5:56:59 AM   ERROR [ExceptionsHandler] Connection terminated due to connection timeout
Error: Connection terminated due to connection timeout
    at Connection.<anonymous> (/usr/src/app/node_modules/pg/lib/client.js:132:73)
    at Object.onceWrapper (node:events:628:28)
    at Connection.emit (node:events:514:28)
    at Socket.<anonymous> (/usr/src/app/node_modules/pg/lib/connection.js:63:12)
    at Socket.emit (node:events:514:28)
    at TCP.<anonymous> (node:net:337:12)
[Nest] 7  - 10/29/2023, 5:57:01 AM   ERROR [ExceptionsHandler] Connection terminated due to connection timeout
Error: Connection terminated due to connection timeout
    at Connection.<anonymous> (/usr/src/app/node_modules/pg/lib/client.js:132:73)
    at Object.onceWrapper (node:events:628:28)
    at Connection.emit (node:events:514:28)
    at Socket.<anonymous> (/usr/src/app/node_modules/pg/lib/connection.js:63:12)
    at Socket.emit (node:events:514:28)
    at TCP.<anonymous> (node:net:337:12)

I have tons of assets (> 120k). Other than that, everything works fine.

The OS that Immich Server is running on

Debian

Version of Immich Server

v1.83.0

Version of Immich Mobile App

v1.82.1

Platform with the issue

• Server
• Web
• Mobile

Your docker-compose.yml content

version: "3.8"
name: immich

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    command: [ "start.sh", "immich" ]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
    env_file:
      - immich.env
    depends_on:
      - redis
      - database
      - typesense
    restart: always

  immich-microservices:
    container_name: immich_microservices
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    # extends:
    #   file: hwaccel.yml
    #   service: hwaccel
    command: [ "start.sh", "microservices" ]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
    env_file:
      - immich.env
    depends_on:
      - redis
      - database
      - typesense
    restart: always

  immich-machine-learning:
    container_name: immich_machine_learning
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    volumes:
      - /mnt/VM/immich/model_cache:/cache
    env_file:
      - immich.env
    restart: always

  immich-web:
    container_name: immich_web
    image: ghcr.io/immich-app/immich-web:${IMMICH_VERSION:-release}
    env_file:
      - immich.env
    restart: always

  typesense:
    container_name: immich_typesense
    image: typesense/typesense:0.24.1@sha256:9bcff2b829f12074426ca044b56160ca9d777a0c488303469143dd9f8259d4dd
    environment:
      - TYPESENSE_API_KEY=${TYPESENSE_API_KEY}
      - TYPESENSE_DATA_DIR=/data
      # remove this to get debug messages
      - GLOG_minloglevel=1
    volumes:
      - /mnt/VM/immich/tsdata:/data
    restart: always

  redis:
    container_name: immich_redis
    image: redis:6.2-alpine@sha256:70a7a5b641117670beae0d80658430853896b5ef269ccf00d1827427e3263fa3
    restart: always

  database:
    container_name: immich_postgres
    image: postgres:14-alpine@sha256:28407a9961e76f2d285dc6991e8e48893503cc3836a4755bbc2d40bcc272a441
    env_file:
      - immich.env
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
    volumes:
      - /mnt/VM/immich/db/postgresql/pgdata:/var/lib/postgresql/data
    restart: always

  immich-proxy:
    container_name: immich_proxy
    image: ghcr.io/immich-app/immich-proxy:${IMMICH_VERSION:-release}
    environment:
      # Make sure these values get passed through from the env file
      - IMMICH_SERVER_URL
      - IMMICH_WEB_URL
    ports:
      - 2283:8080
    depends_on:
      - immich-server
      - immich-web
    restart: always

Your .env content

# This env file goes into both: docker-compose.yml and the various containers.

DB_HOSTNAME=immich_postgres
DB_USERNAME=xxx
DB_PASSWORD=xxx
DB_DATABASE_NAME=immich

TYPESENSE_API_KEY=some-text
REDIS_HOSTNAME=immich_redis

UPLOAD_LOCATION=/mnt/Data/bilder
LOG_LEVEL=simple

JWT_SECRET=xxx

PUBLIC_LOGIN_PAGE_MESSAGE="Willkommen zum Bilder - Gedoens!"

IMMICH_VERSION=v1.83.0

Reproduction steps

1. Run the Metadata Extraction Job
2. Wait for ~20s: Then errors are logged every 1-2 seconds.

Additional information

Let me know if I can provide additional information. And thanks a LOT for immich. :-)

[alextran1502]

Hello, what is your server spec? is Immich hosting on an SSD or HDD?

[florihupf]

I am running the actual images on HDD, but the DB and everything else is on a SSD using ZFS.

Only tricky part might be that I use a jail under Truenas Scale to get the full docker compose experience: https://github.com/Jip-Hop/jailmaker

[alextran1502]

@florihupf Do you think following this guide to install would be helpful? https://immich.app/docs/install/truenas

[florihupf]

Thanks for the pointer, but I prefer a raw docker compose environment due to stability.

I am running immich for a long time with no issues so far.

[florihupf]

Is there a way to find more debug info on what is going on?

[alextran1502]

From my understanding the issue of not able to write to the database is because disk IO has maxed out. What is the job concurrency do you have for Metadata extraction job? Perhaps lowering it would help solve the issue.

The database mapping from your docker compose, is it on an SSD or HDD?

[florihupf]

I had the default concurrency of 5 and the database is on an SSD. Let me try to lower the concurrency.

[florihupf]

Lowering the concurrency seems to help, it is still going, will report when things are done / have failed: fingers crossed :-)

[alextran1502]

Hello, did lower the concurrency help with the issue?

[florihupf]

Hi Alex, yeah, looks like it went through with no issues at concurrency "1" after 11h for about 130k assets. Thanks for the helpful pointer!