[BUG] Immich Stops Working when using CLI

[lonewolfpk]

The bug

I recently discovered an issue that immich stops working almost completely (Functionality is broken).
When Using CLI (uploading), I have just go to the administration tab in immich server. Then suddenly the upload shows error 500 undefined. and my server suddenly went offline. I tried refreshing the page but it didnt work. complete reinstallation of immich is the only solution i have found. I tried this several time to check this can be replicated. You can replicate every time you do this.

The OS that Immich Server is running on

Windows 11 Pro

Version of Immich Server

1.63.2

Version of Immich Mobile App

Not applicable

Platform with the issue

• Server
• Web
• Mobile

Your docker-compose.yml content

Not Applicable.

Your .env content

Not Applicable

Reproduction steps

1.Start CLI upload
2.Go to administration tab
3.Check your Jobs or storage (anything in administration tab)
4.you encounter the crash of 500 undefined error
5.From this point you cannot use immich unless done a complete reset of server.

Additional information

No response

[alextran1502]

Can you please post the logs of the server when the issue happens? How do you install and run Immich?

[alextran1502]

What is the resource do you allocate for Immich?

[lonewolfpk]

Win 11 pro( running inside a VM - sorry to mention before)
Intel i5-10300h - 75% is allocated for VM
16 GB ram (8 GB for VM)
Didn't allocated anything specifically for immich ( maybe using default resources values)

[lonewolfpk]

I found that the microservices (postgres,typesense,redist) stopped when I do these things. The service initially started but stops when uploading and and inside it shows error logs ( I reinstalled so logs got erased some). I installed docker in another drive and it fixed the issue.

[lonewolfpk]

This issue started happening again. This time I have the log reports.
This is report from the cli utility: (like the below it is showing for the remaining assets that need to be uploaded.)

Error asset:  {
  file: 'E:\\admin\\2020\\2020-11\\IMG20201114200256.jpg',
  reason: Error: error request aborted
      at createError (E:\docker\docker apps\immich\immich repo\CLI-main\node_modules\axios\lib\core\createError.js:16:15)
      at IncomingMessage.handlerStreamAborted (E:\docker\docker apps\immich\immich repo\CLI-main\node_modules\axios\lib\adapters\http.js:301:18)
      at IncomingMessage.emit (node:events:511:28)
      at IncomingMessage._destroy (node:_http_incoming:224:10)
      at _destroy (node:internal/streams/destroy:109:10)
      at IncomingMessage.destroy (node:internal/streams/destroy:71:5)
      at Socket.socketCloseListener (node:_http_client:455:11)
      at Socket.emit (node:events:523:35)
      at TCP.<anonymous> (node:net:334:12) {
    config: {
      transitional: [Object],
      adapter: [Function: httpAdapter],
      transformRequest: [Array],
      transformResponse: [Array],
      timeout: 0,
      xsrfCookieName: 'XSRF-TOKEN',
      xsrfHeaderName: 'X-XSRF-TOKEN',
      maxContentLength: Infinity,
      maxBodyLength: Infinity,
      validateStatus: [Function: validateStatus],
      headers: [Object],
      method: 'post',
      maxRedirects: 0,
      url: 'http://192.168.0.161:2283/api/asset/upload',
      data: [FormData]
    },
    code: 'ERR_REQUEST_ABORTED',
    request: ClientRequest {
      _events: [Object: null prototype],
      _eventsCount: 3,
      _maxListeners: undefined,
      outputData: [],
      outputSize: 0,
      writable: true,
      destroyed: true,
      _last: false,
      chunkedEncoding: true,
      shouldKeepAlive: true,
      maxRequestsOnConnectionReached: false,
      _defaultKeepAlive: true,
      useChunkedEncodingByDefault: true,
      sendDate: false,
      _removedConnection: false,
      _removedContLen: false,
      _removedTE: false,
      strictContentLength: false,
      _contentLength: null,
      _hasBody: true,
      _trailer: '',
      finished: true,
      _headerSent: true,
      _closed: true,
      socket: [Socket],
      _header: 'POST /api/asset/upload HTTP/1.1\r\n' +
        'Accept: application/json, text/plain, */*\r\n' +
        'Content-Type: multipart/form-data; boundary=--------------------------009856132569760699902137\r\n' +
        'x-api-key: MSwyznUkClH8WkYst45GxZg2lpREUpTIAMCl01cb80\r\n' +
        'User-Agent: axios/0.26.0\r\n' +
        'Host: 192.168.0.161:2283\r\n' +
        'Connection: keep-alive\r\n' +
        'Transfer-Encoding: chunked\r\n' +
        '\r\n',
      _keepAliveTimeout: 0,
      _onPendingData: [Function: nop],
      agent: [Agent],
      socketPath: undefined,
      method: 'POST',
      maxHeaderSize: undefined,
      insecureHTTPParser: undefined,
      joinDuplicateHeaders: undefined,
      path: '/api/asset/upload',
      _ended: false,
      res: [IncomingMessage],
      aborted: false,
      timeoutCb: [Function: emitRequestTimeout],
      upgradeOrConnect: false,
      parser: null,
      maxHeadersCount: null,
      reusedSocket: true,
      host: '192.168.0.161',
      protocol: 'http:',
      [Symbol(kCapture)]: false,
      [Symbol(kBytesWritten)]: 0,
      [Symbol(kNeedDrain)]: false,
      [Symbol(corked)]: 0,
      [Symbol(kOutHeaders)]: [Object: null prototype],
      [Symbol(errored)]: null,
      [Symbol(kHighWaterMark)]: 16384,
      [Symbol(kRejectNonStandardBodyWrites)]: false,
      [Symbol(kUniqueHeaders)]: null
    },
    response: undefined,
    isAxiosError: true,
    toJSON: [Function: toJSON]
  },
  response: undefined
}

This is the report from the docker desktop server log:
[Nest] 1  - 06/28/2023, 3:35:39 PM   ERROR [ExpressAdapter] Premature close
2023-06-28 08:35:39 Error [ERR_STREAM_PREMATURE_CLOSE]: Premature close
2023-06-28 08:35:39     at new NodeError (node:internal/errors:399:5)
2023-06-28 08:35:39     at ServerResponse.onclose (node:internal/streams/end-of-stream:154:30)
2023-06-28 08:35:39     at ServerResponse.emit (node:events:525:35)
2023-06-28 08:35:39     at emitCloseNT (node:_http_server:984:10)
2023-06-28 08:35:39     at Socket.onServerResponseClose (node:_http_server:277:5)
2023-06-28 08:35:39     at Socket.emit (node:events:525:35)
2023-06-28 08:35:39     at TCP.<anonymous> (node:net:322:12)

This is from immich proxy: ( i did not reset any connection BTW)
2023/06/28 12:45:18 [error] 42#42: *859 recv() failed (104: Connection reset by peer) while reading upstream, client: 172.18.0.1, server: , request: "POST /api/asset/upload HTTP/1.1", upstream: "http://172.18.0.8:3001/asset/upload", host: "192.168.0.161:2283"

[alextran1502]

Is the server stopped working again, or your just cannot upload those files? Can you still access the web?

[lonewolfpk]

I can access the web interface. From that point nothing is uploading.

[lonewolfpk]

Everytime this occurs.
Error asset: {
file: 'D:\admin\2021\2021-05\VID20210508005632.mp4',
reason: Error: Request failed with status code 504
at createError (E:\Docker\Docker Apps\Immich\Immich-App\CLI-main\node_modules\axios\lib\core\createError.js:16:15)
at settle (E:\Docker\Docker Apps\Immich\Immich-App\CLI-main\node_modules\axios\lib\core\settle.js:17:12)
at IncomingMessage.handleStreamEnd (E:\Docker\Docker Apps\Immich\Immich-App\CLI-main\node_modules\axios\lib\adapters\http.js:322:11)
at IncomingMessage.emit (node:events:523:35)
at endReadableNT (node:internal/streams/readable:1367:12)
at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
config: {
transitional: [Object],
adapter: [Function: httpAdapter],
transformRequest: [Array],
transformResponse: [Array],
timeout: 0,
xsrfCookieName: 'XSRF-TOKEN',
xsrfHeaderName: 'X-XSRF-TOKEN',
maxContentLength: Infinity,
maxBodyLength: Infinity,
validateStatus: [Function: validateStatus],
headers: [Object],
method: 'post',
maxRedirects: 0,
url: 'http://192.168.0.90:2283/api/asset/upload',
data: [FormData]
},
request: ClientRequest {
_events: [Object: null prototype],
_eventsCount: 3,
_maxListeners: undefined,
outputData: [],
outputSize: 0,
writable: true,
destroyed: true,
_last: false,
chunkedEncoding: true,
shouldKeepAlive: true,
maxRequestsOnConnectionReached: false,
_defaultKeepAlive: true,
useChunkedEncodingByDefault: true,
sendDate: false,
_removedConnection: false,
_removedContLen: false,
_removedTE: false,
strictContentLength: false,
_contentLength: null,
_hasBody: true,
_trailer: '',
finished: false,
_headerSent: true,
_closed: true,
socket: [Socket],
_header: 'POST /api/asset/upload HTTP/1.1\r\n' +
'Accept: application/json, text/plain, /\r\n' +
'Content-Type: multipart/form-data; boundary=--------------------------540162392401991048968072\r\n' +
'x-api-key: UtJmjR1ZjCnnJyGPp4DXl3umt3CHaOsXXU1QDgbv0\r\n' +
'User-Agent: axios/0.26.0\r\n' +
'Host: 192.168.0.90:2283\r\n' +
'Connection: keep-alive\r\n' +
'Transfer-Encoding: chunked\r\n' +
'\r\n',
_keepAliveTimeout: 0,
_onPendingData: [Function: nop],
agent: [Agent],
socketPath: undefined,
method: 'POST',
maxHeaderSize: undefined,
insecureHTTPParser: undefined,
joinDuplicateHeaders: undefined,
path: '/api/asset/upload',
_ended: true,
res: [IncomingMessage],
aborted: false,
timeoutCb: [Function: emitRequestTimeout],
upgradeOrConnect: false,
parser: null,
maxHeadersCount: null,
reusedSocket: true,
host: '192.168.0.90',
protocol: 'http:',
[Symbol(kCapture)]: false,
[Symbol(kBytesWritten)]: 0,
[Symbol(kNeedDrain)]: true,
[Symbol(corked)]: 0,
[Symbol(kOutHeaders)]: [Object: null prototype],
[Symbol(errored)]: null,
[Symbol(kHighWaterMark)]: 16384,
[Symbol(kRejectNonStandardBodyWrites)]: false,
[Symbol(kUniqueHeaders)]: null
},
response: {
status: 504,
statusText: 'Gateway Time-out',
headers: [Object],
config: [Object],
request: [ClientRequest],
data: '\r\n' +
'<title>504 Gateway Time-out</title>\r\n' +
'\r\n' +
'

504 Gateway Time-out

\r\n' +
'

---

nginx/1.25.0\r\n' +
'\r\n' +
'\r\n'
},
isAxiosError: true,
toJSON: [Function: toJSON]
},
response: '\r\n' +
'<title>504 Gateway Time-out</title>\r\n' +
'\r\n' +
'

504 Gateway Time-out

\r\n' +
'

---

nginx/1.25.0\r\n' +
'\r\n' +
'\r\n'
}

[jrasm91]

Does this also happen when you upload a file through the web interface? Did it always crash on the first file or only when you upload multiple? Can you change the docker compose file t remove the restart: always lines and let the container stay off if it crashes, then verify which one crashed.

[lonewolfpk]

It crashes in the middle. I was uploading 10k+ media.
The microservices crashed

[dcglabs]

I got same issue with using cli upload tool. With docker compose setup.
docker run -it --rm -v "$(pwd):/import" ghcr.io/immich-app/immich-cli:latest upload --recursive /import/ --key 5r86rOgwH0BTHKRnW3RcV2Wf3gUc5iBdzzocEDzw --album dcg_photo --server http://192.168.21.87:8080/api
This upload was for 60k items ~400GB. I had done a couple different batches of 10k without issue. This was last bit of output from the docker cli run. I wonder if doing massive cli upload you should disable all the thumbnail, face, tagging, etc items? This is on a core i3-13100 with 64gb ram. It seemed to get through more than 50% of upload before crashing. Not sure when it did. The album dcg_photo does exist but no items are associated with it.

Error asset:  {
  file: '/import/dcg_cloud/Go Pro/100GOPRO/.@__thumb/defaultGOPR0037.MP4',
  reason: Error: Request failed with status code 500
      at createError (/usr/src/app/node_modules/axios/lib/core/createError.js:16:15)
      at settle (/usr/src/app/node_modules/axios/lib/core/settle.js:17:12)
      at IncomingMessage.handleStreamEnd (/usr/src/app/node_modules/axios/lib/adapters/http.js:322:11)
      at IncomingMessage.emit (node:events:539:35)
      at endReadableNT (node:internal/streams/readable:1345:12)
      at processTicksAndRejections (node:internal/process/task_queues:83:21) {
    config: {
      transitional: [Object],
      adapter: [Function: httpAdapter],
      transformRequest: [Array],
      transformResponse: [Array],
      timeout: 0,
      xsrfCookieName: 'XSRF-TOKEN',
      xsrfHeaderName: 'X-XSRF-TOKEN',
      maxContentLength: Infinity,
      maxBodyLength: Infinity,
      validateStatus: [Function: validateStatus],
      headers: [Object],
      method: 'post',
      maxRedirects: 0,
      url: 'http://192.168.21.87:8080/api/asset/upload',
      data: [FormData]
    },
    request: ClientRequest {
      _events: [Object: null prototype],
      _eventsCount: 3,
      _maxListeners: undefined,
      outputData: [],
      outputSize: 0,
      writable: true,
      destroyed: true,
      _last: true,
      chunkedEncoding: true,
      shouldKeepAlive: false,
      maxRequestsOnConnectionReached: false,
      _defaultKeepAlive: true,
      useChunkedEncodingByDefault: true,
      sendDate: false,
      _removedConnection: false,
      _removedContLen: false,
      _removedTE: false,
      _contentLength: null,
      _hasBody: true,
      _trailer: '',
      finished: true,
      _headerSent: true,
      _closed: true,
      socket: [Socket],
      _header: 'POST /api/asset/upload HTTP/1.1\r\n' +
        'Accept: application/json, text/plain, */*\r\n' +
        'Content-Type: multipart/form-data; boundary=--------------------------203081941416512938797114\r\n' +
        'x-api-key: 5r86rOgwH0BTHKRnW3RcV2Wf3gUc5iBdzzocEDzw\r\n' +
        'User-Agent: axios/0.26.0\r\n' +
        'Host: 192.168.21.87:8080\r\n' +
        'Connection: close\r\n' +
        'Transfer-Encoding: chunked\r\n' +
        '\r\n',
      _keepAliveTimeout: 0,
      _onPendingData: [Function: nop],
      agent: [Agent],
      socketPath: undefined,
      method: 'POST',
      maxHeaderSize: undefined,
      insecureHTTPParser: undefined,
      path: '/api/asset/upload',
      _ended: true,
      res: [IncomingMessage],
      aborted: false,
      timeoutCb: null,
      upgradeOrConnect: false,
      parser: null,
      maxHeadersCount: null,
      reusedSocket: false,
      host: '192.168.21.87',
      protocol: 'http:',
      [Symbol(kCapture)]: false,
      [Symbol(kNeedDrain)]: false,
      [Symbol(corked)]: 0,
      [Symbol(kOutHeaders)]: [Object: null prototype]
    },
    response: {
      status: 500,
      statusText: 'Internal Server Error',
      headers: [Object],
      config: [Object],
      request: [ClientRequest],
      data: [Object]
    },
    isAxiosError: true,
    toJSON: [Function: toJSON]
  },
  response: { statusCode: 500, message: 'Internal server error' }
}
Error asset:  {
  file: '/import/dcg_cloud/Go Pro/100GOPRO/GP050042.MP4',
  reason: Error: Request failed with status code 504
      at createError (/usr/src/app/node_modules/axios/lib/core/createError.js:16:15)
      at settle (/usr/src/app/node_modules/axios/lib/core/settle.js:17:12)
      at IncomingMessage.handleStreamEnd (/usr/src/app/node_modules/axios/lib/adapters/http.js:322:11)
      at IncomingMessage.emit (node:events:539:35)
      at endReadableNT (node:internal/streams/readable:1345:12)
      at processTicksAndRejections (node:internal/process/task_queues:83:21) {
    config: {
      transitional: [Object],
      adapter: [Function: httpAdapter],
      transformRequest: [Array],
      transformResponse: [Array],
      timeout: 0,
      xsrfCookieName: 'XSRF-TOKEN',
      xsrfHeaderName: 'X-XSRF-TOKEN',
      maxContentLength: Infinity,
      maxBodyLength: Infinity,
      validateStatus: [Function: validateStatus],
      headers: [Object],
      method: 'post',
      maxRedirects: 0,
      url: 'http://192.168.21.87:8080/api/asset/upload',
      data: [FormData]
    },
    request: ClientRequest {
      _events: [Object: null prototype],
      _eventsCount: 3,
      _maxListeners: undefined,
      outputData: [],
      outputSize: 0,
      writable: true,
      destroyed: true,
      _last: true,
      chunkedEncoding: true,
      shouldKeepAlive: false,
      maxRequestsOnConnectionReached: false,
      _defaultKeepAlive: true,
      useChunkedEncodingByDefault: true,
      sendDate: false,
      _removedConnection: false,
      _removedContLen: false,
      _removedTE: false,
      _contentLength: null,
      _hasBody: true,
      _trailer: '',
      finished: false,
      _headerSent: true,
      _closed: true,
      socket: [Socket],
      _header: 'POST /api/asset/upload HTTP/1.1\r\n' +
        'Accept: application/json, text/plain, */*\r\n' +
        'Content-Type: multipart/form-data; boundary=--------------------------404389337864740434891698\r\n' +
        'x-api-key: 5r86rOgwH0BTHKRnW3RcV2Wf3gUc5iBdzzocEDzw\r\n' +
        'User-Agent: axios/0.26.0\r\n' +
        'Host: 192.168.21.87:8080\r\n' +
        'Connection: close\r\n' +
        'Transfer-Encoding: chunked\r\n' +
        '\r\n',
      _keepAliveTimeout: 0,
      _onPendingData: [Function: nop],
      agent: [Agent],
      socketPath: undefined,
      method: 'POST',
      maxHeaderSize: undefined,
      insecureHTTPParser: undefined,
      path: '/api/asset/upload',
      _ended: true,
      res: [IncomingMessage],
      aborted: false,
      timeoutCb: null,
      upgradeOrConnect: false,
      parser: null,
      maxHeadersCount: null,
      reusedSocket: false,
      host: '192.168.21.87',
      protocol: 'http:',
      [Symbol(kCapture)]: false,
      [Symbol(kNeedDrain)]: true,
      [Symbol(corked)]: 0,
      [Symbol(kOutHeaders)]: [Object: null prototype]
    },
    response: {
      status: 504,
      statusText: 'Gateway Time-out',
      headers: [Object],
      config: [Object],
      request: [ClientRequest],
      data: '<html>\r\n' +
        '<head><title>504 Gateway Time-out</title></head>\r\n' +
        '<body>\r\n' +
        '<center><h1>504 Gateway Time-out</h1></center>\r\n' +
        '<hr><center>nginx/1.25.0</center>\r\n' +
        '</body>\r\n' +
        '</html>\r\n'
    },
    isAxiosError: true,
    toJSON: [Function: toJSON]
  },
  response: '<html>\r\n' +
    '<head><title>504 Gateway Time-out</title></head>\r\n' +
    '<body>\r\n' +
    '<center><h1>504 Gateway Time-out</h1></center>\r\n' +
    '<hr><center>nginx/1.25.0</center>\r\n' +
    '</body>\r\n' +
    '</html>\r\n'
}

[isaaclyman]

Happens for me as well (docker compose setup, running on a Windows desktop). I've found I can avoid the issue if I pause all jobs on the web interface and upload only a couple hundred assets at a time. I'm guessing the various jobs use up all the available connections on my network card, causing everything else to time out and spiral into an infinite loop.

When this happens, Docker is unable to kill the immich_microservices and/or immich_server containers. I have to restart the machine to bring the server back up.

In the error state, I get a 500 error from the webpage and this log repeats constantly in the immich_server container logs:

[Nest] 7  - 07/31/2023, 7:22:35 PM   ERROR [ExceptionsHandler] Connection terminated due to connection timeout
Error: Connection terminated due to connection timeout
    at Connection.<anonymous> (/usr/src/app/node_modules/pg/lib/client.js:132:73)
    at Object.onceWrapper (node:events:627:28)
    at Connection.emit (node:events:513:28)
    at Socket.<anonymous> (/usr/src/app/node_modules/pg/lib/connection.js:62:12)
    at Socket.emit (node:events:513:28)
    at TCP.<anonymous> (node:net:322:12)

I also typically see errors indicating that one of the containers can't resolve download.geonames.org. I wonder if the file it's referencing could be cached?

[isaaclyman]

Update: Just saw this happen while I was not uploading any assets (none of the CLI/web upload/mobile app were running). The only jobs running were:

• Storage template migration (concurrency: 5)
• Tag objects (concurrency: 2)
• Generate thumbnails (concurrency: 5)

[marank]

I'M experiencing the same. I wanted to upload ~25k images and videos, and after a while, my VM gets unreliable. On the same VM, I have Uptime-Kuma running, and I get error messages from it like the following: "Knex: Timeout acquiring a connection. The pool is probably full. Are you missing a .transacting(trx) call?" This happend never before.

The VM is running on Proxmox on an Intel(R) Xeon(R) CPU E5-2680 v3 with 8 cores and 32GB RAM assigned to it.

Here are some relevant logs from the docker stack:

immich_microservices     | Request #1692557416139: Request to Node 0 failed due to "ECONNABORTED timeout of 10000ms exceeded"
immich_microservices     | Request #1692557416139: Sleeping for 4s and then retrying request...
immich_microservices     | �[32m[Nest] 7  - �[39m08/20/2023, 6:50:28 PM �[32m    LOG�[39m �[38;5;3m[MediaService] �[39m�[32mEncoding success 2458c407-cbcb-4576-b4b6-228c94dae604�[39m
immich_microservices     | Request #1692557427899: Request to Node 0 failed due to "ECONNABORTED timeout of 10000ms exceeded"
immich_microservices     | Request #1692557427899: Sleeping for 4s and then retrying request...
immich_microservices     | Request #1692557427924: Request to Node 0 failed due to "ECONNABORTED timeout of 10000ms exceeded"
immich_microservices     | Request #1692557427924: Sleeping for 4s and then retrying request...
immich_microservices     | Request #1692557416139: Request to Node 0 failed due to "ECONNABORTED timeout of 10000ms exceeded"
immich_microservices     | Request #1692557416139: Sleeping for 4s and then retrying request...
immich_microservices     | /usr/src/app/dist/microservices/app.service.js:74
immich_microservices     |                 throw error;
immich_microservices     |                 ^
immich_microservices     | 
immich_microservices     | 
immich_microservices     | Error: timeout exceeded when trying to connect
immich_microservices     |     at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:200:27)
immich_microservices     |     at listOnTimeout (node:internal/timers:569:17)
immich_microservices     |     at process.processTimers (node:internal/timers:512:7)
immich_microservices     | 
immich_microservices     | 
immich_microservices     | Node.js v18.17.0
immich_microservices     | �[32m[Nest] 7  - �[39m08/20/2023, 6:58:03 PM �[32m    LOG�[39m �[38;5;3m[NestFactory] �[39m�[32mStarting Nest application...�[39m
immich_microservices     | �[32m[Nest] 7  - �[39m08/20/2023, 6:58:03 PM �[32m    LOG�[39m �[38;5;3m[InstanceLoader] �[39m�[32mTypeOrmModule dependencies initialized�[39m�[38;5;3m +98ms�[39m
immich_microservices     | �[32m[Nest] 7  - �[39m08/20/2023, 6:58:03 PM �[32m    LOG�[39m �[38;5;3m[InstanceLoader] �[39m�[32mBullModule dependencies initialized�[39m�[38;5;3m +0ms�[39m
immich_microservices     | �[32m[Nest] 7  - �[39m08/20/2023, 6:58:03 PM �[32m    LOG�[39m �[38;5;3m[InstanceLoader] �[39m�[32mConfigHostModule dependencies initialized�[39m�[38;5;3m +1ms�[39m
immich_microservices     | �[32m[Nest] 7  - �[39m08/20/2023, 6:58:03 PM �[32m    LOG�[39m �[38;5;3m[InstanceLoader] �[39m�[32mDiscoveryModule dependencies initialized�[39m�[38;5;3m +1ms�[39m
immich_microservices     | �[32m[Nest] 7  - �[39m08/20/2023, 6:58:03 PM �[32m    LOG�[39m �[38;5;3m[InstanceLoader] �[39m�[32mConfigModule dependencies initialized�[39m�[38;5;3m +11ms�[39m
immich_microservices     | �[32m[Nest] 7  - �[39m08/20/2023, 6:58:03 PM �[32m    LOG�[39m �[38;5;3m[InstanceLoader] �[39m�[32mBullModule dependencies initialized�[39m�[38;5;3m +0ms�[39m
immich_microservices     | �[32m[Nest] 7  - �[39m08/20/2023, 6:58:03 PM �[32m    LOG�[39m �[38;5;3m[InstanceLoader] �[39m�[32mBullModule dependencies initialized�[39m�[38;5;3m +0ms�[39m
immich_microservices     | �[32m[Nest] 7  - �[39m08/20/2023, 6:58:06 PM �[32m    LOG�[39m �[38;5;3m[InstanceLoader] �[39m�[32mTypeOrmCoreModule dependencies initialized�[39m�[38;5;3m +3481ms�[39m
immich_microservices     | �[32m[Nest] 7  - �[39m08/20/2023, 6:58:06 PM �[32m    LOG�[39m �[38;5;3m[InstanceLoader] �[39m�[32mTypeOrmModule dependencies initialized�[39m�[38;5;3m +0ms�[39m
immich_microservices     | �[32m[Nest] 7  - �[39m08/20/2023, 6:58:06 PM �[32m    LOG�[39m �[38;5;3m[InstanceLoader] �[39m�[32mTypeOrmModule dependencies initialized�[39m�[38;5;3m +0ms�[39m
immich_microservices     | �[32m[Nest] 7  - �[39m08/20/2023, 6:58:06 PM �[32m    LOG�[39m �[38;5;3m[InstanceLoader] �[39m�[32mInfraModule dependencies initialized�[39m�[38;5;3m +6ms�[39m
immich_microservices     | �[32m[Nest] 7  - �[39m08/20/2023, 6:58:06 PM �[32m    LOG�[39m �[38;5;3m[InstanceLoader] �[39m�[32mDomainModule dependencies initialized�[39m�[38;5;3m +5ms�[39m
immich_microservices     | �[32m[Nest] 7  - �[39m08/20/2023, 6:58:06 PM �[32m    LOG�[39m �[38;5;3m[InstanceLoader] �[39m�[32mMicroservicesModule dependencies initialized�[39m�[38;5;3m +0ms�[39m
immich_microservices     | �[32m[Nest] 7  - �[39m08/20/2023, 6:58:06 PM �[32m    LOG�[39m �[38;5;3m[MetadataExtractionProcessor] �[39m�[32mReverse geocoding is enabled�[39m
immich_microservices     | �[32m[Nest] 7  - �[39m08/20/2023, 6:58:06 PM �[32m    LOG�[39m �[38;5;3m[MetadataExtractionProcessor] �[39m�[32mInitializing Reverse Geocoding�[39m
immich_microservices     | �[32m[Nest] 7  - �[39m08/20/2023, 6:58:16 PM �[32m    LOG�[39m �[38;5;3m[MediaService] �[39m�[32mStart encoding video 8793748a-70e7-49c0-9300-a2e71365a1ab {"inputOptions":[],"outputOptions":["-vcodec h264","-acodec aac","-movflags faststart","-fps_mode passthrough","-v verbose","-vf format=yuv420p","-preset ultrafast","-crf 23"],"twoPass":false}�[39m
immich_microservices     | �[32m[Nest] 7  - �[39m08/20/2023, 6:58:16 PM �[32m    LOG�[39m �[38;5;3m[MediaService] �[39m�[32mSuccessfully generated image thumbnail bf639d57-484a-4839-aa34-df957fe2e521�[39m
immich_microservices     | �[32m[Nest] 7  - �[39m08/20/2023, 6:58:22 PM �[32m    LOG�[39m �[38;5;3m[MediaService] �[39m�[32mEncoding success 8793748a-70e7-49c0-9300-a2e71365a1ab�[39m
immich_microservices     | �[32m[Nest] 7  - �[39m08/20/2023, 6:58:28 PM �[32m    LOG�[39m �[38;5;3m[MetadataExtractionProcessor] �[39m�[32mReverse Geocoding Initialized�[39m
immich_microservices     | �[32m[Nest] 7  - �[39m08/20/2023, 6:58:29 PM �[32m    LOG�[39m �[38;5;3m[NestApplication] �[39m�[32mNest application successfully started�[39m�[38;5;3m +1725ms�[39m
immich_microservices     | �[32m[Nest] 7  - �[39m08/20/2023, 6:58:29 PM �[32m    LOG�[39m �[38;5;3m[ImmichMicroservice] �[39m�[32mImmich Microservices is listening on http://[::1]:3002 [v1.74.0] [PRODUCTION] �[39m
immich_microservices     | �[32m[Nest] 7  - �[39m08/20/2023, 6:58:31 PM �[32m    LOG�[39m �[38;5;3m[MediaService] �[39m�[32mStart encoding video ad550acd-b469-44ef-952b-c7024ca5f7b9 {"inputOptions":[],"outputOptions":["-vcodec h264","-acodec aac","-movflags faststart","-fps_mode passthrough","-v verbose","-vf format=yuv420p","-preset ultrafast","-crf 23"],"twoPass":false}�[39m
immich_microservices     | �[32m[Nest] 7  - �[39m08/20/2023, 6:58:35 PM �[32m    LOG�[39m �[38;5;3m[MediaService] �[39m�[32mSuccessfully generated image thumbnail bbefc993-8069-4888-991b-7cfcde71b8b1�[39m
immich_microservices     | �[32m[Nest] 7  - �[39m08/20/2023, 6:58:38 PM �[32m    LOG�[39m �[38;5;3m[MediaService] �[39m�[32mEncoding success ad550acd-b469-44ef-952b-c7024ca5f7b9�[39m
immich_microservices     | Request #1692557914758: Request to Node 0 failed due to "ECONNABORTED timeout of 10000ms exceeded"
immich_microservices     | Request #1692557914758: Sleeping for 4s and then retrying request...
immich_microservices     | �[32m[Nest] 7  - �[39m08/20/2023, 6:58:45 PM �[32m    LOG�[39m �[38;5;3m[MediaService] �[39m�[32mStart encoding video ac256d69-7f52-4145-a904-be5cdd44f379 {"inputOptions":[],"outputOptions":["-vcodec h264","-acodec aac","-movflags faststart","-fps_mode passthrough","-v verbose","-vf format=yuv420p","-preset ultrafast","-crf 23"],"twoPass":false}�[39m
immich_microservices     | �[32m[Nest] 7  - �[39m08/20/2023, 6:58:47 PM �[32m    LOG�[39m �[38;5;3m[MediaService] �[39m�[32mSuccessfully generated image thumbnail e7f9c7c2-3106-455f-aba4-abb5f93d8fcc�[39m
immich_microservices     | �[32m[Nest] 7  - �[39m08/20/2023, 6:58:47 PM �[32m    LOG�[39m �[38;5;3m[MediaService] �[39m�[32mSuccessfully generated image thumbnail 6fb80b9a-cdb4-4fd5-9360-0c2a4d9a46e8�[39m
immich_microservices     | �[32m[Nest] 7  - �[39m08/20/2023, 6:58:47 PM �[32m    LOG�[39m �[38;5;3m[MediaService] �[39m�[32mSuccessfully generated image thumbnail 060b1572-124b-4244-99e7-f3dfd1056381�[39m
immich_microservices     | �[32m[Nest] 7  - �[39m08/20/2023, 6:58:47 PM �[32m    LOG�[39m �[38;5;3m[MediaService] �[39m�[32mSuccessfully generated image thumbnail d7566ee1-2692-4c77-b874-b382ddf5224f�[39m
immich_microservices     | �[32m[Nest] 7  - �[39m08/20/2023, 6:58:48 PM �[32m    LOG�[39m �[38;5;3m[MediaService] �[39m�[32mSuccessfully generated image thumbnail eb422585-0ee4-4288-b36e-27258983baf9�[39m
immich_microservices     | �[32m[Nest] 7  - �[39m08/20/2023, 6:58:48 PM �[32m    LOG�[39m �[38;5;3m[MediaService] �[39m�[32mSuccessfully generated image thumbnail f3896b02-c7cd-4418-bdde-b5d079e7fe66�[39m
immich_microservices     | �[32m[Nest] 7  - �[39m08/20/2023, 6:58:48 PM �[32m    LOG�[39m �[38;5;3m[MediaService] �[39m�[32mSuccessfully generated image thumbnail 13456103-fc30-45cc-a730-daf7533b45f8�[39m
immich_microservices     | Request #1692557914758: Request to Node 0 failed due to "ECONNABORTED timeout of 10000ms exceeded"
immich_microservices     | Request #1692557914758: Sleeping for 4s and then retrying request...
immich_microservices     | Request #1692557931136: Request to Node 0 failed due to "ECONNABORTED timeout of 10000ms exceeded"
immich_microservices     | Request #1692557931136: Sleeping for 4s and then retrying request...
immich_microservices     | �[32m[Nest] 7  - �[39m08/20/2023, 6:59:03 PM �[32m    LOG�[39m �[38;5;3m[MediaService] �[39m�[32mEncoding success ac256d69-7f52-4145-a904-be5cdd44f379�[39m
immich_microservices     | Request #1692557937273: Request to Node 0 failed due to "ECONNABORTED timeout of 10000ms exceeded"
immich_microservices     | Request #1692557937273: Sleeping for 4s and then retrying request...
immich_microservices     | Request #1692557914758: Request to Node 0 failed due to "ECONNABORTED timeout of 10000ms exceeded"
immich_microservices     | Request #1692557914758: Sleeping for 4s and then retrying request...
immich_microservices     | �[32m[Nest] 7  - �[39m08/20/2023, 6:59:13 PM �[32m    LOG�[39m �[38;5;3m[MediaService] �[39m�[32mSuccessfully generated image thumbnail f98feafe-1538-4d20-a406-8f934a5e3a92�[39m
immich_microservices     | Request #1692557944752: Request to Node 0 failed due to "ECONNABORTED timeout of 10000ms exceeded"
immich_microservices     | Request #1692557944752: Sleeping for 4s and then retrying request...
immich_microservices     | Request #1692557944762: Request to Node 0 failed due to "ECONNABORTED timeout of 10000ms exceeded"
immich_microservices     | Request #1692557944762: Sleeping for 4s and then retrying request...
immich_microservices     | Request #1692557931136: Request to Node 0 failed due to "ECONNABORTED timeout of 10000ms exceeded"
immich_microservices     | Request #1692557931136: Sleeping for 4s and then retrying request...
immich_microservices     | �[32m[Nest] 7  - �[39m08/20/2023, 6:59:16 PM �[32m    LOG�[39m �[38;5;3m[MediaService] �[39m�[32mStart encoding video 2458c407-cbcb-4576-b4b6-228c94dae604 {"inputOptions":[],"outputOptions":["-vcodec h264","-acodec aac","-movflags faststart","-fps_mode passthrough","-v verbose","-vf format=yuv420p","-preset ultrafast","-crf 23"],"twoPass":false}�[39m
immich_microservices     | Request #1692557937273: Request to Node 0 failed due to "ECONNABORTED timeout of 10000ms exceeded"
immich_microservices     | Request #1692557937273: Sleeping for 4s and then retrying request...
immich_microservices     | Request #1692557953219: Request to Node 0 failed due to "ECONNABORTED timeout of 10000ms exceeded"
immich_microservices     | Request #1692557953219: Sleeping for 4s and then retrying request...
immich_microservices     | Request #1692557953224: Request to Node 0 failed due to "ECONNABORTED timeout of 10000ms exceeded"
immich_microservices     | Request #1692557953224: Sleeping for 4s and then retrying request...
immich_microservices     | Request #1692557953230: Request to Node 0 failed due to "ECONNABORTED timeout of 10000ms exceeded"
immich_microservices     | Request #1692557953230: Sleeping for 4s and then retrying request...
immich_microservices     | �[32m[Nest] 7  - �[39m08/20/2023, 6:59:23 PM �[32m    LOG�[39m �[38;5;3m[MediaService] �[39m�[32mEncoding success 2458c407-cbcb-4576-b4b6-228c94dae604�[39m
immich_microservices     | Request #1692557956315: Request to Node 0 failed due to "ECONNABORTED timeout of 10000ms exceeded"
immich_microservices     | Request #1692557956315: Sleeping for 4s and then retrying request...
immich_microservices     | �[32m[Nest] 7  - �[39m08/20/2023, 6:59:27 PM �[32m    LOG�[39m �[38;5;3m[MediaService] �[39m�[32mStart encoding video ff2e3ca5-1975-480a-a9c7-cba4a73a0302 {"inputOptions":[],"outputOptions":["-vcodec h264","-acodec aac","-movflags faststart","-fps_mode passthrough","-v verbose","-vf format=yuv420p","-preset ultrafast","-crf 23"],"twoPass":false}�[39m
immich_microservices     | Request #1692557944752: Request to Node 0 failed due to "ECONNABORTED timeout of 10000ms exceeded"
immich_microservices     | Request #1692557944752: Sleeping for 4s and then retrying request...
immich_microservices     | Request #1692557944762: Request to Node 0 failed due to "ECONNABORTED timeout of 10000ms exceeded"
immich_microservices     | Request #1692557944762: Sleeping for 4s and then retrying request...
immich_microservices     | �[32m[Nest] 7  - �[39m08/20/2023, 6:59:29 PM �[32m    LOG�[39m �[38;5;3m[MediaService] �[39m�[32mSuccessfully generated image thumbnail f7b46139-fece-4f15-99aa-e3457af66391�[39m
immich_microservices     | Request #1692557931136: Request to Node 0 failed due to "ECONNABORTED timeout of 10000ms exceeded"
immich_microservices     | Request #1692557931136: Sleeping for 4s and then retrying request...
immich_microservices     | �[32m[Nest] 7  - �[39m08/20/2023, 6:59:30 PM �[32m    LOG�[39m �[38;5;3m[MediaService] �[39m�[32mSuccessfully generated image thumbnail f4f80143-e815-4e20-93f5-8b1bd52912b2�[39m
immich_microservices     | �[32m[Nest] 7  - �[39m08/20/2023, 6:59:30 PM �[32m    LOG�[39m �[38;5;3m[MediaService] �[39m�[32mSuccessfully generated image thumbnail 013254be-6d54-48f3-b842-887808c0b0c6�[39m
immich_microservices     | �[32m[Nest] 7  - �[39m08/20/2023, 6:59:30 PM �[32m    LOG�[39m �[38;5;3m[MediaService] �[39m�[32mSuccessfully generated image thumbnail f90260fe-c89d-4e18-ac30-fd1bfac86b18�[39m
immich_microservices     | �[32m[Nest] 7  - �[39m08/20/2023, 6:59:30 PM �[32m    LOG�[39m �[38;5;3m[MediaService] �[39m�[32mSuccessfully generated image thumbnail dbe3d55b-18ff-4145-bc13-44050c74802a�[39m
immich_microservices     | �[32m[Nest] 7  - �[39m08/20/2023, 6:59:30 PM �[32m    LOG�[39m �[38;5;3m[MediaService] �[39m�[32mSuccessfully generated image thumbnail 09da5613-aaa3-4b75-b16c-821d041ac3c8�[39m
immich_microservices     | �[32m[Nest] 7  - �[39m08/20/2023, 6:59:31 PM �[32m    LOG�[39m �[38;5;3m[MediaService] �[39m�[32mSuccessfully generated image thumbnail c22ce114-2f7a-4253-aa7f-57b789dd0a0b�[39m
immich_microservices     | �[32m[Nest] 7  - �[39m08/20/2023, 6:59:31 PM �[32m    LOG�[39m �[38;5;3m[MediaService] �[39m�[32mSuccessfully generated image thumbnail 912d44fe-c282-4116-975c-417e978e55dd�[39m
immich_microservices     | �[32m[Nest] 7  - �[39m08/20/2023, 6:59:31 PM �[32m    LOG�[39m �[38;5;3m[MediaService] �[39m�[32mSuccessfully generated image thumbnail 01070ba6-bb58-4bce-9d00-87694b2c0e14�[39m
immich_microservices     | �[32m[Nest] 7  - �[39m08/20/2023, 6:59:31 PM �[32m    LOG�[39m �[38;5;3m[MediaService] �[39m�[32mSuccessfully generated image thumbnail 6a9a5d7a-bedb-4e0a-94fb-ab889666921a�[39m
immich_microservices     | �[32m[Nest] 7  - �[39m08/20/2023, 6:59:32 PM �[32m    LOG�[39m �[38;5;3m[MediaService] �[39m�[32mSuccessfully generated image thumbnail f85ad065-89d3-4424-b5c2-3bee18e79dd7�[39m
immich_microservices     | �[32m[Nest] 7  - �[39m08/20/2023, 6:59:32 PM �[32m    LOG�[39m �[38;5;3m[MediaService] �[39m�[32mSuccessfully generated image thumbnail ff7ab7d8-9f0a-409b-8cbe-b09eee179aa6�[39m
immich_microservices     | �[32m[Nest] 7  - �[39m08/20/2023, 6:59:32 PM �[32m    LOG�[39m �[38;5;3m[MediaService] �[39m�[32mSuccessfully generated image thumbnail 9810deed-8831-451d-ac0e-639c79263b8f�[39m
immich_microservices     | �[32m[Nest] 7  - �[39m08/20/2023, 6:59:40 PM �[32m    LOG�[39m �[38;5;3m[MediaService] �[39m�[32mEncoding success ff2e3ca5-1975-480a-a9c7-cba4a73a0302�[39m
immich_microservices     | �[32m[Nest] 7  - �[39m08/20/2023, 6:59:41 PM �[32m    LOG�[39m �[38;5;3m[MediaService] �[39m�[32mSuccessfully generated image thumbnail 3fb27823-b5cf-4c01-aea9-e36517e5b1d5�[39m
immich_microservices     | Request #1692557944762: Request to Node 0 failed due to "ECONNABORTED timeout of 10000ms exceeded"
immich_microservices     | Request #1692557944762: Sleeping for 4s and then retrying request...
immich_microservices     | Request #1692557931136: Request to Node 0 failed due to "ECONNABORTED timeout of 10000ms exceeded"
immich_microservices     | Request #1692557931136: Sleeping for 4s and then retrying request...
immich_microservices     | Request #1692557976148: Request to Node 0 failed due to "ECONNABORTED timeout of 10000ms exceeded"
immich_microservices     | Request #1692557976148: Sleeping for 4s and then retrying request...
immich_microservices     | Request #1692557981149: Request to Node 0 failed due to "ECONNABORTED timeout of 10000ms exceeded"
immich_microservices     | Request #1692557981149: Sleeping for 4s and then retrying request...
immich_microservices     | /usr/src/app/dist/microservices/app.service.js:74
immich_microservices     |                 throw error;
immich_microservices     |                 ^
immich_microservices     | 
immich_microservices     | 
immich_microservices     | Error: timeout exceeded when trying to connect
immich_microservices     |     at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:200:27)
immich_microservices     |     at listOnTimeout (node:internal/timers:569:17)
immich_microservices     |     at process.processTimers (node:internal/timers:512:7)
immich_microservices     | 
immich_microservices     | 
immich_microservices     | Node.js v18.17.0
immich_microservices     | �[32m[Nest] 7  - �[39m08/20/2023, 7:03:43 PM �[32m    LOG�[39m �[38;5;3m[NestFactory] �[39m�[32mStarting Nest application...�[39m
immich_microservices     | �[32m[Nest] 7  - �[39m08/20/2023, 7:03:43 PM �[32m    LOG�[39m �[38;5;3m[InstanceLoader] �[39m�[32mTypeOrmModule dependencies initialized�[39m�[38;5;3m +41ms�[39m
immich_microservices     | �[32m[Nest] 7  - �[39m08/20/2023, 7:03:43 PM �[32m    LOG�[39m �[38;5;3m[InstanceLoader] �[39m�[32mBullModule dependencies initialized�[39m�[38;5;3m +0ms�[39m
immich_microservices     | �[32m[Nest] 7  - �[39m08/20/2023, 7:03:43 PM �[32m    LOG�[39m �[38;5;3m[InstanceLoader] �[39m�[32mConfigHostModule dependencies initialized�[39m�[38;5;3m +1ms�[39m
immich_microservices     | �[32m[Nest] 7  - �[39m08/20/2023, 7:03:43 PM �[32m    LOG�[39m �[38;5;3m[InstanceLoader] �[39m�[32mDiscoveryModule dependencies initialized�[39m�[38;5;3m +0ms�[39m
immich_microservices     | �[32m[Nest] 7  - �[39m08/20/2023, 7:03:43 PM �[32m    LOG�[39m �[38;5;3m[InstanceLoader] �[39m�[32mConfigModule dependencies initialized�[39m�[38;5;3m +12ms�[39m
immich_microservices     | �[32m[Nest] 7  - �[39m08/20/2023, 7:03:43 PM �[32m    LOG�[39m �[38;5;3m[InstanceLoader] �[39m�[32mBullModule dependencies initialized�[39m�[38;5;3m +0ms�[39m
immich_microservices     | �[32m[Nest] 7  - �[39m08/20/2023, 7:03:43 PM �[32m    LOG�[39m �[38;5;3m[InstanceLoader] �[39m�[32mBullModule dependencies initialized�[39m�[38;5;3m +0ms�[39m
immich_microservices     | �[32m[Nest] 7  - �[39m08/20/2023, 7:03:43 PM �[32m    LOG�[39m �[38;5;3m[InstanceLoader] �[39m�[32mTypeOrmCoreModule dependencies initialized�[39m�[38;5;3m +194ms�[39m
immich_microservices     | �[32m[Nest] 7  - �[39m08/20/2023, 7:03:43 PM �[32m    LOG�[39m �[38;5;3m[InstanceLoader] �[39m�[32mTypeOrmModule dependencies initialized�[39m�[38;5;3m +0ms�[39m
immich_microservices     | �[32m[Nest] 7  - �[39m08/20/2023, 7:03:43 PM �[32m    LOG�[39m �[38;5;3m[InstanceLoader] �[39m�[32mTypeOrmModule dependencies initialized�[39m�[38;5;3m +0ms�[39m
immich_microservices     | �[32m[Nest] 7  - �[39m08/20/2023, 7:03:43 PM �[32m    LOG�[39m �[38;5;3m[InstanceLoader] �[39m�[32mInfraModule dependencies initialized�[39m�[38;5;3m +5ms�[39m
immich_microservices     | �[32m[Nest] 7  - �[39m08/20/2023, 7:03:43 PM �[32m    LOG�[39m �[38;5;3m[InstanceLoader] �[39m�[32mDomainModule dependencies initialized�[39m�[38;5;3m +6ms�[39m
immich_microservices     | �[32m[Nest] 7  - �[39m08/20/2023, 7:03:43 PM �[32m    LOG�[39m �[38;5;3m[InstanceLoader] �[39m�[32mMicroservicesModule dependencies initialized�[39m�[38;5;3m +0ms�[39m
immich_microservices     | �[32m[Nest] 7  - �[39m08/20/2023, 7:03:43 PM �[32m    LOG�[39m �[38;5;3m[MetadataExtractionProcessor] �[39m�[32mReverse geocoding is enabled�[39m
immich_microservices     | �[32m[Nest] 7  - �[39m08/20/2023, 7:03:43 PM �[32m    LOG�[39m �[38;5;3m[MetadataExtractionProcessor] �[39m�[32mInitializing Reverse Geocoding�[39m
immich_microservices     | �[32m[Nest] 7  - �[39m08/20/2023, 7:03:51 PM �[32m    LOG�[39m �[38;5;3m[MediaService] �[39m�[32mStart encoding video 1ed80f64-ba9a-4231-9a93-35e856d062e7 {"inputOptions":[],"outputOptions":["-vcodec h264","-acodec aac","-movflags faststart","-fps_mode passthrough","-v verbose","-vf format=yuv420p","-preset ultrafast","-crf 23"],"twoPass":false}�[39m
immich_microservices     | �[32m[Nest] 7  - �[39m08/20/2023, 7:03:52 PM �[32m    LOG�[39m �[38;5;3m[MetadataExtractionProcessor] �[39m�[32mReverse Geocoding Initialized�[39m
immich_microservices     | �[32m[Nest] 7  - �[39m08/20/2023, 7:03:52 PM �[32m    LOG�[39m �[38;5;3m[NestApplication] �[39m�[32mNest application successfully started�[39m�[38;5;3m +78ms�[39m
immich_microservices     | �[32m[Nest] 7  - �[39m08/20/2023, 7:03:52 PM �[32m    LOG�[39m �[38;5;3m[ImmichMicroservice] �[39m�[32mImmich Microservices is listening on http://[::1]:3002 [v1.74.0] [PRODUCTION] �[39m

Logs from the immich cli:

Error asset:  {
  file: '/mnt/media/photos/Jahre/2019/Italien 2019/20190906_204022.mp4',
  reason: Error: Request failed with status code 504
      at createError (/usr/src/app/node_modules/axios/lib/core/createError.js:16:15)
      at settle (/usr/src/app/node_modules/axios/lib/core/settle.js:17:12)
      at IncomingMessage.handleStreamEnd (/usr/src/app/node_modules/axios/lib/adapters/http.js:322:11)
      at IncomingMessage.emit (node:events:526:35)
      at endReadableNT (node:internal/streams/readable:1359:12)
      at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
    config: {
      transitional: [Object],
      adapter: [Function: httpAdapter],
      transformRequest: [Array],
      transformResponse: [Array],
      timeout: 0,
      xsrfCookieName: 'XSRF-TOKEN',
      xsrfHeaderName: 'X-XSRF-TOKEN',
      maxContentLength: Infinity,
      maxBodyLength: Infinity,
      validateStatus: [Function: validateStatus],
      headers: [Object],
      method: 'post',
      maxRedirects: 0,
      url: 'http://10.0.0.4:2283/api/asset/upload',
      data: [FormData]
    },
    request: ClientRequest {
      _events: [Object: null prototype],
      _eventsCount: 3,
      _maxListeners: undefined,
      outputData: [],
      outputSize: 0,
      writable: true,
      destroyed: true,
      _last: true,
      chunkedEncoding: true,
      shouldKeepAlive: false,
      maxRequestsOnConnectionReached: false,
      _defaultKeepAlive: true,
      useChunkedEncodingByDefault: true,
      sendDate: false,
      _removedConnection: false,
      _removedContLen: false,
      _removedTE: false,
      strictContentLength: false,
      _contentLength: null,
      _hasBody: true,
      _trailer: '',
      finished: false,
      _headerSent: true,
      _closed: true,
      socket: [Socket],
      _header: 'POST /api/asset/upload HTTP/1.1\r\n' +
        'Accept: application/json, text/plain, */*\r\n' +
        'Content-Type: multipart/form-data; boundary=--------------------------736417810135127844557825\r\n' +
        'x-api-key: <redacted>\r\n' +
        'User-Agent: axios/0.26.1\r\n' +
        'Host: 10.0.0.4:2283\r\n' +
        'Connection: close\r\n' +
        'Transfer-Encoding: chunked\r\n' +
        '\r\n',
      _keepAliveTimeout: 0,
      _onPendingData: [Function: nop],
      agent: [Agent],
      socketPath: undefined,
      method: 'POST',
      maxHeaderSize: undefined,
      insecureHTTPParser: undefined,
      joinDuplicateHeaders: undefined,
      path: '/api/asset/upload',
      _ended: true,
      res: [IncomingMessage],
      aborted: false,
      timeoutCb: null,
      upgradeOrConnect: false,
      parser: null,
      maxHeadersCount: null,
      reusedSocket: false,
      host: '10.0.0.4',
      protocol: 'http:',
      [Symbol(kCapture)]: false,
      [Symbol(kBytesWritten)]: 0,
      [Symbol(kNeedDrain)]: true,
      [Symbol(corked)]: 0,
      [Symbol(kOutHeaders)]: [Object: null prototype],
      [Symbol(errored)]: null,
      [Symbol(kHighWaterMark)]: 16384,
      [Symbol(kRejectNonStandardBodyWrites)]: false,
      [Symbol(kUniqueHeaders)]: null
    },
    response: {
      status: 504,
      statusText: 'Gateway Time-out',
      headers: [Object],
      config: [Object],
      request: [ClientRequest],
      data: '<html>\r\n' +
        '<head><title>504 Gateway Time-out</title></head>\r\n' +
        '<body>\r\n' +
        '<center><h1>504 Gateway Time-out</h1></center>\r\n' +
        '<hr><center>nginx/1.25.0</center>\r\n' +
        '</body>\r\n' +
        '</html>\r\n'
    },
    isAxiosError: true,
    toJSON: [Function: toJSON]
  },
  response: '<html>\r\n' +
    '<head><title>504 Gateway Time-out</title></head>\r\n' +
    '<body>\r\n' +
    '<center><h1>504 Gateway Time-out</h1></center>\r\n' +
    '<hr><center>nginx/1.25.0</center>\r\n' +
    '</body>\r\n' +
    '</html>\r\n'
}
Error asset:  {
  file: '/mnt/media/photos/Jahre/2019/Italien 2019/20190906_210410.mp4',
  reason: Error: Request failed with status code 504
      at createError (/usr/src/app/node_modules/axios/lib/core/createError.js:16:15)
      at settle (/usr/src/app/node_modules/axios/lib/core/settle.js:17:12)
      at IncomingMessage.handleStreamEnd (/usr/src/app/node_modules/axios/lib/adapters/http.js:322:11)
      at IncomingMessage.emit (node:events:526:35)
      at endReadableNT (node:internal/streams/readable:1359:12)
      at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
    config: {
      transitional: [Object],
      adapter: [Function: httpAdapter],
      transformRequest: [Array],
      transformResponse: [Array],
      timeout: 0,
      xsrfCookieName: 'XSRF-TOKEN',
      xsrfHeaderName: 'X-XSRF-TOKEN',
      maxContentLength: Infinity,
      maxBodyLength: Infinity,
      validateStatus: [Function: validateStatus],
      headers: [Object],
      method: 'post',
      maxRedirects: 0,
      url: 'http://10.0.0.4:2283/api/asset/upload',
      data: [FormData]
    },
    request: ClientRequest {
      _events: [Object: null prototype],
      _eventsCount: 3,
      _maxListeners: undefined,
      outputData: [],
      outputSize: 0,
      writable: true,
      destroyed: true,
      _last: true,
      chunkedEncoding: true,
      shouldKeepAlive: false,
      maxRequestsOnConnectionReached: false,
      _defaultKeepAlive: true,
      useChunkedEncodingByDefault: true,
      sendDate: false,
      _removedConnection: false,
      _removedContLen: false,
      _removedTE: false,
      strictContentLength: false,
      _contentLength: null,
      _hasBody: true,
      _trailer: '',
      finished: false,
      _headerSent: true,
      _closed: true,
      socket: [Socket],
      _header: 'POST /api/asset/upload HTTP/1.1\r\n' +
        'Accept: application/json, text/plain, */*\r\n' +
        'Content-Type: multipart/form-data; boundary=--------------------------284543707829071763435817\r\n' +
        'x-api-key: <redacted>\r\n' +
        'User-Agent: axios/0.26.1\r\n' +
        'Host: 10.0.0.4:2283\r\n' +
        'Connection: close\r\n' +
        'Transfer-Encoding: chunked\r\n' +
        '\r\n',
      _keepAliveTimeout: 0,
      _onPendingData: [Function: nop],
      agent: [Agent],
      socketPath: undefined,
      method: 'POST',
      maxHeaderSize: undefined,
      insecureHTTPParser: undefined,
      joinDuplicateHeaders: undefined,
      path: '/api/asset/upload',
      _ended: true,
      res: [IncomingMessage],
      aborted: false,
      timeoutCb: null,
      upgradeOrConnect: false,
      parser: null,
      maxHeadersCount: null,
      reusedSocket: false,
      host: '10.0.0.4',
      protocol: 'http:',
      [Symbol(kCapture)]: false,
      [Symbol(kBytesWritten)]: 0,
      [Symbol(kNeedDrain)]: true,
      [Symbol(corked)]: 0,
      [Symbol(kOutHeaders)]: [Object: null prototype],
      [Symbol(errored)]: null,
      [Symbol(kHighWaterMark)]: 16384,
      [Symbol(kRejectNonStandardBodyWrites)]: false,
      [Symbol(kUniqueHeaders)]: null
    },
    response: {
      status: 504,
      statusText: 'Gateway Time-out',
      headers: [Object],
      config: [Object],
      request: [ClientRequest],
      data: '<html>\r\n' +
        '<head><title>504 Gateway Time-out</title></head>\r\n' +
        '<body>\r\n' +
        '<center><h1>504 Gateway Time-out</h1></center>\r\n' +
        '<hr><center>nginx/1.25.0</center>\r\n' +
        '</body>\r\n' +
        '</html>\r\n'
    },
    isAxiosError: true,
    toJSON: [Function: toJSON]
  },
  response: '<html>\r\n' +
    '<head><title>504 Gateway Time-out</title></head>\r\n' +
    '<body>\r\n' +
    '<center><h1>504 Gateway Time-out</h1></center>\r\n' +
    '<hr><center>nginx/1.25.0</center>\r\n' +
    '</body>\r\n' +
    '</html>\r\n'
}

[mikex99]

I'm having this issue on windows without using the cli, just uploading images from mobile app. When It happens, the web interface starts throwing error 500 and I cannot stop the immich_server. The immich_server is full of repeated logs.
[Nest] 7 - 09/27/2023, 2:27:29 PM ERROR [ExceptionsHandler] Connection terminated due to connection timeout
Error: Connection terminated due to connection timeout
at Connection. (/usr/src/app/node_modules/pg/lib/client.js:132:73)
at Object.onceWrapper (node:events:628:28)
at Connection.emit (node:events:514:28)
at Socket. (/usr/src/app/node_modules/pg/lib/connection.js:63:12)
at Socket.emit (node:events:514:28)
at TCP. (node:net:323:12)