Smart Search errors for job concurrency more then 1

[OperKH]

I decided to switch to nllb-clip-large-siglip__mrl model. I changed it in settings and run SMART SEARCH job.

In a 2-3 minutes I noticed that my GPU doesn't have a load.

I opened immich_machine_learning logs and saw this:

immich_machine_learning logs

2024-08-15 20:31:49.173463213 [E:onnxruntime:Default, cuda_call.cc:118 CudaCall] CUDA failure 700: an illegal memory access was encountered ; GPU=0 ; hostname=8c9c550f3aa0 ; file=/onnxruntime_src/onnxruntime/core/providers/cuda/gpu_data_transfer.cc ; line=63 ; expr=cudaMemcpyAsync(dst_data, src_data, bytes, cudaMemcpyHostToDevice, static_cast<cudaStream_t>(stream.GetHandle())); 
2024-08-15 20:31:49.173490112 [E:onnxruntime:Default, cuda_call.cc:118 CudaCall] CUDA failure 700: an illegal memory access was encountered ; GPU=0 ; hostname=8c9c550f3aa0 ; file=/onnxruntime_src/onnxruntime/core/providers/cuda/cuda_execution_provider.cc ; line=446 ; expr=cudaStreamSynchronize(static_cast<cudaStream_t>(stream_)); 
2024-08-15 20:31:49.182438963 [E:onnxruntime:Default, cuda_call.cc:118 CudaCall] CUDA failure 700: an illegal memory access was encountered ; GPU=0 ; hostname=8c9c550f3aa0 ; file=/onnxruntime_src/onnxruntime/core/providers/cuda/gpu_data_transfer.cc ; line=63 ; expr=cudaMemcpyAsync(dst_data, src_data, bytes, cudaMemcpyHostToDevice, static_cast<cudaStream_t>(stream.GetHandle())); 
2024-08-15 20:31:49.182468125 [E:onnxruntime:Default, cuda_call.cc:118 CudaCall] CUDA failure 700: an illegal memory access was encountered ; GPU=0 ; hostname=8c9c550f3aa0 ; file=/onnxruntime_src/onnxruntime/core/providers/cuda/cuda_execution_provider.cc ; line=446 ; expr=cudaStreamSynchronize(static_cast<cudaStream_t>(stream_)); 
2024-08-15 20:31:49.188099313 [E:onnxruntime:Default, cuda_call.cc:118 CudaCall] CUDA failure 700: an illegal memory access was encountered ; GPU=0 ; hostname=8c9c550f3aa0 ; file=/onnxruntime_src/onnxruntime/core/providers/cuda/gpu_data_transfer.cc ; line=63 ; expr=cudaMemcpyAsync(dst_data, src_data, bytes, cudaMemcpyHostToDevice, static_cast<cudaStream_t>(stream.GetHandle())); 
2024-08-15 20:31:49.188124564 [E:onnxruntime:Default, cuda_call.cc:118 CudaCall] CUDA failure 700: an illegal memory access was encountered ; GPU=0 ; hostname=8c9c550f3aa0 ; file=/onnxruntime_src/onnxruntime/core/providers/cuda/cuda_execution_provider.cc ; line=446 ; expr=cudaStreamSynchronize(static_cast<cudaStream_t>(stream_)); 
[08/15/24 20:31:49] ERROR    Exception in ASGI application                      
                                                                                
                             ╭─────── Traceback (most recent call last) ───────╮
                             │ /usr/src/app/main.py:152 in predict             │
                             │                                                 │
                             │   149 │   │   inputs = text                     │
                             │   150 │   else:                                 │
                             │   151 │   │   raise HTTPException(400, "Either  │
                             │ ❱ 152 │   response = await run_inference(inputs │
                             │   153 │   return ORJSONResponse(response)       │
                             │   154                                           │
                             │   155                                           │
                             │                                                 │
                             │ /usr/src/app/main.py:175 in run_inference       │
                             │                                                 │
                             │   172 │   │   response[entry["task"]] = output  │
                             │   173 │                                         │
                             │   174 │   without_deps, with_deps = entries     │
                             │ ❱ 175 │   await asyncio.gather(*[_run_inference │
                             │   176 │   if with_deps:                         │
                             │   177 │   │   await asyncio.gather(*[_run_infer │
                             │   178 │   if isinstance(payload, Image):        │
                             │                                                 │
                             │ /usr/src/app/main.py:170 in _run_inference      │
                             │                                                 │
                             │   167 │   │   │   │   message = f"Task {entry[' │
                             │       output of {dep}"                          │
                             │   168 │   │   │   │   raise HTTPException(400,  │
                             │   169 │   │   model = await load(model)         │
                             │ ❱ 170 │   │   output = await run(model.predict, │
                             │   171 │   │   outputs[model.identity] = output  │
                             │   172 │   │   response[entry["task"]] = output  │
                             │   173                                           │
                             │                                                 │
                             │ /usr/src/app/main.py:188 in run                 │
                             │                                                 │
                             │   185 │   if thread_pool is None:               │
                             │   186 │   │   return func(*args, **kwargs)      │
                             │   187 │   partial_func = partial(func, *args, * │
                             │ ❱ 188 │   return await asyncio.get_running_loop │
                             │   189                                           │
                             │   190                                           │
                             │   191 async def load(model: InferenceModel) ->  │
                             │                                                 │
                             │ /usr/local/lib/python3.11/concurrent/futures/th │
                             │ read.py:58 in run                               │
                             │                                                 │
                             │ /usr/src/app/models/base.py:60 in predict       │
                             │                                                 │
                             │    57 │   │   self.load()                       │
                             │    58 │   │   if model_kwargs:                  │
                             │    59 │   │   │   self.configure(**model_kwargs │
                             │ ❱  60 │   │   return self._predict(*inputs, **m │
                             │    61 │                                         │
                             │    62 │   @abstractmethod                       │
                             │    63 │   def _predict(self, *inputs: Any, **mo │
                             │                                                 │
                             │ /usr/src/app/models/clip/visual.py:23 in        │
                             │ _predict                                        │
                             │                                                 │
                             │   20 │                                          │
                             │   21 │   def _predict(self, inputs: Image.Image │
                             │      NDArray[np.float32]:                       │
                             │   22 │   │   image = decode_pil(inputs)         │
                             │ ❱ 23 │   │   res: NDArray[np.float32] = self.se │
                             │   24 │   │   return res                         │
                             │   25 │                                          │
                             │   26 │   @abstractmethod                        │
                             │                                                 │
                             │ /usr/src/app/sessions/ort.py:49 in run          │
                             │                                                 │
                             │    46 │   │   input_feed: dict[str, NDArray[np. │
                             │    47 │   │   run_options: Any = None,          │
                             │    48 │   ) -> list[NDArray[np.float32]]:       │
                             │ ❱  49 │   │   outputs: list[NDArray[np.float32] │
                             │       run_options)                              │
                             │    50 │   │   return outputs                    │
                             │    51 │                                         │
                             │    52 │   @property                             │
                             │                                                 │
                             │ /opt/venv/lib/python3.11/site-packages/onnxrunt │
                             │ ime/capi/onnxruntime_inference_collection.py:22 │
                             │ 0 in run                                        │
                             │                                                 │
                             │    217 │   │   if not output_names:             │
                             │    218 │   │   │   output_names = [output.name  │
                             │    219 │   │   try:                             │
                             │ ❱  220 │   │   │   return self._sess.run(output │
                             │    221 │   │   except C.EPFail as err:          │
                             │    222 │   │   │   if self._enable_fallback:    │
                             │    223 │   │   │   │   print(f"EP Error: {err!s │
                             ╰─────────────────────────────────────────────────╯
                             Fail: [ONNXRuntimeError] : 1 : FAIL : CUDA failure 
                             700: an illegal memory access was encountered ;    
                             GPU=0 ; hostname=8c9c550f3aa0 ;                    
                             file=/onnxruntime_src/onnxruntime/core/providers/cu
                             da/gpu_data_transfer.cc ; line=63 ;                
                             expr=cudaMemcpyAsync(dst_data, src_data, bytes,    
                             cudaMemcpyHostToDevice,                            
                             static_cast<cudaStream_t>(stream.GetHandle()));   

I decided to reduce Smart Search concurrency in settings from 8 to 4, but had the same errors as above.
Then I reduce Smart Search concurrency from 4 to 2 and it was running without errors for all 56 000 assets.

When SMART SEARCH job had finished, I decided to run it again for missing assets. And job started for a 25 000 assets - looks like a half of them not processed.

I looked for the immich_server logs and saw this:

immich_server logs

[Nest] 19  - 08/16/2024, 6:50:43 AM   ERROR [Microservices:JobService] Unable to run job handler (smartSearch/smart-search): QueryFailedError: pgvecto.rs: Bad literal.
INFORMATION: hint = Expect a number.
[Nest] 19  - 08/16/2024, 6:50:43 AM   ERROR [Microservices:JobService] QueryFailedError: pgvecto.rs: Bad literal.
INFORMATION: hint = Expect a number.
    at PostgresQueryRunner.query (/usr/src/app/node_modules/typeorm/driver/postgres/PostgresQueryRunner.js:219:19)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async InsertQueryBuilder.execute (/usr/src/app/node_modules/typeorm/query-builder/InsertQueryBuilder.js:106:33)
    at async SearchRepository.upsert (/usr/src/app/dist/repositories/search.repository.js:203:9)
    at async SmartInfoService.handleEncodeClip (/usr/src/app/dist/services/smart-info.service.js:134:9)
    at async /usr/src/app/dist/services/job.service.js:148:36
    at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:394:28)
    at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:581:24)
[Nest] 19  - 08/16/2024, 6:50:43 AM   ERROR [Microservices:JobService] Object:
{
  "id": "8b0e6f19-15d1-4a1e-afed-aed2a380e0f8"
}

Finally I reduced Smart Search concurrency to 1 and it finished without errors

I started to use Smart Search since v1.91.0 I used XLM-Roberta-Large-ViT-H-14__frozen_laion5b_s13b_b90k model and had 8 jobs without any issues and errors.

I'm using NVIDIA GTX1070
Driver Version: 550.54.14
CUDA Version: 12.4
pgvecto-rs:pg14-v0.2.0@sha256:90724186f0a3517cf6914295b5ab410db9ce23190a2d9d0b9dd6463e3fa298f0

[mertalev]

The GPU was most likely running out of memory at higher concurrencies.

[OperKH]

No, it consumes 2646MiB from 8192MiB.
I tried increase concurrences and it still use the same amount of RAM.

[mertalev]

Hmm, hard to say then. It could be hitting an edge case in how CUDA streams are managed in ONNX Runtime.