tritonclient is_server_ready issue

[FranciscoLozCoding]

Seeing this:

[2026-02-06 20:08:22,551: ERROR/ForkPoolWorker-3] Task job_system.tasks.process_image_task[90c020fb-a15c-42b3-9454-944bad75d648] raised unexpected: UnpickleableExceptionWrapper('tritonclient.utils', 'InferenceServerException', (), 'InferenceServerException()')
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 477, in trace_task
    R = retval = fun(*args, **kwargs)
                 ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 760, in __protected_call__
    return self.run(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/celery/app/autoretry.py", line 60, in run
    ret = task.retry(exc=exc, **retry_kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/celery/app/task.py", line 736, in retry
    raise_with_context(exc)
  File "/usr/local/lib/python3.11/site-packages/celery/app/autoretry.py", line 38, in run
    return task._orig_run(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/job_system/tasks.py", line 203, in process_image_task
    raise self.retry(countdown=retry_delay, exc=exc)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/celery/app/task.py", line 736, in retry
    raise_with_context(exc)
  File "/app/job_system/tasks.py", line 168, in process_image_task
    triton_client = get_triton_client()
                    ^^^^^^^^^^^^^^^^^^^
  File "/app/job_system/tasks.py", line 104, in get_triton_client
    if _triton_client.is_server_ready():
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/tritonclient/grpc/_client.py", line 344, in is_server_ready
    raise_error_grpc(rpc_error)
  File "/usr/local/lib/python3.11/site-packages/tritonclient/grpc/_utils.py", line 77, in raise_error_grpc
    raise get_error_grpc(rpc_error) from None
celery.utils.serialization.UnpickleableExceptionWrapper: InferenceServerException()

Any clues?

Originally posted by @iperezx in #2 (comment)

[FranciscoLozCoding]

Seems like it goes back to a healthy state after some time.

Originally posted by @iperezx in #2 (comment)

[FranciscoLozCoding]

Issue is most likely the cause of removing this line from the code. We will add it back in but make it configurable with deployment files so if we deploy in different gpu hardware we can change it for better utilization of the gpu.

https://github.com/waggle-sensor/sage-nrp-image-search/pull/2/changes/BASE..26a701efdac663934e7a81ae03a0ea65e2a013a0#r2765774759

related docs: https://docs.pytorch.org/docs/stable/notes/cuda.html

[FranciscoLozCoding]

PR opened for this issue: #5

[FranciscoLozCoding]

The issue was RAM not GPU, the pod was hitting the memory limit. The PR will up the requested memory and limit. In terms of the gpu, I still need to test if max_split_size_mb is needed.

[FranciscoLozCoding]

Now I am running into another issue

[MODEL] Final Generated Description: ERROR in apply_chat_template: Cannot use apply_chat_template because this processor does not have a chat template.

I had to switch the k8s deployment file to use seperate volumes for Gemma and CLIP so model dirs don’t share the same storage

TLDR: clip files were appearing in gemma directory

[iperezx]

The issue was RAM not GPU, the pod was hitting the memory limit. The PR will up the requested memory and limit. In terms of the gpu, I still need to test if max_split_size_mb is needed.

prod is set to 16GB. But the overlays are set to 8GB.

[iperezx]

Now I am running into another issue

[MODEL] Final Generated Description: ERROR in apply_chat_template: Cannot use apply_chat_template because this processor does not have a chat template.

I had to switch the k8s deployment file to use seperate volumes for Gemma and CLIP so model dirs don’t share the same storage

TLDR: clip files were appearing in gemma directory

Thats weird behavior on k8s. Don't see that on the local docker compose instance

[FranciscoLozCoding]

The issue was RAM not GPU, the pod was hitting the memory limit. The PR will up the requested memory and limit. In terms of the gpu, I still need to test if max_split_size_mb is needed.

prod is set to 16GB. But the overlays are set to 8GB.

Not sure what you mean by this @iperezx, I dont see an overlay for other deployments that sets triton deployment to 8GB. I'm looking at issue_4 branch.

[FranciscoLozCoding]

The issue was RAM not GPU, the pod was hitting the memory limit. The PR will up the requested memory and limit. In terms of the gpu, I still need to test if max_split_size_mb is needed.

Removed max_split_size_mb while keeping the upped memory limit and the triton pod is running fine now so the issue was RAM and not GPU

Triton now floats around 75% - 92% memory usage of requested memory. That means the Triton pod floats around 9 - 11 GiB of used memory. NRP requires you to be in between20% - 150% so this new requested memory is working great. The old requested memory and memory limit was 8 GiB so you can see why the pod was not healthy.

The memory limit is now set to 16GiB while the requested memory is now 12GiB so that should give it head room if the memory spikes up for some reason.

Link to NRP Gradio dashboard

TLDR: The issue was RAM not GPU, the pod was hitting the memory limit

[iperezx]

The issue was RAM not GPU, the pod was hitting the memory limit. The PR will up the requested memory and limit. In terms of the gpu, I still need to test if max_split_size_mb is needed.

prod is set to 16GB. But the overlays are set to 8GB.

Not sure what you mean by this @iperezx, I dont see an overlay for other deployments that sets triton deployment to 8GB. I'm looking at issue_4 branch.

The base overlay: https://github.com/waggle-sensor/sage-nrp-image-search/blob/main/kubernetes/base/triton.yaml#L43

[FranciscoLozCoding]

The issue was RAM not GPU, the pod was hitting the memory limit. The PR will up the requested memory and limit. In terms of the gpu, I still need to test if max_split_size_mb is needed.

prod is set to 16GB. But the overlays are set to 8GB.

Not sure what you mean by this @iperezx, I dont see an overlay for other deployments that sets triton deployment to 8GB. I'm looking at issue_4 branch.

The base overlay: https://github.com/waggle-sensor/sage-nrp-image-search/blob/main/kubernetes/base/triton.yaml#L43

Ahh yes, once I merge issue_4 that will change. Right now prod was deployed using file versions in issue_4 branch