How to reduce memory usage when evaluating?

[yxchng]

ERROR:mteb.evaluation.MTEB:Error while evaluating NTREXBitextMining: CUDA out of memory. Tried to allocate 848.00 MiB. GPU 0 has a total capacity of 79.11 GiB of which 356.88 Mi
B is free. Process 2751276 has 78.75 GiB memory in use. Of the allocated memory 67.81 GiB is allocated by PyTorch, and 8.04 GiB is reserved by PyTorch but unallocated. If reserv
ed but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorc
h.org/docs/stable/notes/cuda.html#environment-variables) 

currently using 8x80gb H100.

[KennethEnevoldsen]

Have you tried setting the encode_kwargs = {"batch_size": 32} (docs)