Skip to content

directory with large number of images leads to OOM #2

@romerocesar

Description

@romerocesar
/home/cesar/src/kopya/.venv/lib/python3.11/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
  warnings.warn(
/home/cesar/src/kopya/.venv/lib/python3.11/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=MobileNet_V3_Small_Weights.IMAGENET1K_V1`. You can also use `weights=MobileNet_V3_Small_Weights.DEFAULT` to get the most up-to-date weights.
  warnings.warn(msg)
2023-10-20 14:31:19,345: INFO Initialized: MobileNet v3 pretrained on ImageNet dataset sliced at GAP layer
INFO:imagededup.methods.cnn:Initialized: MobileNet v3 pretrained on ImageNet dataset sliced at GAP layer
2023-10-20 14:31:19,363: INFO Device set to cuda ..
INFO:imagededup.methods.cnn:Device set to cuda ..
2023-10-20 14:31:19,838: INFO Start: Image encoding generation
INFO:imagededup.methods.cnn:Start: Image encoding generation
Traceback (most recent call last):
  File "/home/cesar/src/kopya/cli.py", line 79, in <module>
    main()
  File "/home/cesar/src/kopya/cli.py", line 69, in main
    duplicates = find_duplicates(path=path)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/cesar/src/kopya/cli.py", line 44, in find_duplicates
    embeddings = cnn.encode_images(image_dir=path)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/cesar/src/kopya/.venv/lib/python3.11/site-packages/imagededup/methods/cnn.py", line 251, in encode_images
    return self._get_cnn_features_batch(image_dir=image_dir, recursive=recursive, num_workers=num_enc_workers)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/cesar/src/kopya/.venv/lib/python3.11/site-packages/imagededup/methods/cnn.py", line 146, in _get_cnn_features_batch
    arr = self.model(ims.to(self.device))
                     ^^^^^^^^^^^^^^^^^^^
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 38.00 MiB (GPU 0; 23.67 GiB total capacity; 21.50 GiB already allocated; 75.94 MiB free; 21.93 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions