Huge memory usage when saving non-readonly project

[xalexalex]

Hello,

when iterating over the images of a large project (>200 mirax WSIs), memory is allocated but never freed until I get a java.lang.OutOfMemoryError: java.lang.OutOfMemoryError: Java heap space. Full stack trace is here.

I have tried the following JVM options (one at a time) but eventually I still get the OOM error:

• "-XX:MaxRAMPercentage=50",
• "-XX:MaxRAMPercentage=90",
• "-Xmx8g",

Culprit code used to generate the stack trace follows:

with QuPathProject('/path/to/proj', mode='r+') as qp:
    for image in qp.images:
        print(image.name)
        if image.name not in some_list:
            continue
        print('would go on')
        continue

However the error is reproducible even with something as simple as:

with QuPathProject('/path/to/proj', mode='r+') as qp:
    for image in qp.images:
        pass

note that after a recent commit (I think this one, but I haven't bisected), one can iterate over qp.images just fine if the project is in 'r' mode.

However this doesn't fix my problem because I have to iterate over this project in 'r+' mode to edit the detections of all images.

Any help or guidance?

Thanks!

[xalexalex]

Actually the problem is not the iteration, but saving the large project.
I can reproduce with just this:

qp = QuPathProject('/path/to/large/proj', mode='r+')
qp.save()

[ap--]

Hey @xalexalex

Thanks for reporting! Interestingly I can't reproduce this with svs files locally, but I can reproduce the error with mirax slides...

mkdir $HOME/qp-memory-error
cd $HOME/qp-memory-error
python -m venv venv
source venv/bin/activate
pip install paquo
mkdir qupath
paquo get_qupath --install-path ./qupath 0.4.4
export PAQUO_QUPATH_DIR=$HOME/qp-memory-error/qupath/QuPath-0.4.4
wget https://data.cytomine.coop/open/openslide/mirax-mrxs/CMU-1.zip
unzip CMU-1.zip 

# make_project.py 
from paquo.projects import QuPathProject
with QuPathProject("my-project", "w") as p:
    for i in range(n := 200):
        print(f"adding image {i+1} of {n}")
        p.add_image("./CMU-1.mrxs", allow_duplicates=True)

$ python make_project.py 
adding image 1 of 200
...
adding image 133 of 200
adding image 134 of 200
20:42:23.310 [main] [DEBUG] qupath.lib.projects.DefaultProject - Writing project to /home/poehlmann/qp-memory-error/my-project/project.qpproj.tmp
20:42:23.312 [main] [DEBUG] qupath.lib.projects.DefaultProject - Backing up existing project to /home/poehlmann/qp-memory-error/my-project/project.qpproj.backup
20:42:23.313 [main] [DEBUG] qupath.lib.projects.DefaultProject - Renaming project to /home/poehlmann/qp-memory-error/my-project/project.qpproj
Traceback (most recent call last):
  File "ImageServerProvider.java", line 223, in qupath.lib.images.servers.ImageServerProvider.getPreferredUriImageSupport
Exception: Java Exception

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/poehlmann/qp-memory-error/make_project.py", line 6, in <module>
    p.add_image("./CMU-1.mrxs", allow_duplicates=True)
  File "/usr/lib/python3.10/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/home/poehlmann/qp-memory-error/venv/lib/python3.10/site-packages/paquo/projects.py", line 333, in add_image
    support = ImageServerProvider.getPreferredUriImageSupport(
java.lang.java.lang.OutOfMemoryError: java.lang.OutOfMemoryError: Java heap space

I'll have to see where the memory leak originates...

Cheers,
Andreas 😃