High memory consumption

[sneumann]

Hi,
I have a set of ~90 mzML files totalling ~40GB from an Orbitrap Elite converted via msconvert 3.0.11110.
If I run mzml2isa on them, the process memory according to top goes to 0.015TB(!) before eventually being OOM-killed.
This is on mzml2isa 1.0.3 I have only started to debug, will report back.

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                                                                
 3167 sneumann  20   0 15,555g 0,015t  19088 S  92,4 23,7  23:41.80 mzml2isa                                                                                                                               

If there are any known memory caveats I'd be happy about a brief hint.
Yours, Steffen

[sneumann]

Ok, works for a single file, trying on a different machine now. Will report back. Steffen

[sneumann]

Hi, just coming back to this, in a different study we have a whopping 100*10GB=1TB of mzML, so memory efficiency is coming back as important issue, even on a machine with 256GB RAM. Processing a single
Iteration over files happens here:

mzml2isa/mzml2isa/parsing.py

Line 129 in a946af3

metalist = list(map(_parse_file, files_iter))

so I am wondering if somewhere objects are not freed. Not a python expert, so I wouldn't know how release and garbage collection works.
Yours, Steffen