Memory-spike on opening image for the first time (that isn't cached).

[mujeebnazir]

I set up the DSA with recomended docker-compose.yml in devops/dsa with default configuration. Then i imported around 200 TIFF image from s3 via assetstore and then started viewing all those images one after the another and i noticed my memory usage went from 1.5/30gb to 5.6 gb/30gb. But after i stopped viewing images the memory didn't came down.

1. So, what is the flush out mechanish here, won't it create a problem if RAM is smaller and multiple people are viewing different images and the memory usage keeps increasing incrementally without flushing out and will eventually crash the system??
2. Also if you could throw some light on this, i imported some non pyramidal tiff image which i noticed takes much more RAM then pyramidal ??

[manthey]

There are two caches that affect memory use: (1) a tile source cache that caches all of the file handlers and other data need to read image files; this allows switching between images faster, as some formats are slow to open. (2) A tile cache which caches individually served tiles. (1) is in the main process, (2) is usually on memcached or redis (depending on version and how things are set up). Both can be configured to be more or less conservative.

Internally, we use python as the main program language, but each tile source readers might use something else (e.g., some files are read via bioformats which uses java). Python is often slow to release memory, assuming you'll ask for it again later. It shouldn't grow beyond the system though - we've had systems up for months without issue.

Non pyramidal files are slower and more memory intensive. The viewers all want tiles (browsers won't handle images above 16k square, and some browsers it is less). For non-tiled images, we construct tiles on-demand (caching them in memory). To get a thumbnail of a non-tiled image requires reading the whole image, so it will fill the tile cache, but as you look at other images, it will drop older tiles from the cache.

There are a few service end points (api/v1/large_image/cache) to see what is in the cache and to ask it to clear (which should never be needed to do manually).