Looking for ways to get model information like GPU memory usage

[GuanLuo]

Our application requires serving multiple models, in cases we can't load all models into memory, we would want to load as much as possible and later on unload / load model on demand. For this case, it would be much helpful if we can collect GPU memory usage of each ORT session created, to make better decision on sessions to be destroyed. I wonder if ORT keeps track of the memory usage at session level and what are the APIs that it exposes.

[pranavsharma]

We don't track memory usage at a session level for inferencing builds. Is this for Triton? I'll add this as a feature request.

[GuanLuo]

Yes, this is for Triton.

We don't track memory usage at a session level for inferencing builds.

Are there other kinds of builds?

[pranavsharma]

We do this for training builds, but it has not been tested for inferencing since (the last I checked) there might be some thread safety issues.