How to stop GPU occupancy after the model end training?

[pzSuen]

I had run the code in "Putting All Together," but I found that the GPU is still occupying when the training is over.
How can I stop it?

[vlomonaco]

Are you sure this is related to Avalanche and not Pytorch? Please provide the snippet of code to reproduce the issue! :)

[AndreaCossu]

Maybe I misunderstood your question. Are you saying that the GPU is still occupied after the eval call or after the entire script is finished?

[pzSuen]

I mean "after the entire script is finished".

I want a more explicit way to close the strategy if I don't use it later, rather than by del logger. Such as strategy.close() like writer.close() in Tensorboard.

[AntonioCarta]

I think @pzSuen is pointing out at an important use case. In general, since a CL system may run for a long time, we need to give the ability to stop and restart it. This entails moving models back to the cpu, closing logger files and much more. Eventually, the possibility to serialize everything to close and restart the process.

An easy way to start to start to support this would be to close the files after every train and eval call. @AndreaCossu is this doable for the Tensorboard logger? are there any limitations?

[AndreaCossu]

The closing part is not difficult. For the resume of the training / evaluation from the point in which it has been left off, I think we need to proceed logger by logger. For Tensorboard I have to check if there is the possibility to append data to an existing run in order to have a single curve for each metric. Otherwise, we will have to create a new run which starts from the previous state (e.g. previous epoch, experience etc.).

[pzSuen]

@AntonioCarta is right. I ran a shell script to test different strategies, but it couldn't stop and continue with the GPU occupancy.

#!/bin/sh
gpu='3'
for plugin in Naive Replay GDumb AGEM EWC LwF; do

  CUDA_VISIBLE_DEVICES=${gpu} python came_cls.py -a --cl_plugin ${plugin} >>out.txt

done

I had saved my problem by not using Tensorboard, but escaping problems is not a good solution.
What I had said above is just advice.

Thank you~