When tasks involving the train step are run in an interactive session, the checkpoints are being saved to the file server, which is not intended and causes the file server to fill up quickly. This also might be happening for any tasks that are being tracked by clearml but not executed remotely.