Clear memory after each fold in cross validation

[talhaanwarch]

When i start my program, dedicated GPU memory is 0.2-0.5 GB. During fold 1 it reaches to 6 GB (approx), after fold 1 before fold 2, i added sleep command, during this the GPU memory is 3GB. I want it to reach back to 0.5 GB. Here is my code

for train_index, test_index in skf.split(df['observation code'], df['Raunkiaer lifeform']):
    train_split,test_split=df.iloc[train_index].reset_index(drop=True),\
    df.iloc[test_index].reset_index(drop=True)
    
    model = OurModel(train_split,test_split,get_class_label)
    trainer = Trainer(max_epochs=1, auto_lr_find=True, auto_scale_batch_size=False,
                      deterministic=True,
                      gpus=-1,precision=16,
                      accumulate_grad_batches=8,
                      stochastic_weight_avg=True,
                      progress_bar_refresh_rate=0,
                     
                      )
    trainer.fit(model)
    trainer.test(model)
    print('sleep start')
    torch.cuda.empty_cache()
    del model
    sleep(120)
    print('sleep end')

Any solution?
I also noticed in sleep stage as fold increases the memory sizes increase such as after first fold it is 3GB,. after 2nd fold its 4Gb and so on eventually leading to memory error.

[tchaton]

Dear @talhaanwarch,

Mind updating PyTorch Lightning to master. Does this still persist ?

Best,
T.C

[talhaanwarch]

should i install via source code? i am using version 1.4.2, or should i upgrade to 1.4.3?