Epoch training won't go further than 11%, and val/test won't go further than 81%.

[jackbart94]

Hi everyone,

I'm not sure this is the right place to ask this, but probably you can give me an hand. I don't know why for every epoch, regardless of which batch size I specify, it never seems to go above a certain threshold. Anyone knows why is that?

I'll attach the output of a simple 2-epoch test I've ran.

Getting data info: 100%|█████████████████████████████████████████████████████████████████████████████████████| 181/181 [00:00<00:00, 978.03it/s]
Validation dataset prepared!█████████████████████████████████████████████▏                                  | 106/181 [00:00<00:00, 1053.75it/s]
Validation sanity check: 100%|████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:08<00:00,  3.51s/it]tensor(29.5314, device='cuda:0')
val epoch end!
Global seed set to 42                                                                                                                           
Preparing training data...
Getting data info: 100%|█████████████████████████████████████████████████████████████████████████████████████| 801/801 [00:01<00:00, 776.54it/s]
Training dataset prepared!
Epoch 0:  11%|█████▋                                                | 15/141 [00:27<03:47,  1.81s/it, loss=238, v_num=39, train_loss_step=220.0]tensor(237.8313, device='cuda:0')
train epoch end!
Epoch 1:  11%|███▏                          | 15/141 [00:27<03:54,  1.86s/it, loss=236, v_num=39, train_loss_step=223.0, train_loss_epoch=238.0]tensor(234.1873, device='cuda:0')
train epoch end!
Epoch 1:  11%|███▏                          | 15/141 [00:28<03:56,  1.88s/it, loss=236, v_num=39, train_loss_step=223.0, train_loss_epoch=238.0]
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
Preparing validation data...
Getting data info: 100%|█████████████████████████████████████████████████████████████████████████████████████| 181/181 [00:00<00:00, 979.87it/s]
Validation dataset prepared!
                                                                                                                                               tensor(34.4227, device='cuda:0')███████████████████████████████████████████████████████████████▍                 | 22/27 [00:31<00:04,  1.04it/s]
val epoch end!
--------------------------------------------------------------------------------
DATALOADER:0 VALIDATE RESULTS
{'average_validation_loss': 34.42267608642578,
 'val_loss': 34.422672271728516,
 'val_loss_epoch': 34.422672271728516}
--------------------------------------------------------------------------------
                                                                                                                                               LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]                                                                                                        
Preparing test data...
Getting data info: 100%|██████████████████████████████████████████████████████████████████████████████████████| 181/181 [00:04<00:00, 39.79it/s]
Test dataset prepared!
Testing:  81%|███████████████████████████████████████████████████████████████████████████████▊                  | 22/27 [00:30<00:04,  1.01it/s]--------------------------------------------------------------------------------
DATALOADER:0 TEST RESULTS
{'val_loss': 34.45164489746094, 'val_loss_epoch': 34.45164489746094}
--------------------------------------------------------------------------------
Testing:  81%|███████████████████████████████████████████████████████████████████████████████▊                  | 22/27 [00:30<00:07,  1.41s/it]

Thanks for helping!