cuda for training, CPU for validation

[carlpe]

Is it possible to use cuda for training and CPU for validation? I did try it:

os.environ["CUDA_DEVICE_ORDER"] = "PCI_BUS_ID"
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

def validation(epoch_iterator_val):
    model.eval()
    with torch.no_grad():
        torch.device("cpu")
        for batch in epoch_iterator_val:
            val_inputs, val_labels = (batch["image"].cpu(), batch["label"].cpu())
            ...

def train(global_step, train_loader, dice_val_best, global_step_best):
    device = torch.device("cuda:0")
    model.train()
    epoch_loss = 0
    step = 0
    epoch_iterator = tqdm(
        train_loader, desc="Training (X / X Steps) (loss=X.X)", dynamic_ncols=True
    )
    for step, batch in enumerate(epoch_iterator):
        step += 1
        x, y = (batch["image"].cuda(), batch["label"].cuda())
        ....

RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!

[KumoLiu]

Hi @carlpe, of course you can but I'm afraid you didn't put the model on CPU. Could you please check it?
Hope it can help you, thanks!

[KumoLiu]

Hi @carlpe, since the tutorial is using iteration num, so you may not run into model.to("cuda") in the correct time. So you may have several way to solve this.
First, you could add model.to("cuda") this after for step, batch in enumerate(epoch_iterator): this line.
Second, you could change the eval_num to a multiple of validation data length.
Finally, for I always seem to run out of VRAM at the level of validation this problem, I think maybe you could change cache_rate to 0 in your val_ds, if that works then you won't need to put training and validation on different devices, which will greatly speed up the training.
Hope it can help you, thanks!

[carlpe]

Ok thanks ☺️

I ran out of VRAM with cache_rate=0 initially, that's why i wanted to try this. Ill try again later

[carlpe]

using the cuda for training and cpu for validation only made the code run, with cache_rate = 1 ...

low eval_num was just to check if it will run

[KumoLiu]

Did you set cache_rate=0 in both training and validation? Maybe you could also set cache_rate=0.5 or lower in the training part. Thanks!

[carlpe]

Yes i have tried almost every possible combination of cache_rate. I will try your suggestion tomorrow.

The validation with cpu is now taking around 200s/iteration with a 24 core Threadripper, but it is working 😅

[carlpe]

I found who to make it run now with cuda for both val and train.

The val_transforms did not have Spacingd on it since the images were aleady 1x1x1 like i wanted to use.
When I added the following to both val_transforms and train_transforms it allowed me to train and validate with no problems using cuda.

        RandSpatialCropd(
            keys=(["image", "label"]), roi_size=(480, 480, 480), random_size=False
        ),
        Spacingd(
            keys=["image", "label"],
            pixdim=(1, 1, 1),
            mode=("bilinear", "nearest"),
        ),

If I turn these off again it is running very slow even on batch size 1