00_pytorch_fundamentals Exercises Question 5&6

[Perian-Yan]

Hi, I got a different answer to this question. My code is

# set random seed for GPU
torch.cuda.manual_seed(1234)

device = "cuda" if torch.cuda.is_available() else "cpu"
T1_gpu = torch.rand((2,3), device=device)
T2_gpu = torch.rand((2,3), device=device)
print(T1_gpu)
print(T2_gpu)

However, the solution is:

I don't know why we need to set torch.manual_seed(1234) in the second cell again. In this way, what actually does torch.cuda.manual_seed(1234) do?
Thank you in advance!

[AS1100K]

When using manual_seed, it works only with next function or code.
In order to get same output you should fix your code as follows:

device = "cuda" if torch.cuda.is_available() else "cpu"

torch.cuda.manual_seed(1234)
T1_gpu = torch.rand((2,3), device=device)

torch.cuda.manual_seed(1234)
T2_gpu = torch.rand((2,3), device=device)

print(T1_gpu)
print(T2_gpu)

Output:

tensor([[0.1272, 0.8167, 0.5440],
        [0.6601, 0.2721, 0.9737]], device='cuda:0')
tensor([[0.1272, 0.8167, 0.5440],
        [0.6601, 0.2721, 0.9737]], device='cuda:0')

[Perian-Yan]

Thank you so much! But what is the difference between torch.manual_seed() and torch.cuda.maual_seed()?
I found there are two ways to generate a random tensor in GPU, either generating directly in GPU or generating in CPU then moving to GPU.
Consider two different manual_seed, there are 4 combinations:
C_gpu will change every time I run the cell, which is not reproducible. B and D have the same result. I guess B and D are generated in gpu directly, while A is generated in CPU and then moved to GPU later. They are different, however.

[AS1100K]

There is no difference between torch.manual_seed and torch.cuda.manual_seed when running on single GPU or without GPU.

So, torch.cuda.manual_seed sets manual seed for the current GPU

WARNING
If you are working with a multi-GPU model, this function is insufficient to get determinism. To seed all GPUs, use manual_seed_all().

and torch.manual_seed sets manual seed across all CPU(s) and GPU(s).

NOTE: When using torch.manual_seed or torch.cuda.manual_seed, the reproducibility also depends on the device on which tensor was created. So, If we use any of the reproducibility functions, there will be a different Tensor for both CPU and GPU.

Source: PyTorch Reproducibility Documentation, torch.cuda.manual_seed Documentation

[Perian-Yan]

Thank you! It's clear now 👍