Tensors not Updated after Training

[ChrisWang13]

Describe the bug

I have been experimenting with custom weights for diffusion models, particularly focusing on generating human faces using the DiffusionPipeline from the diffusers library. However, even after loading the updated weights, there seems to be no appreciable change in the output.

Evaluate code

import torch
from diffusers import DiffusionPipeline

folder_name = "./train_3000"

# Load the diffusion pipeline and move it to CUDA
pipe = DiffusionPipeline.from_pretrained(
    "CompVis/stable-diffusion-v1-4", torch_dtype=torch.float16
).to("cuda")

old_params = {name: param.clone().to("cuda") for name, param in pipe.unet.named_parameters()}

# Load your custom weights for UNet and textual inversion module
pipe.unet.load_attn_procs(
    folder_name, weight_name="pytorch_custom_diffusion_weights.bin"
)
pipe.load_textual_inversion(folder_name, weight_name="<new1>.bin")

# Tensor to accumulate the changes
change = torch.tensor(0.0, device="cuda") # Ensure this is a floating point tensor to accumulate mean values

# Compare the parameters after loading new weights
for name, new_param in pipe.unet.named_parameters():
    if name in old_params:
        old_param = old_params[name]
        # Calculate the mean absolute change and accumulate it
        change += torch.max(torch.abs(new_param - old_param))

print(change)

Expected output

Some big tensor changes.

Actual output

tensor(0., device='cuda:0', grad_fn=<AddBackward0>)

[dongkyunk]

Do you have --no_safe_serialization in your training code?