[Diffusers] Unloading LoRANetwork after applied to model withour reloading

[alejobrainz]

Hi, I'm successfully using the create_network_from_weights and applying it to a diffusers pipeline, but after inference, I want to unload it without unloading the whole pipeline. This is the code I'm using to load the network:

    vae = pipe.vae
    text_encoder = pipe.text_encoder
    unet = pipe.unet

    lora_path = "./lora/" + lora_path
    sd = load_file(lora_path)
    lora_network, sd = create_network_from_weights(weight, None, vae, text_encoder, unet, sd)
    lora_network.apply_to(text_encoder, unet)
    lora_network.load_state_dict(sd)
    lora_network.to("cuda", dtype=torch.float16)

Please advise,

Alejandro.