How to validate trainnig is running ok by generating img/vid using the latest/currently trained weight?

[dedoogong]

Hi @bubbliiiing . I wonder there is a function of training validation like generating img/vid using the latest/currently trained weight.
I'm running z-image-fun controlnet 2.1 but it seems it doesn't generate any intermediate img/vid during training even in tensorboard like every N iteration.

1)is it true? if so how to do it? do I need to implement it to offload the currently trained latest model and run inference using the latest model(without training parameters) and load the trained model again?

2. It by default saves weights every 50 steps. why so oftenly? is it post training? others like diffusers/diffsysth studio usually save models by every 1000 or 5000 steps. do I need to keep it 50?

3. at 0 step, it generates "sanityi-check/cinematic.gif" and it looks like, it already generated quite nice results. is it a GT image read from a disk? or it is generated image?

thank you very much!

[bubbliiiing]

1、Yes, we don't validate this process by default. It's quite time-consuming, and we don't think it's necessary, so we basically don't use it.

2、The default parameter saves quite frequently, but this parameter can be configured. There's no need to save every 50 steps - we save frequently because our own testing resources might get interrupted.

3、The sanity-check is for your training data, used to see how the training data looks.

[dedoogong]

THANKS! for your kind reply,
one more thing.
do I need to merge the intermeditate trained weights(like diffusion_pytorch_model-00001-of-00002.safetensors, diffusion_pytorch_model-00002-of-00002.safetensors) by myself? or is there already a code for that??

the example i2v inference code shows it takes one single file(maybe after merge) like "transformer_path = "models/Personalized_Model/Z-Image-Turbo-Fun-Controlnet-Union-2.1.safetensors"

I think there is nothing for that, no README at all too.
thank you~!