Loss is not going down

[TerryMelody]

Hello dear authors! I trained ddim in another dataset. In 1200 epoches,, the loss still seems not going down constantly while sometimes loss became large. I wonder whether it is normal?

[heng0214]

I have the same problem with you. Have you already solved this problem？

[yoshi-taka-desu]

we have this problem, too.
Could somebody tell me how to solve or possible causes?

[ashah01]

This is a fairly standard phenomenon in diffusion training since the model samples timesteps uniformly in the forward pass. If the model has already seen and optimized a timestep several times, it will yield a lower loss score if presented with the same timestep later on compared to if it's presented with some other timestep it hasn't seen much before. One quick experiment you can implement to confirm this: try freezing the timestep vector and then train your model normally by iterating through your dataset. The loss should consistently go down.

[TerryMelody]

This is a fairly standard phenomenon in diffusion training since the model samples timesteps uniformly in the forward pass. If the model has already seen and optimized a timestep several times, it will yield a lower loss score if presented with the same timestep later on compared to if it's presented with some other timestep it hasn't seen much before. One quick experiment you can implement to confirm this: try freezing the timestep vector and then train your model normally by iterating through your dataset. The loss should consistently go down.

Got it! Thanks for your reply!