Make stable validation loss stable and deterministic for Flux when using uncached latents

[mturnshek]

Seed wasn't being properly managed for latents generated on the accelerator and stable timestep selection (200, 400, 600, 800) was not handled by the flux sampling function.

Before:

After:

[mturnshek]

I have noticed something else having to do with loss in sd-scripts that I am confused by. This class:

class LossRecorder:
    def __init__(self):
        self.loss_list: List[float] = []
        self.loss_total: float = 0.0

    def add(self, *, epoch: int, step: int, loss: float) -> None:
        if epoch == 0:
            self.loss_list.append(loss)
        else:
            while len(self.loss_list) <= step:
                self.loss_list.append(0.0)
            self.loss_total -= self.loss_list[step]
            self.loss_list[step] = loss
        self.loss_total += loss

    @property
    def moving_average(self) -> float:
        losses = len(self.loss_list)
        if losses == 0:
            return 0
        return self.loss_total / losses

We use the moving_average property for logging loss/validation/step_average. However, I don't understand why we would use that for validation loss especially. It is counterproductive for past validation loss to affect current validation loss.

On my personal fork I have added:

def reset(self) -> None:
        self.loss_list = []
        self.loss_total = 0.0

and call it between validation checks as a stopgap solution, but there is certainly a more robust way to do it.