Latent upscale corrupting gaussian noise

[ntdviet]

I tried all the possible upscale modes and all of them are changing the noise into a square form, significantly reducing quality or changing the color tone. This is extremely annoying when you are trying to finetune the follow-up sampling step.
Is there any possibility to upscale the latent but kind of loss-less?
My current workaround is to VAE decode it to pixels and encode it back again, but in a bigger context, this is pretty time and memory consuming.