Speed up the first run.

[Robson1970]

How to boost up the first run up to 800% especially with hard drives.
Just copy the checkpoint from your HDD to the SDD, or make a copy itself in the same folder and then delete the "copy from the checkpoint".
Example: revAnimated_v122EOL.safetensors 5GByte takes 4 minutes to load the checkpoint compared to 30 sec. after copy the checkpoint from the HDD to the SSD and then delete the file.
ComfyUI start the checkpoint 8 times faster.

[NeedsMoar]

This isn't an issue with comfyui. 4 minute load times from a hard drive indicate an issue with the hard drive though.

[jn-jairo]

Just copy the checkpoint from your HDD to the SDD, or make a copy itself in the same folder and then delete the "copy from the checkpoint".

When you access a file the O.S. keeps a cache of it on RAM, so when you copy the file the O.S. loads it on RAM and when you try to access it later on ComfyUI it is already on RAM.

There is nothing to fix, if you want a faster transfer to RAM put the file on a faster drive.

[Robson1970]

jn-jairo

My config: 13900KS: 2x Red HDD 22Tb: 3xKingston KC3000 SSD 4TB : 64TB DDR5 7000MHz: RTX 4090
I keep my Ceckpoints on the HDD because over 18TB
Thats, what I mean, preload the checkpoint first and then extract the file and not extract the file from the drive.

[NeedsMoar]

Checkpoints should load in ~15s from those hard drives... I don't think they're quite 300MB/s but they're up there. That's kinda why I said something was wrong, 4 minutes is 1990s IDE drive speeds. The 18GB 15k Cheetahs I had in ~2001/2002 would have managed it in under a minute (if the computer had anywhere near enough ram to load a 5GB file back then anyway).

Copying a file to another drive will probably put it in cached memory as far as the opening program is concerned, but when you delete it, you're killing the file you loaded from so Windows (and anything else) has to invalidate the standby memory / cache (meaning it's freed as soon as comfy closes it rather than being kept around until you run out of ram.

Are you saying you have 18TB of checkpoints (actual) on one 22TB (advertised) drive? How much free space is it showing? You could be hitting MFT issues depending on cluster size.

[Robson1970]

Are you saying you have 18TB of checkpoints (actual) on one 22TB (advertised) drive? How much free space is it showing? You could be hitting MFT issues depending on cluster size.

1 Drive only for ComfyUI ,currently 2.18 TB free on the HDD. I can copy the 5 GB file Checkpoint from the HDD in around 30sec. on the Desktop SSD. Once the file is copied ComfyUI starts immediately with the checkpoint.
I made a little video from copy the file and ComfyUI with the"clear" Ram.

https://files.catbox.moe/9vze6v.mp4

[jn-jairo]

Thanks for the video, I misunderstood it, yeah there is something weird going on, even my potato laptop can load faster and I also store the checkpoints on HD because it has more space.

[Robson1970]

Test with Stable Diffusion, same HDD drive and same the checkpoint.
Model loaded in 305 sec. apply weights to model 298 sec.

Disable memmapping for loading .safetensors files.

New Test, load weights in 37sec. like 8 times faster.

[NeedsMoar]

Edit: I missed your post. That's the last thing I always forget that I hit with Nod.ai Shark on an NVMe. Python has an absolutely horrid implementation of memory mapped files on Windows because they sorta blindly tried to implement it as much like the linux version as possible. Nod wasn't that bad but couldn't even use models from standby because of the way they were loaded. I'd completely forgotten this was an issue sometimes. They never quite figured out what triggered it over there.

Yeah it kinda threw up an alarm, there aren't many possibilities here, I'll just go from best news to worst news.

1. Firstly, windows explorer displayed speeds don't really represent the speed the drive is reading / writing at the time. If there's enough ram to cache the file during copy it'll show you as fast a speed as possible and finish in the background to keep the UI responsive. You'll need to look at task manager or perfmon to really see what reads from that drive are doing. That drive may be slowly defragmenting in the background, or even worse compressing files if that's turned on, and delaying copies, it's hard to say.
   You can use sysinternals rammap or other tools to try to view what's going on with disk caches but I kinda don't think that's the issue since 200MB/s is about what I'd expect out of a mostly full current gen drive.

2. I don't know the arrangement of your motherboard but those Intels only have 20 PCIe lanes. It doesn't really matter how they allocated them, gen 4 NVMes will still eat up 12 lanes. Most of the time to give some appearance of things actually working motherboard manufacturers will throw a couple of onboard M.2 slots on the southbridge instead of putting them on the CPU and having them eat into the x16 GPU slot but it's hard to say. Two gen 4 NVMes will probably saturate the southbridge if they're both trying to do a bunch of work too, and that's where the SATA bandwidth needs to be. I'm not familiar with modern Intel CPUs, either, they may just have trouble spreading bandwidth around properly because of some weirdness of the performance core thing. Copying from stuff on the other side of the southbridge which is a PCIe switch to the GPU might be insanely more CPU-heavy than copying from an NVMe on the CPU side would be. It could also be AHCI turned off in bios for some reason, or a loose cable. It's hard to say.

3. The most likely suspect to me would be early drive failure. I haven't had any of the newer generation helium drives die on me yet but I'm not using anything but datacenter drives and the majority of them are SAS 12Gb/s which tend to be made better. You can end up with slowdowns from an NTFS disk being that full (Windows has problems defragmenting it properly or expanding the MFT) but that generally doesn't happen unless you formatted with a small cluster size and have millions of files, and not to the level you're seeing.

I'd strongly suggest checking event viewer for controller errors, and downloading smartctl and running extended self tests on both drives. SATA SMART isn't all that useful IMO but you should see it throwing checksum errors / read re-attempts if it's failing and that's something else to help troubleshoot. Everything else is outside the scope of the comfy issues page because there's obviously some kind of hardware issue going on.

[Robson1970]

@ NeedsMoar

Thanks for the detailed post and given me the hints.

I have same issue on my second drive with the safetensors.
ComfyUI load the .ckpt checkpoint around 10x faster than the .safetensors files.

It´s very strange, some safetensors files are faster than other safetensors files. Maybe there's an option in the safetensors how to handle the file ?

From the AUTOMATIC1111 there was a patch for loading the safetensors files.
AUTOMATIC1111/stable-diffusion-webui#11216
huggingface/safetensors#140

Preload the safetensors in the RAM should fix it.

[jn-jairo]

So, let's try the A1111 method, change this line

ComfyUI/comfy/utils.py

Line 13 in d9d8702

sd = safetensors.torch.load_file(ckpt, device=device.type)

to this

        sd = safetensors.torch.load(open(ckpt, 'rb').read())

and see if this solves your issue.

[Robson1970]

So, let's try the A1111 method, change this line

ComfyUI/comfy/utils.py

Line 13 in d9d8702

sd = safetensors.torch.load_file(ckpt, device=device.type)

to this

        sd = safetensors.torch.load(open(ckpt, 'rb').read())

and see if this solves your issue.

You are godlike! Master of scripts !

You saved me from buying 20TB of SSD.

The safetensors files load faster than I expected. The same checkpoint before load now in 29 sec. instead of 4min.
The 16GB safetensor load at >220GB/s - 280GB/s from the HDD in 76sec.

You are the time saver, the load time of safetensors are 8-10 faster now.
It's like christmas. Cheers, mate.

[freeAhao]

So, let's try the A1111 method, change this line

ComfyUI/comfy/utils.py

Line 13 in d9d8702

sd = safetensors.torch.load_file(ckpt, device=device.type)

to this

        sd = safetensors.torch.load(open(ckpt, 'rb').read())

and see if this solves your issue.

Huge Thanks!!!

[cheadrian]

So, let's try the A1111 method, change this line

ComfyUI/comfy/utils.py

Line 13 in d9d8702

sd = safetensors.torch.load_file(ckpt, device=device.type)

to this

        sd = safetensors.torch.load(open(ckpt, 'rb').read())

I can confirm this increased the speed of reading from the HDD x6 times, from about 15 MB/s to 120 MB/s.