Loading Models from External Storage; How Fast is Fast Enough? #254
CallMeAl1973
started this conversation in
General
Replies: 3 comments
-
I do it and there is no impact at performance, just the initial load of the model to ram that is slower. |
Beta Was this translation helpful? Give feedback.
0 replies
-
Awesome. Thanks! What sort of external drive are you using? |
Beta Was this translation helpful? Give feedback.
0 replies
-
A common SanDisk USB 3.0 pendrive. |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
I'm toying with the idea of purchasing an external SSD and attempting to run models from there. Curious how much of a performance hit I could expect to take? I'm currently running models from internal storage, an NVME SSD, and typically get around 1.5 tokens per second using 30B models.
My machine has a USB 3.2 Gen 2 Type-C port, but I doubt any budget SSDs will come anywhere close to taking full advantage of that. I was considering something like this:
https://www.storagereview.com/review/adata-se880-portable-ssd-review
I'm sure i'll take theoretical performance hit, but I'mcurious approximately how much of one, and would it be noticeable?
Thanks,
--Al
Beta Was this translation helpful? Give feedback.
All reactions