How to do disk offloading? #882
Unanswered
TomLucidor
asked this question in
Q&A
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
Assuming that a whole model can fit on memory, BUT that the KV cache generated is likely not able to fit along side it, is it possible to use the disk as a kind of backup to help add capacity?
Beta Was this translation helpful? Give feedback.
All reactions