Checkpointing - RAM Requirement?

[Grace-Chang2]

Hi,

For checkpointing, is there an equation or table for the minimum RAM required to run 10 checkpointing writes and 10 checkpointing reads?

I ran checkpointing with model=llama8b on a single 128GB client and during the read phase of the default behavior it hung due to RAM limitations. Was able to run the checkpointing workload with 2 clients at 128GB each (256GB RAM total)

Is there any resource pointing to the RAM needed for the llama70b, 405b, 1T models?
I know others have been able to run llama70b with 2TB RAM (4 clients with 512GB), and llama405b with 4TB RAM (16 clients with 256GB each)

Is there some equation for it? Would having more clients decrease the RAM needed for each server. For example could I have used 2 or 3 64GB clients to run the llama8b workload.

[zhenghh04]

You can check the table in this google doc for now: https://docs.google.com/document/d/1mnlHJ3_0N4bkz7i4InhkktAcVZPPklUN3TqW0_fIzYs/edit?usp=sharing

We will add info to the documentation later.

[Grace-Chang2]

Thank you @zhenghh04 !

I have a follow up question:

Essentially, I have tested the default write 10 read 10 checkpointing behavior.
Based on the minimum client memory equation, if I only used 1 server for llama8b, then the server would only need 105GB RAM.
M = 105GB / (8 / 8 processes) = 105GB
But my server has 128GB RAM and consistently breaks (run out of memory signal 9) when starting the first read load.

mlpstorage checkpointing run --client-host-memory-in-gb 128 --model llama3-8b --num-processes 8 --checkpoint-folder /mnt/pscale2/checkpoint_test --hosts 100.91.125.238

Does it really only need 105GB or 128GB?

[zhenghh04]

There might be some overhead. You can see whether it runs if you reduces the number of layers from 32 to 16, in your configuration file. @Grace-Chang2

Sorry for the late response.