175B Profiling

[azton]

Have functional implementation of deepspeed using zero3. Now need to analyze the performance and scalability--do we need pipeline/model parallelism to really run this?

[azton]

First trial; 32 Nodes increasing batch size until cuda OOM failures. Requires roughly 1-2 hours to run 3-5 steps for profiling.