[QUESTION] THD format with CP

[wplf]

Your question
Ask a clear and concise question about Megatron-LM.

Thank you for great works!

I'd like to know if you have a Unit test about thd format with cp?

[yaox12]

Yes, we have the test in TE https://github.com/NVIDIA/TransformerEngine/blob/4ddb0a7bea787294282d0fe0715adf5ea4a39779/tests/pytorch/fused_attn/test_fused_attn_with_cp.py#L47.

[wplf]

Yes，thank you very much. I've seen that before.
However, I don't see CP and thd support in Megatron-LM, is that right?