GPU Failure Injection For LLM Training Framework Test

[APX103]

Hello, I am a development engineer of a LLM training framework.

I want to test the performance of my framework when it encounters a GPU fault.

Is there any "ready-made way" to help me do GPU fault injection~

If not, how can I achieve it? Can you show me a way?

[APX103]

help！！！