feat: automatically retry translate tasks when they can't find a GPU (#1058)

I had a translate task fail recently with:
```
[task 2025-03-08T05:47:06.300Z] + python3 /builds/worker/checkouts/vcs/pipeline/translate/translate.py --input /builds/worker/fetches/file.5.zst --models_glob '/builds/worker/fetches/*.npz' '/builds/worker/fetches/model*/*.npz' --artifacts /builds/worker/artifacts --vocab /builds/worker/fetches/vocab.spm --marian_dir '$MOZ_FETCHES_DIR' --gpus '0 1 2 3' --workspace 12000 --decoder ctranslate2 -- --maxi-batch 10000 --mini-batch-words 5000
[task 2025-03-08T05:47:06.633Z] [translate] Input file: /builds/worker/fetches/file.5.zst
[task 2025-03-08T05:47:06.633Z] [translate] Output file: /builds/worker/artifacts/file.5.out.zst
[task 2025-03-08T05:47:06.750Z] [translate_ctranslate2] Converting the Marian model to Ctranslate2:
[task 2025-03-08T05:47:06.750Z] [translate_ctranslate2] /builds/worker/fetches/model1/final.model.npz.best-chrf.npz
[task 2025-03-08T05:47:06.750Z] [translate_ctranslate2] Outputing model to:
[task 2025-03-08T05:47:06.750Z] [translate_ctranslate2] /builds/worker/fetches/model1/final.model.npz.best-chrf
[task 2025-03-08T05:47:06.758Z] [translate_ctranslate2] Loading vocab:
[task 2025-03-08T05:47:06.758Z] [translate_ctranslate2] /builds/worker/fetches/vocab.spm
[task 2025-03-08T05:47:12.792Z] Traceback (most recent call last):
[task 2025-03-08T05:47:12.792Z]   File "/builds/worker/checkouts/vcs/pipeline/translate/translate.py", line 256, in <module>
[task 2025-03-08T05:47:12.792Z]     main()
[task 2025-03-08T05:47:12.792Z]   File "/builds/worker/checkouts/vcs/pipeline/translate/translate.py", line 187, in main
[task 2025-03-08T05:47:12.792Z]     translate_with_ctranslate2(
[task 2025-03-08T05:47:12.792Z]   File "/builds/worker/checkouts/vcs/pipeline/translate/translate_ctranslate2.py", line 156, in translate_with_ctranslate2
[task 2025-03-08T05:47:12.792Z]     translator = ctranslate2.Translator(
[task 2025-03-08T05:47:12.792Z] RuntimeError: CUDA failed with error no CUDA-capable device is detected
```

We can and should automatically retry for these cases. This is similar to the work done in #722 for bicleaner.

Author: bhearsum

pipeline/translate/translate.py

@@ -7,6 +7,7 @@
 from glob import glob
 import os
 from pathlib import Path
+import sys
 import tempfile
 
 from pipeline.common.command_runner import apply_command_args, run_command
@@ -253,4 +254,12 @@ def main() -> None:
 
 
 if __name__ == "__main__":
-    main()
+    try:
+        main()
+    except RuntimeError as e:
+        # On GCP instances, we occasionally find that a GPU is not found even
+        # when it has been requested. Exiting with a unique error code in these
+        # cases allows us to automatically retry such tasks in Taskcluster.
+        if len(e.args) > 0 and "no CUDA-capable device is detected" in e.args[0]:
+            logger.exception("couldn't find GPU, exiting with 9002")
+            sys.exit(9002)

taskcluster/kinds/translate-corpus/kind.yml

@@ -103,7 +103,8 @@ tasks:
                 CUDNN_DIR: fetches/cuda-toolkit
                 MARIAN: $MOZ_FETCHES_DIR
             # 128 happens when cloning this repository fails
-            retry-exit-status: [128]
+            # 9002 happens if no GPU is attached
+            retry-exit-status: [128, 9002]
 
         run:
             using: run-task

taskcluster/kinds/translate-mono-src/kind.yml

@@ -103,7 +103,8 @@ tasks:
                 CUDNN_DIR: fetches/cuda-toolkit
                 MARIAN: $MOZ_FETCHES_DIR
             # 128 happens when cloning this repository fails
-            retry-exit-status: [128]
+            # 9002 happens if no GPU is attached
+            retry-exit-status: [128, 9002]
 
         marian-args:
             from-parameters: training_config.marian-args.decoding-teacher

taskcluster/kinds/translate-mono-trg/kind.yml

@@ -104,7 +104,8 @@ tasks:
                 CUDNN_DIR: fetches/cuda-toolkit
                 MARIAN: $MOZ_FETCHES_DIR
             # 128 happens when cloning this repository fails
-            retry-exit-status: [128]
+            # 9002 happens if no GPU is attached
+            retry-exit-status: [128, 9002]
 
         # Don't run unless explicitly scheduled
         run-on-tasks-for: []