LAUNCH ERROR Device not found

[CarlosAmaral]

Dear community,

I'm trying out paddlex and I'm trying to run an example here: https://paddlepaddle.github.io/PaddleX/latest/en/practical_tutorials/ocr_det_license_tutorial.html#41-data-preparation

I'm running it on mac m1 and I'm getting the following output after running python3 main.py -c paddlex/configs/text_detection/PP-OCRv4_server_det.yaml \ -o Global.mode=train \ -o Global.dataset_dir=./dataset/ccpd_text_det

Output:

python3 main.py -c paddlex/configs/text_detection/PP-OCRv4_server_det.yaml \
    -o Global.mode=train \
    -o Global.dataset_dir=./dataset/ccpd_text_det
/Users/path/PaddleX/env/lib/python3.9/site-packages/urllib3/__init__.py:35: NotOpenSSLWarning: urllib3 v2 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with 'LibreSSL 2.8.3'. See: https://github.com/urllib3/urllib3/issues/3020
  warnings.warn(
/Users/path/PaddleX/env/lib/python3.9/site-packages/paddle/utils/cpp_extension/extension_utils.py:686: UserWarning: No ccache found. Please be aware that recompiling all source files may be required. You can download and install ccache from: https://github.com/ccache/ccache/blob/master/doc/INSTALL.md
  warnings.warn(warning_message)
['/Users/path/PaddleX/env/bin/python3', '-m', 'paddle.distributed.launch', '--devices', '0,1,2,3', '--log_dir', '/Users/path/PaddleX/output/distributed_train_logs', 'tools/train.py', '-c', '/Users/path/.paddlex/tmpcsnfat8f/textdetmodel_PP-OCRv4_server_det.yml']

Log path: /Users/path/PaddleX/output/train.log 

/Users/path/PaddleX/env/lib/python3.9/site-packages/paddle/utils/cpp_extension/extension_utils.py:686: UserWarning: No ccache found. Please be aware that recompiling all source files may be required. You can download and install ccache from: https://github.com/ccache/ccache/blob/master/doc/INSTALL.md
  warnings.warn(warning_message)
LAUNCH ERROR 2025-01-21 23:08:53,838 Device not found 0 from 0,1,2,3 for setting 
LAUNCH ERROR 2025-01-21 23:08:53,838 Device not found 1 from 0,1,2,3 for setting 
LAUNCH ERROR 2025-01-21 23:08:53,838 Device not found 2 from 0,1,2,3 for setting 
LAUNCH ERROR 2025-01-21 23:08:53,838 Device not found 3 from 0,1,2,3 for setting 
LAUNCH INFO 2025-01-21 23:08:53,840 -----------  Configuration  ----------------------
LAUNCH INFO 2025-01-21 23:08:53,840 auto_cluster_config: 0
LAUNCH INFO 2025-01-21 23:08:53,840 auto_parallel_config: None
LAUNCH INFO 2025-01-21 23:08:53,840 auto_tuner_json: None
LAUNCH INFO 2025-01-21 23:08:53,840 devices: 0,1,2,3
LAUNCH INFO 2025-01-21 23:08:53,840 elastic_level: -1
LAUNCH INFO 2025-01-21 23:08:53,840 elastic_timeout: 30
LAUNCH INFO 2025-01-21 23:08:53,840 enable_gpu_log: True
LAUNCH INFO 2025-01-21 23:08:53,840 gloo_port: 6767
LAUNCH INFO 2025-01-21 23:08:53,840 host: None
LAUNCH INFO 2025-01-21 23:08:53,840 ips: None
LAUNCH INFO 2025-01-21 23:08:53,840 job_id: default
LAUNCH INFO 2025-01-21 23:08:53,840 legacy: False
LAUNCH INFO 2025-01-21 23:08:53,840 log_dir: /Users/path/PaddleX/output/distributed_train_logs
LAUNCH INFO 2025-01-21 23:08:53,840 log_level: INFO
LAUNCH INFO 2025-01-21 23:08:53,840 log_overwrite: False
LAUNCH INFO 2025-01-21 23:08:53,840 master: None
LAUNCH INFO 2025-01-21 23:08:53,840 max_restart: 3
LAUNCH INFO 2025-01-21 23:08:53,840 nnodes: 1
LAUNCH INFO 2025-01-21 23:08:53,840 nproc_per_node: None
LAUNCH INFO 2025-01-21 23:08:53,840 rank: -1
LAUNCH INFO 2025-01-21 23:08:53,840 run_mode: collective
LAUNCH INFO 2025-01-21 23:08:53,840 server_num: None
LAUNCH INFO 2025-01-21 23:08:53,841 servers: 
LAUNCH INFO 2025-01-21 23:08:53,841 sort_ip: False
LAUNCH INFO 2025-01-21 23:08:53,841 start_port: 6070
LAUNCH INFO 2025-01-21 23:08:53,841 trainer_num: None
LAUNCH INFO 2025-01-21 23:08:53,841 trainers: 
LAUNCH INFO 2025-01-21 23:08:53,841 training_script: tools/train.py
LAUNCH INFO 2025-01-21 23:08:53,841 training_script_args: ['-c', '/Users/path/.paddlex/tmpcsnfat8f/textdetmodel_PP-OCRv4_server_det.yml']
LAUNCH INFO 2025-01-21 23:08:53,841 with_gloo: 1
LAUNCH INFO 2025-01-21 23:08:53,841 --------------------------------------------------
LAUNCH INFO 2025-01-21 23:08:53,841 Job: default, mode collective, replicas 1[1:1], elastic False
Traceback (most recent call last):
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/Users/path/PaddleX/env/lib/python3.9/site-packages/paddle/distributed/launch/__main__.py", line 17, in <module>
    launch()
  File "/Users/path/PaddleX/env/lib/python3.9/site-packages/paddle/distributed/launch/main.py", line 1362, in launch
    c.run()
  File "/Users/path/PaddleX/env/lib/python3.9/site-packages/paddle/distributed/launch/controllers/controller.py", line 81, in run
    self.build_pod()
  File "/Users/path/PaddleX/env/lib/python3.9/site-packages/paddle/distributed/launch/controllers/collective.py", line 51, in build_pod
    return self._build_pod_with_master(skip_run)
  File "/Users/path/PaddleX/env/lib/python3.9/site-packages/paddle/distributed/launch/controllers/collective.py", line 217, in _build_pod_with_master
    selected_dev_list = self.ctx.node.device.get_selected_devices(
  File "/Users/path/PaddleX/env/lib/python3.9/site-packages/paddle/distributed/launch/context/device.py", line 85, in get_selected_devices
    return [str(self._labels.index(d)) for d in devs]
  File "/Users/path/PaddleX/env/lib/python3.9/site-packages/paddle/distributed/launch/context/device.py", line 85, in <listcomp>
    return [str(self._labels.index(d)) for d in devs]
ValueError: substring not found
Traceback (most recent call last):
  File "/Users/path/PaddleX/paddlex/utils/result_saver.py", line 29, in wrap
    result = func(self, *args, **kwargs)
  File "/Users/path/PaddleX/paddlex/engine.py", line 41, in run
    self._model.train()
  File "/Users/path/PaddleX/paddlex/model.py", line 94, in train
    trainer.train()
  File "/Users/path/PaddleX/paddlex/modules/base/trainer.py", line 71, in train
    train_result = self.pdx_model.train(**train_args)
  File "/Users/path/PaddleX/paddlex/repo_apis/PaddleOCR_api/text_rec/model.py", line 146, in train
    return self.runner.train(
  File "/Users/path/PaddleX/paddlex/repo_apis/PaddleOCR_api/text_rec/runner.py", line 56, in train
    return self.run_cmd(
  File "/Users/path/PaddleX/paddlex/repo_apis/base/runner.py", line 355, in run_cmd
    raise CalledProcessError(
paddlex.utils.errors.others.CalledProcessError: Command ['/Users/path/PaddleX/env/bin/python3', '-m', 'paddle.distributed.launch', '--devices', '0,1,2,3', '--log_dir', '/Users/path/PaddleX/output/distributed_train_logs', 'tools/train.py', '-c', '/Users/path/.paddlex/tmpcsnfat8f/textdetmodel_PP-OCRv4_server_det.yml'] returned non-zero exit status 1.


I'd greatly appreciate help on this, thank you!

[GreatV]

The error LAUNCH ERROR Device not found indicates that the PaddlePaddle framework is trying to use GPU devices (0,1,2,3) that are not available or compatible with your system. Since you're running this on a Mac with an M1 chip, there are a few key points to address:

---

Root Cause

1. GPU Devices Not Present: The error log shows that PaddlePaddle is attempting to use GPU devices (0,1,2,3), but your Mac M1 does not have discrete GPUs compatible with CUDA. Instead, it uses the Apple Metal framework for GPU acceleration, which is currently supported in PaddlePaddle only with specific configurations.

2. Incorrect Launch Configuration: The training script is being executed in distributed training mode (paddle.distributed.launch), which is trying to distribute tasks across multiple GPU devices. Since these devices are not available on your system, the script fails.

3. Dependency Issues: The warning about LibreSSL and ccache suggests that there might be some additional environment setup issues, though they are not critical for resolving this specific error.

---

Solution

1. Run in CPU Mode

Since your Mac M1 does not support CUDA GPUs, you need to explicitly run PaddlePaddle in CPU mode. Modify your command to disable GPU usage by setting the environment variable CUDA_VISIBLE_DEVICES to an empty string. Run the following command:

CUDA_VISIBLE_DEVICES="" python3 main.py -c paddlex/configs/text_detection/PP-OCRv4_server_det.yaml \
    -o Global.mode=train \
    -o Global.dataset_dir=./dataset/ccpd_text_det

This ensures that the training script will only utilize the CPU.

---

2. Install PaddlePaddle with M1 Support

PaddlePaddle does support the Apple M1 chip with the Metal backend. Ensure that you have installed the correct version of PaddlePaddle for your Mac M1. Use the following installation command:

pip install paddlepaddle-mac -i https://mirror.baidu.com/pypi/simple

This version is optimized for the M1 architecture and uses the Metal Performance Shaders (MPS) backend for GPU acceleration.

---

3. Avoid Distributed Training

The paddle.distributed.launch module is unnecessary for single-node or single-device training. Instead, directly use the training script. Modify your command to:

python3 tools/train.py -c paddlex/configs/text_detection/PP-OCRv4_server_det.yaml \
    -o Global.mode=train \
    -o Global.dataset_dir=./dataset/ccpd_text_det

This bypasses the distributed launch module and runs the training script directly.

---

4. Use Compatible PaddlePaddle Configurations

The configuration file PP-OCRv4_server_det.yaml might contain settings specific to GPU training. Open the file and check for the following settings:

• Ensure that the use_gpu option is set to False:

  use_gpu: False

• Make sure the batch size is small enough for CPU training (e.g., batch_size: 4).

---

Additional Notes

• Warnings about LibreSSL and ccache: These warnings indicate minor issues in your environment but are unrelated to the main error. However, you can address them by:

  • Upgrading OpenSSL to a supported version.
  • Installing ccache to speed up recompilation:

    brew install ccache

• Performance on M1: Training on an M1 chip might be slower compared to systems with NVIDIA GPUs. Consider using pre-trained models or cloud-based GPU instances if training time is a concern.

---

References

If the above steps do not resolve your issue, consider looking at related discussions or issues:

• PaddleOCR Training Guide
• PaddlePaddle Issues and Discussions

You can also refer to the discussion linked earlier (Discussion #14069) if you encounter similar issues like Illegal Instruct (core dumped) on unsupported architectures.

---

By following these steps, you should be able to resolve the Device not found error and successfully train the model on your Mac M1 system. Let me know if you need further assistance!

---

Response generated by feifei-bot | chatgpt-4o-latest