feat: Add debugger flag that can be turned on via RAY_DEBUG=legacy (#1312)

Signed-off-by: Guyue Huang <[email protected]>
Signed-off-by: Guyue Huang <[email protected]>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>

Author: guyueh1

docs/debugging.md

@@ -68,10 +68,10 @@ Note that you can jump between breakpoints across all workers with this process.
 
 ![Ray Debugger Extension Step 4](./assets/ray-debug-step4.png)
 
-## Debug the Driver Script
+## Debug with legacy Ray debugger
 
-By default, setting breakpoints in the driver script (outside of  `@ray.remote`) will not pause program execution when using Ray. To enable pausing at these breakpoints, set the environment variable to `RAY_DEBUG=legacy`:
+To use legacy ray debugger, you can use two ways
+1. In general, set `RAY_DEBUG=legacy` and add `--ray-debugger-external` to your `ray start` command
+2. If you are using `ray.sub` in a slurm cluster, you can simply set `RAY_DEBUG=legacy` before `sbatch ray.sub`, the script can detect this environment variable and attach `--ray-debugger-external` automatically. 
 
-```sh
-RAY_DEBUG=legacy uv run ....
-```
+After you start ray with these changes, you can add `breakpoint` to your code. When you run the program, it will stop at where breakpoints are inserted. Then you can use a separate terminal to attach to the header node via `bash <JOB_ID>-attach.sh` (this script should automatically be generated by `ray.sub`), and run `ray debug` to see all the breakpoints. You can enter any breakpoint and interactively debug. Please refer to [Ray documentation](https://docs.ray.io/en/latest/ray-observability/user-guides/debug-apps/ray-debugging.html) for more info on this debugging approach.

ray.sub

@@ -64,6 +64,10 @@ RAY_CLIENT_SERVER_PORT=${RAY_CLIENT_SERVER_PORT:-10001}
 #REDIT_SHARD_PORTS=${REDIT_SHARD_PORTS:-"random"} ??
 DASHBOARD_PORT=${DASHBOARD_PORT:-8265}  # Also used by debugger
 DASHBOARD_AGENT_LISTEN_PORT=${DASHBOARD_AGENT_LISTEN_PORT:-52365}
+RAY_DEBUGGER_ARGS=
+if [ "${RAY_DEBUG:-}" = "legacy" ]; then
+  RAY_DEBUGGER_ARGS="--ray-debugger-external"
+fi
 
 # Setting ulimit is recommended by ray best practices page
 # @ https://docs.ray.io/en/latest/cluster/vms/user-guides/large-cluster-best-practices.html
@@ -276,8 +280,9 @@ ray start --head \
     --dashboard-agent-grpc-port=$((${DASHBOARD_AGENT_GRPC_PORT} + 1)) \
     --dashboard-agent-listen-port=$((${DASHBOARD_AGENT_LISTEN_PORT} + 1)) \
     --metrics-export-port=$((${METRICS_EXPORT_PORT} + 1)) \
+    $RAY_DEBUGGER_ARGS \
     \
-    --block
+    --block 
 EOFINNER
 chmod +x /launch-head.sh
 
@@ -375,8 +380,9 @@ ray start --address "$ip_head" \
           --dashboard-agent-grpc-port=${DASHBOARD_AGENT_GRPC_PORT} \
           --dashboard-agent-listen-port=${DASHBOARD_AGENT_LISTEN_PORT} \
           --metrics-export-port=${METRICS_EXPORT_PORT} \
+          $RAY_DEBUGGER_ARGS \
           \
-          --block
+          --block 
 EOFINNER
 
 count=0