Skip to content

Commit 2946e04

Browse files
authored
ci: limit aot parallel build jobs based on available memory (#1612)
1 parent efe55ee commit 2946e04

File tree

1 file changed

+11
-1
lines changed

1 file changed

+11
-1
lines changed

scripts/task_test_aot_build_import.sh

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,17 @@
22

33
set -eo pipefail
44
set -x
5-
: ${MAX_JOBS:=$(nproc)}
5+
6+
# MAX_JOBS = min(nproc, max(1, MemAvailable_GB/4))
7+
MEM_AVAILABLE_GB=$(free -g | awk '/^Mem:/ {print $7}')
8+
NPROC=$(nproc)
9+
MAX_JOBS=$(( MEM_AVAILABLE_GB / 4 ))
10+
if (( MAX_JOBS < 1 )); then
11+
MAX_JOBS=1
12+
elif (( NPROC < MAX_JOBS )); then
13+
MAX_JOBS=$NPROC
14+
fi
15+
616
: ${CUDA_VISIBLE_DEVICES:=""}
717
export TORCH_CUDA_ARCH_LIST=$(python3 -c '
818
import torch

0 commit comments

Comments
 (0)