CUDA available but not used? #732

breandan · 2021-03-07T05:09:32Z

breandan
Mar 7, 2021

Since the deployment environment does not support internet connectivity, I built a fat JAR using the following deps:

implementation("ai.djl:api:0.10.0")
implementation("org.slf4j:slf4j-simple:1.7.30")
implementation("ai.djl.mxnet:mxnet-engine:0.10.0")
implementation("ai.djl.mxnet:mxnet-native-cu102mkl:1.7.0-backport")

It uses the GPU on my local development machine, but after deploying, DJL falls back to using the CPU:

$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Wed_Oct_23_19:24:38_PDT_2019
Cuda compilation tools, release 10.2, V10.2.89

$ java -jar fatjar.jar
[main] INFO ai.djl.training.listener.LoggingTrainingListener - Training on: cpu().
[main] INFO ai.djl.training.listener.LoggingTrainingListener - Load MXNet Engine Version 1.7.0 in 0.054 ms.

Same version of CUDA on both machines. Any ideas what I might be doing wrong?

frankfliu · 2021-03-07T06:20:24Z

frankfliu
Mar 7, 2021

Can you enable debug level, it will print more information regard engine loading?

Anther way to debug is add the following code at beginning of your code:

    Engine.debugEnvironment();

4 replies

breandan Mar 7, 2021
Author

Below is the output from the remote environment when I enable debugging:

Click to expand!

[breandan@blg4102 RL-FlappyBird]$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Wed_Oct_23_19:24:38_PDT_2019
Cuda compilation tools, release 10.2, V10.2.

[breandan@blg4102 RL-FlappyBird]$ java -cp target/RL-FlappyBird-1.0-SNAPSHOT-jar-with-dependencies.jar com.kingyu.rlbird.ai.TrainBird
----------- System Properties -----------
java.specification.version: 13
sun.management.compiler: HotSpot 64-Bit Tiered Compilers
sun.jnu.encoding: UTF-8
java.runtime.version: 13.0.1+9
java.class.path: target/RL-FlappyBird-1.0-SNAPSHOT-jar-with-dependencies.jar
user.name: breandan
java.vm.vendor: Oracle Corporation
path.separator: :
sun.arch.data.model: 64
os.version: 3.10.0-1127.19.1.el7.x86_64
java.runtime.name: OpenJDK Runtime Environment
file.encoding: UTF-8
java.vendor.url: https://java.oracle.com/
jnidispatch.path: /home/breandan/.cache/JNA/temp/jna14787945592496677511.tmp
java.vm.name: OpenJDK 64-Bit Server VM
java.vm.specification.version: 13
os.name: Linux
jna.platform.library.path: /usr/lib64:/lib64:/usr/lib:/lib:/usr/lib64/mysql:/usr/lib64//bind9-export:/opt/dell/srvadmin/lib64
sun.java.launcher: SUN_STANDARD
user.country: US
sun.boot.library.path: /cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/java/13.0.1/lib
jna.loaded: true
sun.java.command: com.kingyu.rlbird.ai.TrainBird
java.vendor.url.bug: https://bugreport.java.com/bugreport/
java.io.tmpdir: /tmp
jdk.debug: release
sun.cpu.endian: little
java.version: 13.0.1
user.home: /home/breandan
user.dir: /lustre03/project/6025349/breandan/RL-FlappyBird
os.arch: amd64
user.language: en
java.specification.vendor: Oracle Corporation
java.vm.specification.name: Java Virtual Machine Specification
java.version.date: 2019-10-15
java.home: /cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/java/13.0.1
file.separator: /
java.vm.compressedOopsMode: Zero based
line.separator: 

java.library.path: /usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib
java.vm.info: mixed mode, sharing
java.vm.specification.vendor: Oracle Corporation
java.specification.name: Java Platform API Specification
java.vendor: Oracle Corporation
java.vm.version: 13.0.1+9
sun.io.unicode.encoding: UnicodeLittle
java.class.version: 57.0

--------- Environment Variables ---------
PATH: /cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/java/13.0.1:/cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/java/13.0.1/bin:/cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/cudacore/10.2.89:/cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/cudacore/10.2.89/nvvm/bin:/cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/cudacore/10.2.89/bin:/opt/software/slurm/bin:/opt/software/slurm/bin:/cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/maven/3.5.0:/cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/maven/3.5.0/bin:/cvmfs/soft.computecanada.ca/easybuild/software/2017/avx512/Compiler/intel2018.3/openmpi/3.1.2/bin:/cvmfs/restricted.computecanada.ca/easybuild/software/2017/Core/ifort/2018.3.222/compilers_and_libraries_2018.3.222/linux/bin/intel64:/cvmfs/restricted.computecanada.ca/easybuild/software/2017/Core/icc/2018.3.222/compilers_and_libraries_2018.3.222/linux/bin/intel64:/cvmfs/soft.computecanada.ca/nix/var/nix/profiles/gcc-7.3.0/bin:/cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/imkl/2018.3.222/mkl/bin:/cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/imkl/2018.3.222/bin:/cvmfs/soft.computecanada.ca/custom/bin/computecanada:/cvmfs/soft.computecanada.ca/easybuild/bin:/cvmfs/soft.computecanada.ca/custom/bin:/cvmfs/soft.computecanada.ca/nix/var/nix/profiles/16.09/bin:/cvmfs/soft.computecanada.ca/nix/var/nix/profiles/16.09/sbin:/opt/software/slurm/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/puppetlabs/bin:/home/breandan/.local/bin:/home/breandan/bin
PKG_CONFIG_PATH: /cvmfs/soft.computecanada.ca/easybuild/software/2017/avx512/Compiler/intel2018.3/openmpi/3.1.2/lib/pkgconfig:/cvmfs/soft.computecanada.ca/nix/var/nix/profiles/16.09/lib/pkgconfig:/cvmfs/soft.computecanada.ca/nix/var/nix/profiles/16.09/share/pkgconfig
SLURM_LAUNCH_NODE_IPADDR: 10.74.73.5
LMOD_AVAIL_EXTENSIONS: no
LMOD_SETTARG_FULL_SUPPORT: no
INTEL_LICENSE_FILE: /cvmfs/soft.computecanada.ca/config/licenses/intel/beluga.lic
_LMFILES_: /cvmfs/soft.computecanada.ca/custom/modules/nixpkgs/16.09.lua:/cvmfs/soft.computecanada.ca/easybuild/modules/2017/Core/imkl/2018.3.222.lua:/cvmfs/soft.computecanada.ca/easybuild/modules/2017/Core/gcccore/.7.3.0.lua:/cvmfs/soft.computecanada.ca/easybuild/modules/2017/Core/icc/.2018.3.222.lua:/cvmfs/soft.computecanada.ca/easybuild/modules/2017/Core/ifort/.2018.3.222.lua:/cvmfs/soft.computecanada.ca/easybuild/modules/2017/Core/intel/2018.3.lua:/cvmfs/soft.computecanada.ca/easybuild/modules/2017/avx512/Compiler/intel2018.3/openmpi/3.1.2.lua:/cvmfs/soft.computecanada.ca/custom/modules/StdEnv/2018.3.lua:/cvmfs/soft.computecanada.ca/easybuild/modules/2017/Core/maven/3.5.0.lua:/cvmfs/soft.computecanada.ca/easybuild/modules/2017/Core/cudacore/.10.2.89.lua:/cvmfs/soft.computecanada.ca/easybuild/modules/2017/avx512/Compiler/intel2018.3/cuda/10.2.lua:/cvmfs/soft.computecanada.ca/easybuild/modules/2017/Core/java/13.0.1.lua
SLURM_TOPOLOGY_ADDR: ibsw40.ibsw30.ibsw18.blg4102
EBDEVELCUDA: /cvmfs/soft.computecanada.ca/easybuild/software/2017/avx512/Compiler/intel2018.3/cuda/10.2/easybuild/avx512-Compiler-intel2018.3-cuda-10.2-easybuild-devel
SLURM_SRUN_COMM_PORT: 33744
EBDEVELGCCCORE: /cvmfs/soft.computecanada.ca/nix/var/nix/profiles/gcc-7.3.0/easybuild/Core-gcccore-.7.3.0-easybuild-devel
OMPI_MCA_mtl: ^mxm
RSNT_SLURM_MPI_TYPE: pmix_v3
LMOD_VERSION: 8.4.16
LESSOPEN: |/cvmfs/soft.computecanada.ca/nix/var/nix/profiles/16.09/bin/lesspipe.sh %s
SLURM_UMASK: 0027
SLURM_SUBMIT_DIR: /lustre03/project/6025349/breandan/RL-FlappyBird
__LMOD_STACK_LESSOPEN: fC9jdm1mcy9zb2Z0LmNvbXB1dGVjYW5hZGEuY2Evbml4L3Zhci9uaXgvcHJvZmlsZXMvMTYuMDkvYmluL2xlc3NwaXBlLnNoICVz:fHwvdXNyL2Jpbi9sZXNzcGlwZS5zaCAlcw==
EASYBUILD_BUILDPATH: /tmp/breandan
LMOD_FAMILY_BASE_OS: nixpkgs
MKL_EXAMPLES: /cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/imkl/2018.3.222/mkl/examples/
EBDEVELIFORT: /cvmfs/restricted.computecanada.ca/easybuild/software/2017/Core/ifort/2018.3.222/easybuild/Core-ifort-.2018.3.222-easybuild-devel
SLURM_CLUSTER_NAME: beluga
EASYBUILD_CONFIGFILES: /cvmfs/soft.computecanada.ca/easybuild/config.cfg
EBVERSIONCUDACORE: 10.2.89
SLURM_STEP_ID: 0
ARCH: x86_64
EBDEVELICCIFORT: /cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/intel/2018.3/easybuild/Core-intel-2018.3-easybuild-devel
TERM: xterm-256color
XDG_CONFIG_DIRS: /cvmfs/soft.computecanada.ca/nix/var/nix/profiles/16.09/etc/xdg:/etc/xdg
SELINUX_LEVEL_REQUESTED: 
SLURM_JOB_ID: 16829331
_ModuleTable009_: dGVsMjAxOC4zL2N1ZGExMC4yIiwiL2N2bWZzL3NvZnQuY29tcHV0ZWNhbmFkYS5jYS9lYXN5YnVpbGQvbW9kdWxlcy8yMDE3L0NVREEvY3VkYTEwLjIiLCIvY3ZtZnMvc29mdC5jb21wdXRlY2FuYWRhLmNhL2Vhc3lidWlsZC9tb2R1bGVzLzIwMTcvYXZ4NTEyL01QSS9pbnRlbDIwMTguMy9vcGVubXBpMy4xIiwiL2N2bWZzL3NvZnQuY29tcHV0ZWNhbmFkYS5jYS9lYXN5YnVpbGQvbW9kdWxlcy8yMDE3L2F2eDUxMi9Db21waWxlci9pbnRlbDIwMTguMyIsIi9ob21lL2JyZWFuZGFuLy5sb2NhbC9lYXN5YnVpbGQvbW9kdWxlcy8yMDE3L0NvcmUiLCIvY3ZtZnMvc29mdC5jb21wdXRlY2FuYWRhLmNhL2Vhc3lidWlsZC9tb2R1bGVzLzIwMTcvQ29yZSIsIi9jdm1mcy9zb2Z0LmNv
SLURM_TMPDIR: /localscratch/breandan.16829331.0
_ModuleTable_Sz_: 10
__LMOD_REF_COUNT_MODULEPATH: /opt/software/modulefiles:3;/cvmfs/soft.computecanada.ca/easybuild/modules/2017/avx512/CUDA/intel2018.3/cuda10.2:1;/cvmfs/soft.computecanada.ca/easybuild/modules/2017/CUDA/cuda10.2:1;/cvmfs/soft.computecanada.ca/easybuild/modules/2017/avx512/MPI/intel2018.3/openmpi3.1:1;/cvmfs/soft.computecanada.ca/easybuild/modules/2017/avx512/Compiler/intel2018.3:1;/home/breandan/.local/easybuild/modules/2017/Core:1;/cvmfs/soft.computecanada.ca/easybuild/modules/2017/Core:1;/cvmfs/soft.computecanada.ca/custom/modules:1
LMOD_ADMIN_FILE: /cvmfs/soft.computecanada.ca/config/lmod/admin.list
SLURM_TASK_PID: 62893
PIP_CONFIG_FILE: /cvmfs/soft.computecanada.ca/config/python/pip-avx512.conf
EBDEVELJAVA: /cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/java/13.0.1/easybuild/Core-java-13.0.1-easybuild-devel
SLURM_TASKS_PER_NODE: 1
SLURM_JOB_UID: 3067255
LMOD_FAMILY_COMPILER: intel
_: /cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/java/13.0.1/bin/java
LMOD_FAMILY_COMPILER_VERSION: 2018.3
LMOD_FAMILY_BASE_OS_VERSION: 16.09
SACCT_FORMAT: Account,User,JobID,Start,End,AllocCPUS,Elapsed,AllocTRES%30,CPUTime,AveRSS,MaxRSS,MaxRSSTask,MaxRSSNode,NodeList,ExitCode,State%20
SSL_CERT_FILE: /etc/pki/tls/certs/ca-bundle.crt
EBDEVELCUDACORE: /cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/cudacore/10.2.89/easybuild/Core-cudacore-.10.2.89-easybuild-devel
EBDEVELOPENMPI: /cvmfs/soft.computecanada.ca/easybuild/software/2017/avx512/Compiler/intel2018.3/openmpi/3.1.2/easybuild/avx512-Compiler-intel2018.3-openmpi-3.1.2-easybuild-devel
EBDEVELMAVEN: /cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/maven/3.5.0/easybuild/Core-maven-3.5.0-easybuild-devel
FONTCONFIG_FILE: /cvmfs/soft.computecanada.ca/nix/var/nix/profiles/16.09/etc/fonts/fonts.conf
BASH_FUNC_ml%%: () {  eval $($LMOD_DIR/ml_cmd "$@")
}
ACLOCAL_PATH: /cvmfs/soft.computecanada.ca/nix/var/nix/profiles/16.09/share/aclocal
RSNT_INTERCONNECT: infiniband
_ModuleTable008_: ZSJdPSJuaXhwa2dzLzE2LjA5Iix9LG9wZW5tcGk9e1siZm4iXT0iL2N2bWZzL3NvZnQuY29tcHV0ZWNhbmFkYS5jYS9lYXN5YnVpbGQvbW9kdWxlcy8yMDE3L2F2eDUxMi9Db21waWxlci9pbnRlbDIwMTguMy9vcGVubXBpLzMuMS4yLmx1YSIsWyJmdWxsTmFtZSJdPSJvcGVubXBpLzMuMS4yIixbImxvYWRPcmRlciJdPTcscHJvcFQ9e3R5cGVfPXtbIm1waSJdPTEsfSx9LFsic3RhY2tEZXB0aCJdPTEsWyJzdGF0dXMiXT0iYWN0aXZlIixbInVzZXJOYW1lIl09Im9wZW5tcGkvMy4xLjIiLH0sfSxtcGF0aEE9eyIvb3B0L3NvZnR3YXJlL21vZHVsZWZpbGVzIiwiL2N2bWZzL3NvZnQuY29tcHV0ZWNhbmFkYS5jYS9lYXN5YnVpbGQvbW9kdWxlcy8yMDE3L2F2eDUxMi9DVURBL2lu
EBROOTMAVEN: /cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/maven/3.5.0
EBROOTNIXPKGS: /cvmfs/soft.computecanada.ca/nix/var/nix/profiles/16.09
SLURM_JOB_NAME: sh
TZ: EST5EDT
EBROOTICC: /cvmfs/restricted.computecanada.ca/easybuild/software/2017/Core/icc/2018.3.222
CC_CLUSTER: beluga
EBVERSIONJAVA: 13.0.1
_ModuleTable007_: YS5jYS9lYXN5YnVpbGQvbW9kdWxlcy8yMDE3L0NvcmUvbWF2ZW4vMy41LjAubHVhIixbImZ1bGxOYW1lIl09Im1hdmVuLzMuNS4wIixbImxvYWRPcmRlciJdPTkscHJvcFQ9e3R5cGVfPXtbInRvb2xzIl09MSx9LH0sWyJzdGFja0RlcHRoIl09MCxbInN0YXR1cyJdPSJhY3RpdmUiLFsidXNlck5hbWUiXT0ibWF2ZW4iLH0sbml4cGtncz17WyJmbiJdPSIvY3ZtZnMvc29mdC5jb21wdXRlY2FuYWRhLmNhL2N1c3RvbS9tb2R1bGVzL25peHBrZ3MvMTYuMDkubHVhIixbImZ1bGxOYW1lIl09Im5peHBrZ3MvMTYuMDkiLFsibG9hZE9yZGVyIl09MSxwcm9wVD17bG1vZD17WyJzdGlja3kiXT0xLH0sfSxbInN0YWNrRGVwdGgiXT0xLFsic3RhdHVzIl09ImFjdGl2ZSIsWyJ1c2VyTmFt
SLURM_MPI_TYPE: pmix_v3
FPATH: /cvmfs/soft.computecanada.ca/custom/software/lmod/lmod/init/ksh_funcs
SLURM_JOB_QOS: normal
SLURM_STEP_NODELIST: blg4102
SLURM_STEPID: 0
LOGNAME: breandan
SLURM_GTIDS: 0
SLURM_JOB_CPUS_PER_NODE: 1
SLURM_STEP_GPUS: 0
CURL_CA_BUNDLE: /etc/pki/tls/certs/ca-bundle.crt
SLURM_NODELIST: blg4102
LMOD_CMD: /cvmfs/soft.computecanada.ca/custom/software/lmod/lmod/libexec/lmod
SHELL: /bin/bash
BASH_ENV: /cvmfs/soft.computecanada.ca/custom/software/lmod/lmod/init/bash
OLDPWD: /home/breandan/.djl.ai/mxnet
LMOD_PACKAGE_PATH: /cvmfs/soft.computecanada.ca/config/lmod/
SLURM_JOB_ACCOUNT: def-jinguo_gpu
LIBRARY_PATH: /cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/java/13.0.1/lib:/cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/cudacore/10.2.89/lib64/stubs:/cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/cudacore/10.2.89/lib64:/cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/maven/3.5.0/lib:/cvmfs/soft.computecanada.ca/easybuild/software/2017/avx512/Compiler/intel2018.3/openmpi/3.1.2/lib:/cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/ifort/2018.3.222/compilers_and_libraries_2018.3.222/linux/compiler/lib/intel64:/cvmfs/restricted.computecanada.ca/easybuild/software/2017/Core/ifort/2018.3.222/compilers_and_libraries_2018.3.222/linux/compiler/lib/intel64:/cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/icc/2018.3.222/compilers_and_libraries_2018.3.222/linux/compiler/lib/intel64:/cvmfs/restricted.computecanada.ca/easybuild/software/2017/Core/icc/2018.3.222/compilers_and_libraries_2018.3.222/linux/tbb/lib/intel64/gcc4.4:/cvmfs/restricted.computecanada.ca/easybuild/software/2017/Core/icc/2018.3.222/compilers_and_libraries_2018.3.222/linux/compiler/lib/intel64:/cvmfs/soft.computecanada.ca/nix/var/nix/profiles/gcc-7.3.0/lib64:/cvmfs/soft.computecanada.ca/nix/var/nix/profiles/gcc-7.3.0/lib:/cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/imkl/2018.3.222/mkl/lib/intel64:/cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/imkl/2018.3.222/lib/intel64:/cvmfs/soft.computecanada.ca/nix/var/nix/profiles/16.09/lib
_ModuleTable006_: ZWwvMjAxOC4zIixbImxvYWRPcmRlciJdPTYscHJvcFQ9e3R5cGVfPXtbInRvb2xzIl09MSx9LH0sWyJzdGFja0RlcHRoIl09MSxbInN0YXR1cyJdPSJhY3RpdmUiLFsidXNlck5hbWUiXT0iaW50ZWwvMjAxOC4zIix9LGphdmE9e1siZm4iXT0iL2N2bWZzL3NvZnQuY29tcHV0ZWNhbmFkYS5jYS9lYXN5YnVpbGQvbW9kdWxlcy8yMDE3L0NvcmUvamF2YS8xMy4wLjEubHVhIixbImZ1bGxOYW1lIl09ImphdmEvMTMuMC4xIixbImxvYWRPcmRlciJdPTEyLHByb3BUPXt0eXBlXz17WyJ0b29scyJdPTEsfSx9LFsic3RhY2tEZXB0aCJdPTAsWyJzdGF0dXMiXT0iYWN0aXZlIixbInVzZXJOYW1lIl09ImphdmEiLH0sbWF2ZW49e1siZm4iXT0iL2N2bWZzL3NvZnQuY29tcHV0ZWNhbmFk
SLURM_JOB_USER: breandan
GXX_ROOT: /cvmfs/soft.computecanada.ca/nix/var/nix/profiles/gcc-7.3.0/lib/gcc/x86_64-pc-linux-gnu/7.3.0
LMOD_SHORT_TIME: 3600
SLURM_TOPOLOGY_ADDR_PATTERN: switch.switch.switch.node
CPLUS_INCLUDE_PATH: /cvmfs/soft.computecanada.ca/nix/var/nix/profiles/16.09/include
MANPATH: /cvmfs/soft.computecanada.ca/easybuild/software/2017/avx512/Compiler/intel2018.3/openmpi/3.1.2/share/man:/cvmfs/restricted.computecanada.ca/easybuild/software/2017/Core/ifort/2018.3.222/compilers_and_libraries_2018.3.222/linux/man/common:/cvmfs/restricted.computecanada.ca/easybuild/software/2017/Core/icc/2018.3.222/compilers_and_libraries_2018.3.222/linux/man/common:/cvmfs/soft.computecanada.ca/nix/var/nix/profiles/gcc-7.3.0/share/man:/cvmfs/soft.computecanada.ca/nix/var/nix/profiles/16.09/share/man:/cvmfs/soft.computecanada.ca/custom/software/lmod/lmod/share/man::/opt/puppetlabs/puppet/share/man
BASH_FUNC_ml(): () {  eval $($LMOD_DIR/ml_cmd "$@")
}
EBROOTICCIFORT: /cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/intel/2018.3
SLURM_WORKING_CLUSTER: beluga:10.75.73.154:6817:8704:101
SELINUX_ROLE_REQUESTED: 
_ModuleTable005_: dmUiLFsidXNlck5hbWUiXT0iaWZvcnQvLjIwMTguMy4yMjIiLH0saW1rbD17WyJmbiJdPSIvY3ZtZnMvc29mdC5jb21wdXRlY2FuYWRhLmNhL2Vhc3lidWlsZC9tb2R1bGVzLzIwMTcvQ29yZS9pbWtsLzIwMTguMy4yMjIubHVhIixbImZ1bGxOYW1lIl09Imlta2wvMjAxOC4zLjIyMiIsWyJsb2FkT3JkZXIiXT0yLHByb3BUPXt0eXBlXz17WyJtYXRoIl09MSx9LH0sWyJzdGFja0RlcHRoIl09MSxbInN0YXR1cyJdPSJhY3RpdmUiLFsidXNlck5hbWUiXT0iaW1rbC8yMDE4LjMuMjIyIix9LGludGVsPXtbImZuIl09Ii9jdm1mcy9zb2Z0LmNvbXB1dGVjYW5hZGEuY2EvZWFzeWJ1aWxkL21vZHVsZXMvMjAxNy9Db3JlL2ludGVsLzIwMTguMy5sdWEiLFsiZnVsbE5hbWUiXT0iaW50
EBDEVELICC: /cvmfs/restricted.computecanada.ca/easybuild/software/2017/Core/icc/2018.3.222/easybuild/Core-icc-.2018.3.222-easybuild-devel
TMOUT: 172800
BASH_FUNC_module%%: () {  eval $($LMOD_CMD bash "$@") && eval $(${LMOD_SETTARG_CMD:-:} -s sh)
}
EBVERSIONCUDA: 10.2
EBVERSIONIFORT: 2018.3.222
GPU_DEVICE_ORDINAL: 0
SLURM_JOB_GID: 3067255
SLURM_STEP_NUM_TASKS: 1
CUDA_PATH: /cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/cudacore/10.2.89
TZDIR: /cvmfs/soft.computecanada.ca/nix/var/nix/profiles/16.09/share/zoneinfo
HOSTNAME: beluga5.int.ets1.calculquebec.ca
MODULESHOME: /cvmfs/soft.computecanada.ca/custom/software/lmod/lmod
_ModuleTable004_: L21vZHVsZXMvMjAxNy9Db3JlL2ljYy8uMjAxOC4zLjIyMi5sdWEiLFsiZnVsbE5hbWUiXT0iaWNjLy4yMDE4LjMuMjIyIixbImxvYWRPcmRlciJdPTQscHJvcFQ9e30sWyJyZWZfY291bnQiXT0xLFsic3RhY2tEZXB0aCJdPTIsWyJzdGF0dXMiXT0iYWN0aXZlIixbInVzZXJOYW1lIl09ImljYy8uMjAxOC4zLjIyMiIsfSxpZm9ydD17WyJmbiJdPSIvY3ZtZnMvc29mdC5jb21wdXRlY2FuYWRhLmNhL2Vhc3lidWlsZC9tb2R1bGVzLzIwMTcvQ29yZS9pZm9ydC8uMjAxOC4zLjIyMi5sdWEiLFsiZnVsbE5hbWUiXT0iaWZvcnQvLjIwMTguMy4yMjIiLFsibG9hZE9yZGVyIl09NSxwcm9wVD17fSxbInJlZl9jb3VudCJdPTEsWyJzdGFja0RlcHRoIl09MixbInN0YXR1cyJdPSJhY3Rp
LMOD_sys: Linux
XDG_RUNTIME_DIR: /run/user/3067255
HOME: /home/breandan
HISTCONTROL: ignoredups
SLURMD_NODENAME: blg4102
PERL5OPT: -I/cvmfs/soft.computecanada.ca/nix/var/nix/profiles/16.09/lib/perl5 -I/cvmfs/soft.computecanada.ca/nix/var/nix/profiles/16.09/lib/perl5/site_perl
__LMOD_SET_FPATH: 1
SLURM_NODEID: 0
EBVERSIONIMKL: 2018.3.222
RSNT_ARCH: avx512
LMOD_DIR: /cvmfs/soft.computecanada.ca/custom/software/lmod/lmod/libexec
_ModuleTable003_: MCxwcm9wVD17fSxbInJlZl9jb3VudCJdPTEsWyJzdGFja0RlcHRoIl09MSxbInN0YXR1cyJdPSJhY3RpdmUiLFsidXNlck5hbWUiXT0iY3VkYWNvcmUvLjEwLjIuODkiLH0sZ2NjY29yZT17WyJmbiJdPSIvY3ZtZnMvc29mdC5jb21wdXRlY2FuYWRhLmNhL2Vhc3lidWlsZC9tb2R1bGVzLzIwMTcvQ29yZS9nY2Njb3JlLy43LjMuMC5sdWEiLFsiZnVsbE5hbWUiXT0iZ2NjY29yZS8uNy4zLjAiLFsibG9hZE9yZGVyIl09Myxwcm9wVD17fSxbInJlZl9jb3VudCJdPTIsWyJzdGFja0RlcHRoIl09MyxbInN0YXR1cyJdPSJhY3RpdmUiLFsidXNlck5hbWUiXT0iZ2NjY29yZS8uNy4zLjAiLH0saWNjPXtbImZuIl09Ii9jdm1mcy9zb2Z0LmNvbXB1dGVjYW5hZGEuY2EvZWFzeWJ1aWxk
EBROOTOPENMPI: /cvmfs/soft.computecanada.ca/easybuild/software/2017/avx512/Compiler/intel2018.3/openmpi/3.1.2
PWD: /home/breandan/projects/def-jinguo/breandan/RL-FlappyBird
__LMOD_Priority_MODULEPATH: /opt/software/modulefiles:-10
SLURM_JOB_NUM_NODES: 1
SLURM_PRIO_PROCESS: 0
SLURM_PTY_PORT: 43288
OMPI_MCA_btl_openib_if_include: mlx5_0
INTEL_PYTHONHOME: /cvmfs/restricted.computecanada.ca/easybuild/software/2017/Core/ifort/2018.3.222/debugger_2018/python/intel64
PAGER: less -R
SELINUX_USE_CURRENT_RANGE: 
_ModuleTable002_: ImZuIl09Ii9jdm1mcy9zb2Z0LmNvbXB1dGVjYW5hZGEuY2EvZWFzeWJ1aWxkL21vZHVsZXMvMjAxNy9hdng1MTIvQ29tcGlsZXIvaW50ZWwyMDE4LjMvY3VkYS8xMC4yLmx1YSIsWyJmdWxsTmFtZSJdPSJjdWRhLzEwLjIiLFsibG9hZE9yZGVyIl09MTEscHJvcFQ9e3R5cGVfPXtbInRvb2xzIl09MSx9LH0sWyJzdGFja0RlcHRoIl09MCxbInN0YXR1cyJdPSJhY3RpdmUiLFsidXNlck5hbWUiXT0iY3VkYS8xMC4yIix9LGN1ZGFjb3JlPXtbImZuIl09Ii9jdm1mcy9zb2Z0LmNvbXB1dGVjYW5hZGEuY2EvZWFzeWJ1aWxkL21vZHVsZXMvMjAxNy9Db3JlL2N1ZGFjb3JlLy4xMC4yLjg5Lmx1YSIsWyJmdWxsTmFtZSJdPSJjdWRhY29yZS8uMTAuMi44OSIsWyJsb2FkT3JkZXIiXT0x
EBROOTIMKL: /cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/imkl/2018.3.222
EBROOTGCCCORE: /cvmfs/soft.computecanada.ca/nix/var/nix/profiles/gcc-7.3.0
LC_ALL: en_US.UTF-8
SLURM_MEM_PER_NODE: 32768
EBDEVELIMKL: /cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/imkl/2018.3.222/easybuild/Core-imkl-2018.3.222-easybuild-devel
CPATH: /cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/java/13.0.1/include:/cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/cudacore/10.2.89/nvvm/include:/cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/cudacore/10.2.89/include:/cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/cudacore/10.2.89/extras/CUPTI/include:/cvmfs/soft.computecanada.ca/easybuild/software/2017/avx512/Compiler/intel2018.3/openmpi/3.1.2/include:/cvmfs/restricted.computecanada.ca/easybuild/software/2017/Core/ifort/2018.3.222/include:/cvmfs/restricted.computecanada.ca/easybuild/software/2017/Core/icc/2018.3.222/compilers_and_libraries_2018.3.222/linux/tbb/include:/cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/imkl/2018.3.222/mkl/include/fftw:/cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/imkl/2018.3.222/mkl/include
SHLVL: 3
EBVERSIONGCCCORE: 7.3.0
SLURM_JOB_NODELIST: blg4102
LOADEDMODULES: nixpkgs/16.09:imkl/2018.3.222:gcccore/.7.3.0:icc/.2018.3.222:ifort/.2018.3.222:intel/2018.3:openmpi/3.1.2:StdEnv/2018.3:maven/3.5.0:cudacore/.10.2.89:cuda/10.2:java/13.0.1
SCRATCH: /scratch/breandan
HISTSIZE: 1000
JAVA_HOME: /cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/java/13.0.1
_ModuleTable001_: X01vZHVsZVRhYmxlXz17WyJNVHZlcnNpb24iXT0zLFsiY19yZWJ1aWxkVGltZSJdPWZhbHNlLFsiY19zaG9ydFRpbWUiXT1mYWxzZSxkZXB0aFQ9e30sZmFtaWx5PXtbImJhc2Vfb3MiXT0ibml4cGtncyIsWyJjb21waWxlciJdPSJpbnRlbCIsWyJtcGkiXT0ib3Blbm1waSIsfSxtVD17U3RkRW52PXtbImZuIl09Ii9jdm1mcy9zb2Z0LmNvbXB1dGVjYW5hZGEuY2EvY3VzdG9tL21vZHVsZXMvU3RkRW52LzIwMTguMy5sdWEiLFsiZnVsbE5hbWUiXT0iU3RkRW52LzIwMTguMyIsWyJsb2FkT3JkZXIiXT04LHByb3BUPXtsbW9kPXtbInN0aWNreSJdPTEsfSx9LFsic3RhY2tEZXB0aCJdPTAsWyJzdGF0dXMiXT0iYWN0aXZlIixbInVzZXJOYW1lIl09IlN0ZEVudiIsfSxjdWRhPXtb
EBVERSIONOPENMPI: 3.1.2
XDG_SESSION_ID: 50487
CMAKE_PREFIX_PATH: /cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/java/13.0.1:/cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/cudacore/10.2.89:/cvmfs/soft.computecanada.ca/nix/var/nix/profiles/gcc-7.3.0
SLURM_STEP_TASKS_PER_NODE: 1
EBVERSIONICCIFORT: 2018.3
SLURM_SUBMIT_HOST: beluga5.int.ets1.calculquebec.ca
LMOD_FAMILY_MPI_VERSION: 3.1.2
__LMOD_REF_COUNT__LMFILES_: /cvmfs/soft.computecanada.ca/custom/modules/nixpkgs/16.09.lua:1;/cvmfs/soft.computecanada.ca/easybuild/modules/2017/Core/imkl/2018.3.222.lua:1;/cvmfs/soft.computecanada.ca/easybuild/modules/2017/Core/gcccore/.7.3.0.lua:1;/cvmfs/soft.computecanada.ca/easybuild/modules/2017/Core/icc/.2018.3.222.lua:1;/cvmfs/soft.computecanada.ca/easybuild/modules/2017/Core/ifort/.2018.3.222.lua:1;/cvmfs/soft.computecanada.ca/easybuild/modules/2017/Core/intel/2018.3.lua:1;/cvmfs/soft.computecanada.ca/easybuild/modules/2017/avx512/Compiler/intel2018.3/openmpi/3.1.2.lua:1;/cvmfs/soft.computecanada.ca/custom/modules/StdEnv/2018.3.lua:1;/cvmfs/soft.computecanada.ca/easybuild/modules/2017/Core/maven/3.5.0.lua:1;/cvmfs/soft.computecanada.ca/easybuild/modules/2017/Core/cudacore/.10.2.89.lua:1;/cvmfs/soft.computecanada.ca/easybuild/modules/2017/avx512/Compiler/intel2018.3/cuda/10.2.lua:1;/cvmfs/soft.computecanada.ca/easybuild/modules/2017/Core/java/13.0.1.lua:1
LOCALE_ARCHIVE: /cvmfs/soft.computecanada.ca/nix/var/nix/profiles/16.09/lib/locale/locale-archive
SSH_TTY: /dev/pts/50
SSH_CLIENT: 74.14.213.251 40212 22
OMPI_MCA_btl: ^openib
__Init_Default_Modules: 1
SLURM_NNODES: 1
C_INCLUDE_PATH: /cvmfs/soft.computecanada.ca/nix/var/nix/profiles/16.09/include
RSNT_CUDA_DRIVER_VERSION: 450.51.06
EBROOTIFORT: /cvmfs/restricted.computecanada.ca/easybuild/software/2017/Core/ifort/2018.3.222
SSH_CONNECTION: 74.14.213.251 40212 10.74.73.5 22
SLURM_JOB_PARTITION: gpubase_interac,gpubase_bygpu_b1,gpubackfill
SLURM_LOCALID: 0
RSNT_ENABLE_STDENV2020_TRANSITION: no
NLSPATH: /cvmfs/restricted.computecanada.ca/easybuild/software/2017/Core/ifort/2018.3.222/idb/intel64/locale/%l_%t/%N:/cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/imkl/2018.3.222/idb/intel64/locale/%l_%t/%N
NIXUSER_PROFILE: /cvmfs/soft.computecanada.ca/nix/var/nix/profiles/16.09
BASH_FUNC_module(): () {  eval $($LMOD_CMD bash "$@") && eval $(${LMOD_SETTARG_CMD:-:} -s sh)
}
MODULERCFILE: :/cvmfs/soft.computecanada.ca/config/lmod//modulerc_beluga:/cvmfs/soft.computecanada.ca/config/lmod//modulerc
LMOD_AVAIL_STYLE: grouped:system
XDG_DATA_DIRS: /cvmfs/soft.computecanada.ca/nix/var/nix/profiles/16.09/share:/usr/local/share:/usr/share
TBBROOT: /cvmfs/restricted.computecanada.ca/easybuild/software/2017/Core/icc/2018.3.222/compilers_and_libraries_2018.3.222/linux/tbb
MODULEPATH: /opt/software/modulefiles:/cvmfs/soft.computecanada.ca/easybuild/modules/2017/avx512/CUDA/intel2018.3/cuda10.2:/cvmfs/soft.computecanada.ca/easybuild/modules/2017/CUDA/cuda10.2:/cvmfs/soft.computecanada.ca/easybuild/modules/2017/avx512/MPI/intel2018.3/openmpi3.1:/cvmfs/soft.computecanada.ca/easybuild/modules/2017/avx512/Compiler/intel2018.3:/home/breandan/.local/easybuild/modules/2017/Core:/cvmfs/soft.computecanada.ca/easybuild/modules/2017/Core:/cvmfs/soft.computecanada.ca/custom/modules
EBROOTJAVA: /cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/java/13.0.1
CUDA_VISIBLE_DEVICES: 0
CUDA_ROOT: /cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/cudacore/10.2.89
MODULEPATH_ROOT: /cvmfs/soft.computecanada.ca/easybuild/modules
MAIL: /var/spool/mail/breandan
LMOD_FAMILY_MPI: openmpi
EBVERSIONMAVEN: 3.5.0
PYTHONPATH: /cvmfs/soft.computecanada.ca/custom/python/site-packages
OMPI_MCA_pml: ^yalla
CC_RESTRICTED: true
SLURM_PTY_WIN_ROW: 24
_ModuleTable010_: bXB1dGVjYW5hZGEuY2EvY3VzdG9tL21vZHVsZXMiLH0sWyJzeXN0ZW1CYXNlTVBBVEgiXT0iL2N2bWZzL3NvZnQuY29tcHV0ZWNhbmFkYS5jYS9jdXN0b20vbW9kdWxlcyIsfQ==
TMPDIR: /tmp
SLURM_STEP_NUM_NODES: 1
CUDA_HOME: /cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/cudacore/10.2.89
SLURM_CPUS_ON_NODE: 1
SLURM_RESTART_COUNT: 0
LS_COLORS: rs=0:di=38;5;27:ln=38;5;51:mh=44;38;5;15:pi=40;38;5;11:so=38;5;13:do=38;5;5:bd=48;5;232;38;5;11:cd=48;5;232;38;5;3:or=48;5;232;38;5;9:mi=05;48;5;232;38;5;15:su=48;5;196;38;5;15:sg=48;5;11;38;5;16:ca=48;5;196;38;5;226:tw=48;5;10;38;5;16:ow=48;5;10;38;5;21:st=48;5;21;38;5;15:ex=38;5;34:*.tar=38;5;9:*.tgz=38;5;9:*.arc=38;5;9:*.arj=38;5;9:*.taz=38;5;9:*.lha=38;5;9:*.lz4=38;5;9:*.lzh=38;5;9:*.lzma=38;5;9:*.tlz=38;5;9:*.txz=38;5;9:*.tzo=38;5;9:*.t7z=38;5;9:*.zip=38;5;9:*.z=38;5;9:*.Z=38;5;9:*.dz=38;5;9:*.gz=38;5;9:*.lrz=38;5;9:*.lz=38;5;9:*.lzo=38;5;9:*.xz=38;5;9:*.bz2=38;5;9:*.bz=38;5;9:*.tbz=38;5;9:*.tbz2=38;5;9:*.tz=38;5;9:*.deb=38;5;9:*.rpm=38;5;9:*.jar=38;5;9:*.war=38;5;9:*.ear=38;5;9:*.sar=38;5;9:*.rar=38;5;9:*.alz=38;5;9:*.ace=38;5;9:*.zoo=38;5;9:*.cpio=38;5;9:*.7z=38;5;9:*.rz=38;5;9:*.cab=38;5;9:*.jpg=38;5;13:*.jpeg=38;5;13:*.gif=38;5;13:*.bmp=38;5;13:*.pbm=38;5;13:*.pgm=38;5;13:*.ppm=38;5;13:*.tga=38;5;13:*.xbm=38;5;13:*.xpm=38;5;13:*.tif=38;5;13:*.tiff=38;5;13:*.png=38;5;13:*.svg=38;5;13:*.svgz=38;5;13:*.mng=38;5;13:*.pcx=38;5;13:*.mov=38;5;13:*.mpg=38;5;13:*.mpeg=38;5;13:*.m2v=38;5;13:*.mkv=38;5;13:*.webm=38;5;13:*.ogm=38;5;13:*.mp4=38;5;13:*.m4v=38;5;13:*.mp4v=38;5;13:*.vob=38;5;13:*.qt=38;5;13:*.nuv=38;5;13:*.wmv=38;5;13:*.asf=38;5;13:*.rm=38;5;13:*.rmvb=38;5;13:*.flc=38;5;13:*.avi=38;5;13:*.fli=38;5;13:*.flv=38;5;13:*.gl=38;5;13:*.dl=38;5;13:*.xcf=38;5;13:*.xwd=38;5;13:*.yuv=38;5;13:*.cgm=38;5;13:*.emf=38;5;13:*.axv=38;5;13:*.anx=38;5;13:*.ogv=38;5;13:*.ogx=38;5;13:*.aac=38;5;45:*.au=38;5;45:*.flac=38;5;45:*.mid=38;5;45:*.midi=38;5;45:*.mka=38;5;45:*.mp3=38;5;45:*.mpc=38;5;45:*.ogg=38;5;45:*.ra=38;5;45:*.wav=38;5;45:*.axa=38;5;45:*.oga=38;5;45:*.spx=38;5;45:*.xspf=38;5;45:
MKLROOT: /cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/imkl/2018.3.222/mkl
LMOD_RC: /cvmfs/soft.computecanada.ca/config/lmod//lmodrc.lua
EBVERSIONICC: 2018.3.222
EBROOTCUDA: /cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/cudacore/10.2.89
SLURM_PTY_WIN_COL: 92
EBROOTCUDACORE: /cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/cudacore/10.2.89
SRUN_DEBUG: 3
LANG: en_US.UTF-8
MAVEN_OPTS: -Xmx1024m -XX:MaxPermSize=1024m
LMOD_PKG: /cvmfs/soft.computecanada.ca/custom/software/lmod/lmod
LMOD_SYSTEM_DEFAULT_MODULES: StdEnv
OMPI_MCA_coll: ^hcoll
UCX_NET_DEVICES: mlx5_0:1
SLURM_STEP_LAUNCHER_PORT: 33744
PERL5LIB: /cvmfs/soft.computecanada.ca/nix/var/nix/profiles/16.09/lib/perl5:/cvmfs/soft.computecanada.ca/nix/var/nix/profiles/16.09/lib/perl5/site_perl
SQUEUE_FORMAT: %.15i %.8u %.12a %.14j %.3t %.10L %.5D %.4C %.10b %.7m %N (%r) 
CMAKE_LIBRARY_PATH: /cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/cudacore/10.2.89/lib64:/cvmfs/soft.computecanada.ca/nix/var/nix/profiles/gcc-7.3.0/lib64
SQUEUE_SORT: -t,e,S
LMOD_ROOT: /cvmfs/soft.computecanada.ca/custom/software/lmod
EBVERSIONNIXPKGS: 16.09
__LMOD_REF_COUNT_LOADEDMODULES: nixpkgs/16.09:1;imkl/2018.3.222:1;gcccore/.7.3.0:1;icc/.2018.3.222:1;ifort/.2018.3.222:1;intel/2018.3:1;openmpi/3.1.2:1;StdEnv/2018.3:1;maven/3.5.0:1;cudacore/.10.2.89:1;cuda/10.2:1;java/13.0.1:1
USER: breandan
SLURM_JOBID: 16829331
SLURM_SRUN_COMM_HOST: 10.74.73.5
SLURM_PROCID: 0

-------------- Directories --------------
temp directory: /tmp
Engine cache directory: /home/breandan/.djl.ai

------------------ CUDA -----------------
GPU Count: 0
Default Device: cpu()

----------------- Engines ---------------
Default Engine: MXNet
MXNet:1.7.0, capabilities: [
  SIGNAL_HANDLER,
  LAPACK,
  CPU_SSE2,
  CPU_SSE3,
  OPENCV,
  CPU_SSE,
  CPU_AVX,
  F16C,
  BLAS_OPEN,
  CPU_SSE4_2,
  DIST_KVSTORE,
  CPU_SSE4_1,
  OPENMP,
  MKLDNN,
]
MXNet Library: /home/breandan/.djl.ai/mxnet/1.7.0-backport-mkl-linux-x86_64/libmxnet.so
cpu()

In contrast, this is the output from my local development machine:

Click to expand!

breandan@romulus:~/IdeaProjects/RL-FlappyBird$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Wed_Oct_23_19:24:38_PDT_2019
Cuda compilation tools, release 10.2, V10.2.89

breandan@romulus:~/IdeaProjects/RL-FlappyBird$ java -cp target/RL-FlappyBird-1.0-SNAPSHOT-jar-with-dependencies.jar com.kingyu.rlbird.ai.TrainBird
----------- System Properties -----------
sun.desktop: gnome
awt.toolkit: sun.awt.X11.XToolkit
java.specification.version: 11
sun.cpu.isalist: 
sun.jnu.encoding: UTF-8
java.class.path: target/RL-FlappyBird-1.0-SNAPSHOT-jar-with-dependencies.jar
java.vm.vendor: AdoptOpenJDK
sun.arch.data.model: 64
java.vendor.url: https://adoptopenjdk.net/
user.timezone: 
os.name: Linux
java.vm.specification.version: 11
sun.java.launcher: SUN_STANDARD
user.country: US
sun.boot.library.path: /usr/lib/jvm/jdk-11.0.9+11/lib
sun.java.command: com.kingyu.rlbird.ai.TrainBird
jdk.debug: release
sun.cpu.endian: little
user.home: /home/breandan
user.language: en
java.specification.vendor: Oracle Corporation
java.version.date: 2020-10-20
java.home: /usr/lib/jvm/jdk-11.0.9+11
file.separator: /
java.vm.compressedOopsMode: Zero based
line.separator: 

java.specification.name: Java Platform API Specification
java.vm.specification.vendor: Oracle Corporation
java.awt.graphicsenv: sun.awt.X11GraphicsEnvironment
sun.management.compiler: HotSpot 64-Bit Tiered Compilers
java.runtime.version: 11.0.9+11
user.name: breandan
path.separator: :
os.version: 4.15.0-132-generic
java.runtime.name: OpenJDK Runtime Environment
file.encoding: UTF-8
jnidispatch.path: /home/breandan/.cache/JNA/temp/jna6715223297428396536.tmp
java.vm.name: OpenJDK 64-Bit Server VM
jna.platform.library.path: /usr/lib/x86_64-linux-gnu:/lib/x86_64-linux-gnu:/lib64:/usr/lib:/lib:/usr/lib/i386-linux-gnu:/lib/i386-linux-gnu:/lib32:/usr/lib32:/usr/lib/i386-linux-gnu/sse2:/usr/local/cuda-11.0/targets/x86_64-linux/lib:/usr/local/cuda/targets/x86_64-linux/lib:/usr/lib/x86_64-linux-gnu/libfakeroot
java.vendor.version: AdoptOpenJDK
jna.loaded: true
java.vendor.url.bug: https://github.com/AdoptOpenJDK/openjdk-support/issues
java.io.tmpdir: /tmp
java.version: 11.0.9
user.dir: /home/breandan/IdeaProjects/RL-FlappyBird
os.arch: amd64
java.vm.specification.name: Java Virtual Machine Specification
java.awt.printerjob: sun.print.PSPrinterJob
sun.os.patch.level: unknown
java.library.path: /usr/local/cuda-10.2/lib64:/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib
java.vm.info: mixed mode
java.vendor: AdoptOpenJDK
java.vm.version: 11.0.9+11
sun.io.unicode.encoding: UnicodeLittle
java.class.version: 55.0

--------- Environment Variables ---------
PATH: /home/breandan/anaconda3/condabin:/usr/local/cuda-10.2/bin:/usr/lib/jvm/jdk-11.0.9+11/bin:/home/breandan/.pyenv/plugins/pyenv-virtualenv/shims:/home/breandan/.pyenv/shims:/home/breandan/.pyenv/bin:/home/breandan/.cargo/bin:/home/breandan/.local/bin:/home/breandan/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/breandan/bin
PYENV_SHELL: bash
XAUTHORITY: /run/user/1000/gdm/Xauthority
PM_PACKAGES_ROOT: /home/breandan/packman-repo
XMODIFIERS: @im=ibus
GDMSESSION: ubuntu
XDG_DATA_DIRS: /usr/share/ubuntu:/home/breandan/.local/share/flatpak/exports/share/:/var/lib/flatpak/exports/share/:/usr/local/share/:/usr/share/:/var/lib/snapd/desktop
TEXTDOMAINDIR: /usr/share/locale/
GTK_IM_MODULE: ibus
CONDA_PYTHON_EXE: /home/breandan/anaconda3/bin/python
DBUS_SESSION_BUS_ADDRESS: unix:path=/run/user/1000/bus
XDG_CURRENT_DESKTOP: ubuntu:GNOME
SSH_AGENT_PID: 3721
DESKTOP_AUTOSTART_ID: 10e6d3c809bace1102161239909290793400000036430018
LD_LIBRARY_PATH: /usr/local/cuda-10.2/lib64
QT4_IM_MODULE: xim
SESSION_MANAGER: local/romulus:@/tmp/.ICE-unix/3643,unix/romulus:/tmp/.ICE-unix/3643
USERNAME: breandan
LOGNAME: breandan
PWD: /home/breandan/IdeaProjects/RL-FlappyBird
IM_CONFIG_PHASE: 2
TOOLBOX_VERSION: 1.20.7940
PYENV_VIRTUALENV_INIT: 1
LESSOPEN: | /usr/bin/lesspipe %s
SHELL: /bin/bash
GIO_LAUNCHED_DESKTOP_FILE: /home/breandan/.config/autostart/jetbrains-toolbox.desktop
OLDPWD: /home/breandan
GNOME_DESKTOP_SESSION_ID: this-is-deprecated
GTK_MODULES: gail:atk-bridge
CLUTTER_IM_MODULE: xim
TERM_SESSION_ID: 31dc2e6c-19ba-4f23-8d7b-7add5118a559
TEXTDOMAIN: im-config
LC_ALL: en_US.UTF-8
XDG_SESSION_DESKTOP: ubuntu
LS_COLORS: rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:
SHLVL: 1
LESSCLOSE: /usr/bin/lesspipe %s %s
QT_IM_MODULE: ibus
CONDA_EXE: /home/breandan/anaconda3/bin/conda
TERM: xterm-256color
XDG_CONFIG_DIRS: /etc/xdg/xdg-ubuntu:/etc/xdg
LANG: en_US.UTF-8
XDG_SESSION_TYPE: x11
XDG_SESSION_ID: 2
DTSHELL_COMMANDS: /home/breandan/CLionProjects/duckietown-shell-commands
DISPLAY: :1
_CE_M: 
CONDA_SHLVL: 0
_: /usr/lib/jvm/jdk-11.0.9+11/bin/java
GPG_AGENT_INFO: /run/user/1000/gnupg/S.gpg-agent:0:1
DESKTOP_SESSION: ubuntu
USER: breandan
XDG_MENU_PREFIX: gnome-
GIO_LAUNCHED_DESKTOP_FILE_PID: 4526
WINDOWPATH: 2
QT_ACCESSIBILITY: 1
TERMINAL_EMULATOR: JetBrains-JediTerm
XDG_SEAT: seat0
SSH_AUTH_SOCK: /run/user/1000/keyring/ssh
_CE_CONDA: 
GNOME_SHELL_SESSION_MODE: ubuntu
XDG_RUNTIME_DIR: /run/user/1000
XDG_VTNR: 2
HOME: /home/breandan

-------------- Directories --------------
temp directory: /tmp
Engine cache directory: /home/breandan/.djl.ai

------------------ CUDA -----------------
GPU Count: 1
Default Device: gpu(0)
CUDA: 102
ARCH: 52
GPU(0) memory used: 1875574784 bytes

----------------- Engines ---------------
Default Engine: MXNet
MXNet:1.7.0, capabilities: [
      CUDNN,
      SIGNAL_HANDLER,
      LAPACK,
      CPU_SSE2,
      CPU_SSE3,
      OPENCV,
      CUDA,
      CPU_SSE,
      CPU_AVX,
      F16C,
      BLAS_OPEN,
      NCCL,
      CPU_SSE4_2,
      DIST_KVSTORE,
      CUDA_RTC,
      CPU_SSE4_1,
      OPENMP,
      MKLDNN,
]
MXNet Library: /home/breandan/.djl.ai/mxnet/1.7.0-backport-cu102mkl-linux-x86_64/libmxnet.so
gpu(0)

Afterwards, I tried manually transferring the contents of ~/.djl.ai to the server, this is the contents of ~/.djl.ai:

[breandan@blg4102 mxnet]$ tree /home/breandan/.djl.ai
/home/breandan/.djl.ai
└── mxnet
    ├── 1.7.0-backport-cu102mkl-linux-x86_64
    │   ├── libgfortran.so.3
    │   ├── libgomp.so.1
    │   ├── libmxnet.so
    │   └── libquadmath.so.0
    └── 1.7.0-backport-mkl-linux-x86_64
        ├── libgfortran.so.3
        ├── libgomp.so.1
        ├── libmxnet.so
        └── libquadmath.so.0

3 directories, 8 files

Please let me know if I can provide any further details.

frankfliu Mar 7, 2021

I think the different is: LD_LIBRARY_PATH: /usr/local/cuda-10.2/lib64

DJL use libcudart.so to detect GPU, which assume JVM can load libcudart.so from either LD_LIBRARY_PATH or java.library.path.

frankfliu Mar 7, 2021

Can you also show me what in your jar file, I suspect you have multiple mxnet.properties in your fatjar file.

breandan Mar 7, 2021
Author

Aha! That was it. I did the following to find libcudart.so:

[breandan@blg4102 RL-FlappyBird]$ which nvcc
/cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/cudacore/10.2.89/bin/nvcc
[breandan@blg4102 RL-FlappyBird]$ find /cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/cudacore/10.2.89 | grep libcudart
/cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/cudacore/10.2.89/targets/x86_64-linux/lib/libcudart.so
/cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/cudacore/10.2.89/targets/x86_64-linux/lib/libcudart.so.10.2
/cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/cudacore/10.2.89/targets/x86_64-linux/lib/libcudart_static.a
/cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/cudacore/10.2.89/targets/x86_64-linux/lib/libcudart.so.10.2.89
/cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/cudacore/10.2.89/doc/man/man7/libcudart.7
/cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/cudacore/10.2.89/doc/man/man7/libcudart.so.7
[breandan@blg4102 RL-FlappyBird]$ export LD_LIBRARY_PATH=/cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/cudacore/10.2.89/targets/x86_64-linux/lib/

Finally, DJL was able to detect the GPU:

-------------- Directories --------------
temp directory: /tmp
Engine cache directory: /home/breandan/.djl.ai

------------------ CUDA -----------------
GPU Count: 1
Default Device: gpu(0)
CUDA: 102
ARCH: 70
GPU(0) memory used: 1208483840 bytes

----------------- Engines ---------------
Default Engine: MXNet
MXNet:1.7.0, capabilities: [
	CUDNN,
	SIGNAL_HANDLER,
	LAPACK,
	CPU_SSE2,
	CPU_SSE3,
	OPENCV,
	CUDA,
	CPU_SSE,
	CPU_AVX,
	F16C,
	BLAS_OPEN,
	NCCL,
	CPU_SSE4_2,
	DIST_KVSTORE,
	CUDA_RTC,
	CPU_SSE4_1,
	OPENMP,
	MKLDNN,
]
MXNet Library: /home/breandan/.djl.ai/mxnet/1.7.0-backport-cu102mkl-linux-x86_64/libmxnet.so
gpu(0)

Maybe it is be possible to automatically resolve the libcudart.so location relative to nvcc. Thanks for your help!

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

CUDA available but not used? #732

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Replies: 1 comment 4 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Select a reply

Uh oh!

CUDA available but not used? #732

Uh oh!

Uh oh!

breandan Mar 7, 2021

Replies: 1 comment · 4 replies

Uh oh!

frankfliu Mar 7, 2021

Uh oh!

Uh oh!

breandan Mar 7, 2021 Author

Uh oh!

frankfliu Mar 7, 2021

Uh oh!

frankfliu Mar 7, 2021

Uh oh!

Uh oh!

breandan Mar 7, 2021 Author

breandan
Mar 7, 2021

Replies: 1 comment 4 replies

frankfliu
Mar 7, 2021

breandan Mar 7, 2021
Author

breandan Mar 7, 2021
Author