Tensorflow KMP_AFFINITY

[nisargjoshi10]

Hi all,

I have been using deepmd-kit for training some models. Whenever I train, the model gets stuck here (as shown below) for a couple of hours before it actually starts training. I'm wondering if there's a way to bypass this step and accelerate the training. I'm training the model on NVIDIA a40 GPUs.

2022-01-20 10:21:46.058470: I tensorflow/core/common_runtime/process_util.cc:146] Creating new thread pool with default inter op setting: 2. Tune using inter_op_parallelism_threads for best performance.
2022-01-20 10:21:48.592713: I tensorflow/core/platform/profile_utils/cpu_utils.cc:114] CPU Frequency: 2100000000 Hz
OMP: Info #155: KMP_AFFINITY: Initial OS proc set respected: 25,27,44,46,48,50
OMP: Info #216: KMP_AFFINITY: decoding x2APIC ids.
OMP: Info #157: KMP_AFFINITY: 6 available OS procs
OMP: Info #159: KMP_AFFINITY: Nonuniform topology
OMP: Info #287: KMP_AFFINITY: topology layer "LL cache" is equivalent to "socket".
OMP: Info #287: KMP_AFFINITY: topology layer "L3 cache" is equivalent to "socket".
OMP: Info #287: KMP_AFFINITY: topology layer "L2 cache" is equivalent to "core".
OMP: Info #287: KMP_AFFINITY: topology layer "L1 cache" is equivalent to "core".
OMP: Info #192: KMP_AFFINITY: 2 sockets x 4 cores/socket x 1 thread/core (6 total cores)
OMP: Info #218: KMP_AFFINITY: OS proc to physical thread map:
OMP: Info #172: KMP_AFFINITY: OS proc 44 maps to socket 0 core 25 thread 0
OMP: Info #172: KMP_AFFINITY: OS proc 48 maps to socket 0 core 26 thread 0
OMP: Info #172: KMP_AFFINITY: OS proc 50 maps to socket 0 core 27 thread 0
OMP: Info #172: KMP_AFFINITY: OS proc 46 maps to socket 0 core 28 thread 0
OMP: Info #172: KMP_AFFINITY: OS proc 25 maps to socket 1 core 10 thread 0
OMP: Info #172: KMP_AFFINITY: OS proc 27 maps to socket 1 core 22 thread 0
OMP: Info #254: KMP_AFFINITY: pid 79780 tid 79904 thread 1 bound to OS proc set 48
OMP: Info #254: KMP_AFFINITY: pid 79780 tid 79907 thread 2 bound to OS proc set 50
OMP: Info #254: KMP_AFFINITY: pid 79780 tid 79909 thread 4 bound to OS proc set 25
OMP: Info #254: KMP_AFFINITY: pid 79780 tid 79908 thread 3 bound to OS proc set 46
OMP: Info #254: KMP_AFFINITY: pid 79780 tid 79910 thread 5 bound to OS proc set 27
OMP: Info #254: KMP_AFFINITY: pid 79780 tid 79911 thread 6 bound to OS proc set 44
OMP: Info #254: KMP_AFFINITY: pid 79780 tid 79903 thread 7 bound to OS proc set 48
OMP: Info #254: KMP_AFFINITY: pid 79780 tid 79914 thread 8 bound to OS proc set 50
OMP: Info #254: KMP_AFFINITY: pid 79780 tid 79916 thread 10 bound to OS proc set 25
OMP: Info #254: KMP_AFFINITY: pid 79780 tid 79915 thread 9 bound to OS proc set 46
OMP: Info #254: KMP_AFFINITY: pid 79780 tid 79917 thread 11 bound to OS proc set 27
OMP: Info #254: KMP_AFFINITY: pid 79780 tid 79918 thread 12 bound to OS proc set 44

Thanks for the help!

[njzjz]

This log doesn't provide useful information. There are two operators that may slow down the pre-training process if you have a lot of data or systems. One is neighbor stating. We have provided an option in #1313 to skip it. Another stating is to initialize the model parameters. We haven't provided an option to skip it.

[nisargjoshi10]

providing more information from the slurm output and also attaching my input script. I am training water system and dont think there's a lot of data. By neighbor stating, do you refer to the sel value?

2022-01-20 10:20:07.261916: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
WARNING:tensorflow:From /gscratch/pfaendtner/nisargj/codes/miniconda/envs/deepmd/lib/python3.9/site-packages/tensorflow/python/compat/v2_compat.py:96: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version.
Instructions for updating:
non-resource variables are not supported in the long term
WARNING:root:Environment variable KMP_BLOCKTIME is empty. Use the default value 0
WARNING:root:Environment variable KMP_AFFINITY is empty. Use the default value granularity=fine,verbose,compact,1,0
WARNING:deepmd.train.run_options:Switch to serial execution due to lack of horovod module.
2022-01-20 10:21:44.291672: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  SSE4.1 SSE4.2 AVX AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2022-01-20 10:21:44.297523: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcuda.so.1
2022-01-20 10:21:45.547622: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1733] Found device 0 with properties:
pciBusID: 0000:c1:00.0 name: NVIDIA A40 computeCapability: 8.6
coreClock: 1.74GHz coreCount: 84 deviceMemorySize: 44.56GiB deviceMemoryBandwidth: 648.29GiB/s
2022-01-20 10:21:45.547679: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
2022-01-20 10:21:45.556157: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublas.so.11
2022-01-20 10:21:45.556227: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublasLt.so.11
2022-01-20 10:21:45.557592: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcufft.so.10
2022-01-20 10:21:45.557934: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcurand.so.10
2022-01-20 10:21:45.559173: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcusolver.so.11
2022-01-20 10:21:45.560735: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcusparse.so.11
2022-01-20 10:21:45.560976: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudnn.so.8
2022-01-20 10:21:45.561913: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1871] Adding visible gpu devices: 0
2022-01-20 10:21:45.561942: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
2022-01-20 10:21:46.056471: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1258] Device interconnect StreamExecutor with strength 1 edge matrix:
2022-01-20 10:21:46.056544: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1264]      0
2022-01-20 10:21:46.056555: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1277] 0:   N
2022-01-20 10:21:46.058119: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1418] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 43680 MB memory) -> physical GPU (device: 0, name: NVIDIA A40, pci bus id: 0000:c1:00.0, compute capability: 8.6)
2022-01-20 10:21:46.058470: I tensorflow/core/common_runtime/process_util.cc:146] Creating new thread pool with default inter op setting: 2. Tune using inter_op_parallelism_threads for best performance.
2022-01-20 10:21:48.592713: I tensorflow/core/platform/profile_utils/cpu_utils.cc:114] CPU Frequency: 2100000000 Hz
OMP: Info #155: KMP_AFFINITY: Initial OS proc set respected: 25,27,44,46,48,50
OMP: Info #216: KMP_AFFINITY: decoding x2APIC ids.
OMP: Info #157: KMP_AFFINITY: 6 available OS procs
OMP: Info #159: KMP_AFFINITY: Nonuniform topology
OMP: Info #287: KMP_AFFINITY: topology layer "LL cache" is equivalent to "socket".
OMP: Info #287: KMP_AFFINITY: topology layer "L3 cache" is equivalent to "socket".
OMP: Info #287: KMP_AFFINITY: topology layer "L2 cache" is equivalent to "core".
OMP: Info #287: KMP_AFFINITY: topology layer "L1 cache" is equivalent to "core".
OMP: Info #192: KMP_AFFINITY: 2 sockets x 4 cores/socket x 1 thread/core (6 total cores)
OMP: Info #218: KMP_AFFINITY: OS proc to physical thread map:
OMP: Info #172: KMP_AFFINITY: OS proc 44 maps to socket 0 core 25 thread 0
OMP: Info #172: KMP_AFFINITY: OS proc 48 maps to socket 0 core 26 thread 0
OMP: Info #172: KMP_AFFINITY: OS proc 50 maps to socket 0 core 27 thread 0
OMP: Info #172: KMP_AFFINITY: OS proc 46 maps to socket 0 core 28 thread 0
OMP: Info #172: KMP_AFFINITY: OS proc 25 maps to socket 1 core 10 thread 0
OMP: Info #172: KMP_AFFINITY: OS proc 27 maps to socket 1 core 22 thread 0
OMP: Info #254: KMP_AFFINITY: pid 79780 tid 79904 thread 1 bound to OS proc set 48
OMP: Info #254: KMP_AFFINITY: pid 79780 tid 79904 thread 1 bound to OS proc set 48
OMP: Info #254: KMP_AFFINITY: pid 79780 tid 79907 thread 2 bound to OS proc set 50
OMP: Info #254: KMP_AFFINITY: pid 79780 tid 79909 thread 4 bound to OS proc set 25
OMP: Info #254: KMP_AFFINITY: pid 79780 tid 79908 thread 3 bound to OS proc set 46
OMP: Info #254: KMP_AFFINITY: pid 79780 tid 79910 thread 5 bound to OS proc set 27
OMP: Info #254: KMP_AFFINITY: pid 79780 tid 79911 thread 6 bound to OS proc set 44
OMP: Info #254: KMP_AFFINITY: pid 79780 tid 79903 thread 7 bound to OS proc set 48
OMP: Info #254: KMP_AFFINITY: pid 79780 tid 79914 thread 8 bound to OS proc set 50
OMP: Info #254: KMP_AFFINITY: pid 79780 tid 79916 thread 10 bound to OS proc set 25
OMP: Info #254: KMP_AFFINITY: pid 79780 tid 79915 thread 9 bound to OS proc set 46
OMP: Info #254: KMP_AFFINITY: pid 79780 tid 79917 thread 11 bound to OS proc set 27
OMP: Info #254: KMP_AFFINITY: pid 79780 tid 79918 thread 12 bound to OS proc set 44
DEEPMD INFO    training data with min nbor dist: 0.8439658862406113
DEEPMD INFO    training data with max nbor size: [82, 44]
DEEPMD INFO     _____               _____   __  __  _____           _     _  _
DEEPMD INFO    |  __ \             |  __ \ |  \/  ||  __ \         | |   (_)| |
DEEPMD INFO    | |  | |  ___   ___ | |__) || \  / || |  | | ______ | | __ _ | |_
DEEPMD INFO    | |  | | / _ \ / _ \|  ___/ | |\/| || |  | ||______|| |/ /| || __|
DEEPMD INFO    | |__| ||  __/|  __/| |     | |  | || |__| |        |   < | || |_
DEEPMD INFO    |_____/  \___| \___||_|     |_|  |_||_____/         |_|\_\|_| \__|
DEEPMD INFO    Please read and cite:
DEEPMD INFO    Wang, Zhang, Han and E, Comput.Phys.Comm. 228, 178-184 (2018)
DEEPMD INFO    installed to:         /tmp/pip-req-build-_x83ia9_/_skbuild/linux-x86_64-3.9/cmake-install
DEEPMD INFO    source :              v2.0.1
DEEPMD INFO    source brach:         HEAD
DEEPMD INFO    source commit:        2f6020b
DEEPMD INFO    source commit at:     2021-09-10 22:12:15 +0800
DEEPMD INFO    build float prec:     double
DEEPMD INFO    build with tf inc:    /gscratch/pfaendtner/nisargj/codes/miniconda/envs/deepmd/lib/python3.9/site-packages/tensorflow/include
DEEPMD INFO    build with tf lib:
DEEPMD INFO    ---Summary of the training---------------------------------------
DEEPMD INFO    running on:           g3043
DEEPMD INFO    computing device:     gpu:0
DEEPMD INFO    CUDA_VISIBLE_DEVICES: 0
DEEPMD INFO    Count of visible GPU: 1
DEEPMD INFO    num_intra_threads:    0
DEEPMD INFO    num_inter_threads:    0
DEEPMD INFO    -----------------------------------------------------------------
2022-01-20 14:03:49.402435: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1733] Found device 0 with properties:
pciBusID: 0000:c1:00.0 name: NVIDIA A40 computeCapability: 8.6
coreClock: 1.74GHz coreCount: 84 deviceMemorySize: 44.56GiB deviceMemoryBandwidth: 648.29GiB/s
2022-01-20 14:03:49.402897: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1871] Adding visible gpu devices: 0
2022-01-20 14:03:49.402964: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1258] Device interconnect StreamExecutor with strength 1 edge matrix:
2022-01-20 14:03:49.402977: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1264]      0
2022-01-20 14:03:49.402983: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1277] 0:   N
2022-01-20 14:03:49.403429: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1418] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 43680 MB memory) -> physical GPU (device: 0, name: NVIDIA A40, pci bus id: 0000:c1:00.0, compute capability: 8.6)
DEEPMD INFO    ---Summary of DataSystem: training     -----------------------------------------------
DEEPMD INFO    found 1 system(s):

Input script

{
    "_comment": " model parameters",
    "model": {
        "type_map":     ["H", "O"],
        "descriptor" :{
            "type":             "se_e2_a",
            "sel":              [92, 46],
            "rcut_smth":        0.50,
            "rcut":             6.00,
            "neuron":           [25, 50, 100],
            "axis_neuron":      12,
            "seed":             38291751,
            "activation_function": "tanh",
            "resnet_dt": false,
            "type_one_side": false,
            "precision": "float64",
            "trainable": true,
            "exclude_types": [],
            "set_davg_zero": false,
            "_comment":         " that's all"
        },
        "fitting_net" : {
            "neuron":           [240, 240, 240],
            "resnet_dt":        true,
            "seed":             3218943636,
            "type":             "ener",
            "numb_fparam": 0,
            "numb_aparam": 0,
            "activation_function": "tanh",
            "precision": "float64",

            "trainable": true,
            "rcond": 0.001,
            "atom_ener": [],
            "_comment":         " that's all"
        },
        "data_stat_nbatch": 10,
        "data_stat_protect": 0.0,
        "_comment":     " that's all"
    },

    "learning_rate" :{
        "type":         "exp",
        "decay_steps":  80000,
        "start_lr":     0.002,
        "stop_lr":      1e-8,
        "_comment":     "that's all"
    },

    "loss" :{
        "start_pref_e": 0.02,
        "limit_pref_e": 2,
        "start_pref_f": 1000,
        "limit_pref_f": 1,
        "start_pref_v": 0,
        "limit_pref_v": 0,
      "type": "ener",
        "start_pref_ae": 0.0,
        "limit_pref_ae": 0.0,
        "_comment":     " that's all"
    },

    "training" : {
        "disp_file": "lcurve.out",
        "disp_freq": 2000,
        "numb_test": 5,
        "save_freq": 50000,
        "save_ckpt": "model.ckpt",
        "disp_training": true,
        "time_training": true,
        "profiling": false,
        "profiling_file": "timeline.json",
        "seed": 1271622617,
        "training_data": {
            "systems":          ["/gscratch/pfaendtner/nisargj/waterbox/1fs_stuff/WTE_16/data/train"],
            "batch_size":       "auto",
            "_comment":         "that's all"
        },
         "validation_data":{
            "systems":          ["/gscratch/pfaendtner/nisargj/waterbox/1fs_stuff/WTE_16/data/validate"],
            "batch_size":       "auto",
            "numb_btch":        1,
            "_comment":         "that's all"
        },
        "numb_steps":   16000000,
        "tensorboard":   true,
        "tensorboard_log_dir":"log",
        "tensorboard_freq": 1000,
        "_comment":     "that's all"
    },

    "_comment":         "that's all"
}

[nisargjoshi10]

seems like it is actually due to the large dataset. would you recommend dividing the raw file into many set files to make it faster or would there be no alternative but just wait?

[njzjz]

#1313 provides an option to skip waiting. You can apply the patch.