set allow_growth in default_tf_session_config (#1067)

njzjz · web-flow · commit 4c10232a740d · 2021-08-31T10:27:30.000+08:00
Currently, TF only initializes GPUs once, and never releases memory. As a result, we must set `allow_growth` in the first Session (or every session?), otherwise it doesn't work.

Please confirm the `allow_growth` config in `trainer.py` is expected, otherwise we should remove it instead.
diff --git a/deepmd/env.py b/deepmd/env.py
@@ -119,6 +119,7 @@ def get_tf_session_config() -> Any:
     set_tf_default_nthreads()
     intra, inter = get_tf_default_nthreads()
     config = tf.ConfigProto(
+        gpu_options=tf.GPUOptions(allow_growth=True),
         intra_op_parallelism_threads=intra, inter_op_parallelism_threads=inter
     )
     return config

Original file line number	Diff line number	Diff line change
`@@ -119,6 +119,7 @@ def get_tf_session_config() -> Any:`
`119`	`119`	`set_tf_default_nthreads()`
`120`	`120`	`intra, inter = get_tf_default_nthreads()`
`121`	`121`	`config = tf.ConfigProto(`
	`122`	`+ gpu_options=tf.GPUOptions(allow_growth=True),`
`122`	`123`	`intra_op_parallelism_threads=intra, inter_op_parallelism_threads=inter`
`123`	`124`	`)`
`124`	`125`	`return config`