Reset tf graph for train_and_eval to solve the memory leak.

mingxingtan · mingxingtan · commit 262754b80761 · 2020-12-07T23:32:34.000-08:00
Estimator rebuilds the graph for every epoch in train_and_eval mode.
This is probably the main reason of increasing memory usage.
diff --git a/efficientdet/main.py b/efficientdet/main.py
@@ -355,6 +355,7 @@ def run_train_and_eval(e):
         if p.exitcode != 0:
           return p.exitcode
       else:
+        tf.compat.v1.reset_default_graph()
         run_train_and_eval(e)
 
   else:
