torchx - fix race condition issue that local_scheduler LogIterator that reads early (meta-pytorch#1099)

Summary:
torchx/cli/test:cmd_run_test - test_run_with_log (https://www.internalfb.com/intern/test/281475186013299?ref_report_id=0) regularly failed due to assertion on local_scheduler output is missing expected content.  This is causing noise to oncall due to failed release test blocking torchx release. https://fburl.com/conveyor/a5u31rby


issue looked to be in the LogIterator abort early if content has not written: https://www.internalfb.com/code/fbsource/[922fd5827417][history]/fbcode/torchx/schedulers/local_scheduler.py?lines=1185-1189 

The propose fixed is add a small delay before fp_log is setup.

Differential Revision: D80716088

Co-authored-by: Tony Kao <[email protected]>

Author: tonykao8080

torchx/schedulers/local_scheduler.py

@@ -1159,6 +1159,7 @@ def __iter__(self) -> "LogIterator":
             self._check_finished()  # check to see if app has finished running
 
             if os.path.isfile(self._log_file):
+                time.sleep(0.1)  # fix timing issue
                 self._log_fp = open(
                     self._log_file,
                     mode="rt",