unhandled exception during timeout handling

[kasper93]

Hi,

Looks like when timeout happens, it may leave a process running, if it was spawned as child. I didn't debug it further, maybe it doesn't have to be a child, but after some time of cvise running, I had multiple processes just spinning in the background in infinite loop, which was created during reduction. This is likely related to #172, but in my case on Windows, but symptoms looks similar.

But I want to report different issue on Windows, call to self.save_extra_dir(self.temporary_folders[future]) in timeout handling causes unhandled exception. Files are locked on Windows, so either needs to kill owning process or save it differently. Removing this save_extra_dir seems to workaround the issue, by not saving those files.

00:00:15 WARNING Test timed out.
Traceback (most recent call last):
  File "E:/MSYS/mingw64/lib/python3.12/shutil.py", line 847, in move
    os.rename(src, real_dst)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'E:/MSYS/tmp/cvise-LinesPass-0-93wh4jw5/cvise-s081pc4j' -> 'cvise_extra_0009/cvise-s081pc4j'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "E:/MSYS/mingw64/bin/cvise", line 439, in <module>
    reducer.reduce(pass_group, skip_initial=args.skip_initial_passes)
  File "E:/MSYS/mingw64/share/cvise/cvise.py", line 163, in reduce
    self._run_additional_passes(pass_group['first'])
  File "E:/MSYS/mingw64/share/cvise/cvise.py", line 186, in _run_additional_passes
    self.test_manager.run_pass(p)
  File "E:/MSYS/mingw64/share/cvise/utils/testing.py", line 587, in run_pass
    success_env = self.run_parallel_tests()
                  ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "E:/MSYS/mingw64/share/cvise/utils/testing.py", line 497, in run_parallel_tests
    quit_loop = self.process_done_futures()
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "E:/MSYS/mingw64/share/cvise/utils/testing.py", line 410, in process_done_futures
    self.save_extra_dir(self.temporary_folders[future])
  File "E:/MSYS/mingw64/share/cvise/utils/testing.py", line 392, in save_extra_dir
    shutil.move(test_case_path, extra_dir)
  File "E:/MSYS/mingw64/lib/python3.12/shutil.py", line 865, in move
    rmtree(src)
  File "E:/MSYS/mingw64/lib/python3.12/shutil.py", line 781, in rmtree
    return _rmtree_unsafe(path, onexc)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "E:/MSYS/mingw64/lib/python3.12/shutil.py", line 639, in _rmtree_unsafe
    onexc(os.rmdir, path, err)
  File "E:/MSYS/mingw64/lib/python3.12/shutil.py", line 637, in _rmtree_unsafe
    os.rmdir(path)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'E:/MSYS/tmp/cvise-LinesPass-0-93wh4jw5/cvise-s081pc4j'

Thanks,
Kacper

[emaxx-google]

I think #283 that I landed earlier should've partially mitigated the save_extra_dir() issue, so you might want to try the tip-of-the-tree version of C-Vise.

An even better solution for this Windows issue would be to make sure the children processes are killed before we reach this timeout handler. This is something that Pebble - the library for process pool management we use - doesn't provide, and it seems pretty hard to achieve via it. But it's good timing since I was anyway prototyping a custom process pool implementation (to address other issues with Pebble), and I could try adding a robust solution for this too.

As for the hung children processes, yes, this is a common complaint, and while #172 proposed fixing it on *nix via killpg, Windows would need something different. Some quick search points to JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE - hopefully we can get to exploring that at some point. For the time being, I could only suggest trying the current tip-of-the-tree C-Vise version, since some of my recent changes (like #311) should've reduced the number of "orphaned" children to some extent.