Skip to content

ResourceWarning: subprocess n is still running #84

@jontwo

Description

@jontwo
  • whitebox version: 2.3.5
  • Python version: 3.12.9
  • Operating System: Ubuntu 22.04.5 LTS

When running Whitebox tests with pytest -Werror we are getting a lot of ResourceWarnings because subprocess.Popen is not being used as a context manager.

../../venv/farm312/lib/python3.12/site-packages/_pytest/runner.py:341: in from_call
    result: TResult | None = func()
../../venv/farm312/lib/python3.12/site-packages/_pytest/runner.py:242: in <lambda>
    lambda: runtest_hook(item=item, **kwds), when=when, reraise=reraise
../../venv/farm312/lib/python3.12/site-packages/pluggy/_hooks.py:513: in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
../../venv/farm312/lib/python3.12/site-packages/pluggy/_manager.py:120: in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
../../venv/farm312/lib/python3.12/site-packages/_pytest/threadexception.py:92: in pytest_runtest_call
    yield from thread_exception_runtest_hook()
../../venv/farm312/lib/python3.12/site-packages/_pytest/threadexception.py:68: in thread_exception_runtest_hook
    yield
../../venv/farm312/lib/python3.12/site-packages/_pytest/unraisableexception.py:95: in pytest_runtest_call
    yield from unraisable_exception_runtest_hook()
../../venv/farm312/lib/python3.12/site-packages/_pytest/unraisableexception.py:85: in unraisable_exception_runtest_hook
    warnings.warn(pytest.PytestUnraisableExceptionWarning(msg))
E   pytest.PytestUnraisableExceptionWarning: Exception ignored in: <function Popen.__del__ at 0x7f96b6f1fba0>
E
E   Traceback (most recent call last):
E     File "/usr/lib/python3.12/subprocess.py", line 1129, in __del__
E       _warn("subprocess %s is still running" % self.pid,
E   ResourceWarning: subprocess 1165850 is still running

This looks to be the same as #39 but it's definitely a whitebox_tools.py issue. If I add a function like this to the top of the file

def run_process(proc, cancel_op, verbose, callback):
    while proc is not None:
        line = proc.stdout.readline()
        sys.stdout.flush()
        if line != '':
            if not cancel_op:
                if verbose:
                    callback(line.strip())
            else:
                cancel_op = False
                proc.terminate()
                return 2
        else:
            break

    return 0

then call it from all the methods that use Popen (for me that is set_verbose_mode, set_compress_rasters, and run_tool), then the resource warnings go away.

                with Popen(args2, shell=False, stdout=PIPE,
                           stderr=STDOUT, bufsize=1, universal_newlines=True) as proc:
                    return run_process(proc, self.cancel_op, self.verbose, callback)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions