ValueError: filedescriptor out of range during test_subprocess on NetBSD

[furkanonder]

Bug report

Bug description:

home$ ./python -m test test_subprocess -m test_no_leaking

Output:

Using random seed: 2548767682
0:00:00 load avg: 0.02 Run 1 test sequentially in a single process
0:00:00 load avg: 0.02 [1/1] test_subprocess
/home/blue/cpython/Lib/subprocess.py:1136: ResourceWarning: subprocess 16019 is still running
  _warn("subprocess %s is still running" % self.pid,
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/home/blue/cpython/Lib/test/support/__init__.py:816: ResourceWarning: unclosed file <_io.FileIO name=1015 mode='wb' closefd=True>
  gc.collect()
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/home/blue/cpython/Lib/test/support/__init__.py:816: ResourceWarning: unclosed file <_io.FileIO name=1016 mode='rb' closefd=True>
  gc.collect()
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/home/blue/cpython/Lib/test/support/__init__.py:816: ResourceWarning: unclosed file <_io.FileIO name=1018 mode='rb' closefd=True>
  gc.collect()
ResourceWarning: Enable tracemalloc to get the object allocation traceback
test test_subprocess failed -- Traceback (most recent call last):
  File "/home/blue/cpython/Lib/test/test_subprocess.py", line 1247, in test_no_leaking
    data = p.communicate(b"lime")[0]
           ~~~~~~~~~~~~~^^^^^^^^^
  File "/home/blue/cpython/Lib/subprocess.py", line 1218, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
                     ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/blue/cpython/Lib/subprocess.py", line 2121, in _communicate
    ready = selector.select(timeout)
  File "/home/blue/cpython/Lib/selectors.py", line 314, in select
    r, w, _ = self._select(self._readers, self._writers, [], timeout)
              ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: filedescriptor out of range in select()

test_subprocess failed (1 error)

== Tests result: FAILURE ==

1 test failed:
    test_subprocess

Total duration: 1.1 sec
Total tests: run=2 (filtered)
Total test files: run=1/1 (filtered) failed=1
Result: FAILURE
home$

CPython versions tested on:

CPython main branch

Operating systems tested on:

Other

Linked PRs

• gh-124986: Fix max_handles for NetBSD in test_no_leaking #124987
• gh-124986: Fix test_no_leaking in test_subprocess on NetBSD and FreeBSD #132476
• [3.13] gh-124986: Fix test_no_leaking in test_subprocess on NetBSD and FreeBSD (GH-132476) #132498

[furkanonder]

home$ cat fd_count.c
#include <stdio.h>
#include <sys/select.h>

int main() {
    printf("FD_SETSIZE: %d\n", FD_SETSIZE);
    return 0;
}
home$ gcc fd_count.c -o fd_count.o
home$ ./fd_count.o
FD_SETSIZE: 256
home$ ulimit -Sn
1024
home$ ulimit -Hn
3404
home$ ulimit -n
1024

I don't think there is a leak, the test was successful when I increased the number of file descriptors with ulimit.

home# ulimit -n 2048
home# ./python -m test test_subprocess -m test_no_leaking
Using random seed: 389439817
0:00:00 load avg: 0.11 Run 1 test sequentially in a single process
0:00:00 load avg: 0.11 [1/1] test_subprocess

== Tests result: SUCCESS ==

1 test OK.

Total duration: 467 ms
Total tests: run=2 (filtered) skipped=2
Total test files: run=1/1 (filtered)
Result: SUCCESS
home#

I can set the max_handles=1020 as final value, above the 1020 the test is failing. In the system FD_SETSIZE is equal to 256. That's why I thought to set max_handles as 2 times the FD_SETSIZE value in the test.

[serhiy-storchaka]

For this test, we need to open a number of file descriptors slightly below the file descriptor level. And on platforms which use select() instead of poll(), all file descriptors opened by Popen should be below the select() limit (FD_SETSIZE). In other words, the file descriptor limit should not be larger than the select() limit (if select() is used).

On NetBSD, the file descriptor limit is 1024, and FD_SETSIZE is 256. On FreeBSD they are 116280 and 1024 -- the test is always skipped because it does not try so much file descriptors.

Two ways to fix the test.

1. Reduce the file descriptor limit below FD_SETSIZE. This is complicated, but allows to not skip the test on FreeBSD.
2. Close lower file descriptors -- so even if the total number of file descriptors is close to the limit, there will be free file descriptors in the select() limit.

I will try both ways.