ResourceWarnings in test_importlib in select buildbots

[jaraco]

In #132947 (comment) and the subsequent investigation, we learned that the recent changes to importlib.metadata cause ResourceWarnings to be emitted, causing select buildbot runs to fail.

After extensive investigation and code bisection, the ultimate cause is still not known. The problem has been temporarily remedied by bypassing a test that triggers the failure, but a more permanent solution is needed.

Here's a summary of the situation:

• The resource warning is caused by an open handle on the test_packages_distributions_example2

  cpython/Lib/test/test_importlib/metadata/test_main.py

  Lines 350 to 356 in 3706ef6

  	def test_packages_distributions_example2(self):
  	"""
  	Test packages_distributions on a wheel built
  	by trampolim.
  	"""
  	self._fixture_on_path('example2-1.0.0-py3-none-any.whl')
  	assert packages_distributions()['example2'] == ['example2']

  That seems to implicate something holding onto that zip file, which gets added to sys.path during fixture setup.

• The resource warning isn't triggered until test_importlib completes.

• The warning is only emitted in a few buildbots, but not on local tests, GHA tests, or other buidlbots.

• The changes in importlib.metadata.__init__ seem to be implicated, but even after removing nearly every change in that file, the problem remained.

Testing using buildbot is slow and tedious. Probably the next best step is to find a way to replicate the issue in a development environment, either by getting access to a buildbot image or by creating a comparable image that replicates the issue.

Linked PRs

• gh-138313: Suppress ResourceWarnings and restore skipped test #138597

[AA-Turner]

only emitted in a few buildbots

Which buildbots are affected? Is it a consistent set?

[jaraco]

Only some of the buildbots fail. Maybe there's something about the subset of implicated buildbots that can give a clue.

The failed buildbots are:

ARM64 Raspbian 3.x (tier-3)
s390x RHEL8 LTO + PGO 3.x (tier-3)
s390x RHEL9 LTO + PGO 3.x (tier-3)
s390x RHEL8 LTO 3.x (tier-3)
s390x Fedora Stable LTO + PGO 3.x (tier-3)
s390x RHEL9 LTO 3.x (tier-3)
s390x Fedora Stable LTO 3.x (tier-3)
AMD64 RHEL8 LTO + PGO 3.x (tier-1)
AMD64 RHEL8 LTO 3.x (tier-1)

Yes, the set is consistent. Except for the Raspbian build, the LTO (Link Time Optimization?) seems like it may be a factor.

I'm thinking trying to replicate the AMD64 RHEL8 LTO 3.x may be the easiest.

[jaraco]

Gemini helped create a Dockerfile in an attempt to replicate the issue, but while the docker image builds, it doesn't replicate the issue, even if I'm sure that the architecture is AMD64:

docker build --platform linux/amd64 .

Oh, but maybe that's because the workaround is in place. I need to make sure the test is not skipped.

I adapted the dockerfile thus, but when I run it, it fails for unrelated reasons.

Running just the importlib tests succeeds... although test_packages_distributions_example2 is indicated without an "ok" so maybe that means it's being skipped.

[jaraco]

Stan has graciously offered me access to the raspbian builder, and I've gained access to that buildbot, so I'll be testing in that environment.

[jaraco]

I've been unsuccessful in reproducing the failure in the raspbian buildbot.

I cloned main, build Python thus:

./configure --with-pydebug; CFLAGS=-UNDEBUG make

Then ran the tests with:

GITHUB_ACTION=yes make buildbottests

I've proven to myself that setting GITHUB_ACTION=yes does in fact re-activate the afflicted test.

And yet the tests succeed.

I'm not completely confident that I've replicated the commands that this particular buildbot runs. I should ask CoPilot to help elicit the exact commands the buildbot would run... or maybe see if I can find those details in the logs of one of the runs. I'm now wishing I'd saved links to more of the failed buildbot runs.

[jaraco]

I've created a PR to remove the workaround and gather a new baseline of failing buildbots.

[jaraco]

Aha. I'm now noticing that the Raspbian failure is different - with the issue being "env changed" and not a ResourceWarning. I see the resource warning in this build.

Perhaps the Raspbian failure is unrelated, or maybe it's a different manifestation triggered by the same test.

There are failures in two buildbots that seem to be entirely unrelated (don't event implicate test_importlib):

[jaraco]

Here are the list of tests that failed most recently:

buildbot/AMD64 Fedora Stable LTO + PGO PR
buildbot/AMD64 RHEL8 LTO PR
buildbot/AMD64 Windows PGO NoGIL PR
buildbot/AMD64 Windows Server 2022 NoGIL PR
buildbot/ARM Raspbian PR
buildbot/ARM64 Raspbian PR
buildbot/s390x Fedora Stable LTO + PGO PR
buildbot/s390x Fedora Stable LTO PR
buildbot/s390x RHEL8 LTO + PGO PR
buildbot/s390x RHEL9 LTO + PGO PR

It's apparent the list is not exactly stable. It's fairly stable, but there are some differences (ignoring unrelated Windows failures):

• A new failure on AMD64 Fedora Stable
• s390x RHEL8/9 without PGO has passed.
• Raspbian failures now includes two architectures adding ARM.
• The "3.x" has gone away. Maybe that was a recent change across all buildbots.

So while the failures are fairly stable, they're also somewhat intermittent.

Aha. I'm now noticing that the Raspbian failure is different - with the issue being "env changed" and not a ResourceWarning. I see the resource warning in this build.

Perhaps the Raspbian failure is unrelated, or maybe it's a different manifestation triggered by the same test.

Correction: I've gone back and looked at the Raspbian failures, and they are indeed the same. I see the ResourceWarning in those. Also, I've noticed that (env changed) indicated as the failure for test_importlib across all the failures.

[jaraco]

On the Raspbian buildbot host, I've run the exact commands extracted from the buildbot logs during a failure. I'm also now using the same branch/commit as failed in the PR.

./configure --prefix '$(PWD)/target' CFLAGS=-UNDEBUG
make -j4 all
make buildbottest 'TESTOPTS=-j2 -u-cpu --junit-xml test-results.xml -j4 ${BUILDBOT_TESTOPTS}' TESTPYTHONOPTS= TESTTIMEOUT=2400

And yet, the tests still pass. Perhaps $BUILDBOT_TESTOPTS is relevant.

[jaraco]

Success! I've replicated the issue in the buildbot host.

I noticed that the buildbot runners were also doing this command before running tests:

make pythoninfo

So I ran that and then ran the tests again, and this time the failure occurred. So either the failure is intermittent or make pythoninfo is required to trigger the failure.