Remove setuptools runtime dependency from Pyramid

[russellballestrini]

Vendor the minimal subset of pkg_resources into pyramid._pkg_resources (310 lines, from setuptools 80.x, MIT license) to eliminate the setuptools runtime dependency while preserving the asset override architecture (OverrideProvider, register_loader_type, DefaultProvider).

Add pyramid.compat_resources as the public API for downstream Pylons packages that participate in the override system.

• pyramid._pkg_resources: vendored provider chain
• pyramid.compat_resources: public re-exports for ecosystem
• setup.py: remove setuptools from install_requires
• asset.py, path.py, static.py, config/assets.py: import swaps
• docs/conf.py: use importlib.metadata for version detection
• tests/test__pkg_resources.py: 67 tests, 97.66% coverage

---

Removing the setuptools Runtime Dependency from Pyramid and the Pylons Ecosystem

Problem

setuptools 82.0.0 removed pkg_resources entirely (Pylons/pyramid#3731). Pyramid depends on pkg_resources at runtime for its asset resolution system -- asset overrides, static views, template lookups, and the provider chain all route through pkg_resources.resource_filename, resource_exists, resource_isdir, and the DefaultProvider/OverrideProvider class hierarchy.

This is not a simple importlib.resources.files() swap. You can't just replace pkg_resources with importlib.resources for packages that participate in Pyramid's asset override system -- doing so would bypass the OverrideProvider chain entirely, potentially breaking config.override_asset() silently.

Approach

This work was done by fxhp using a machine learning coding agent. The agent audited all 108 Pylons GitHub repos, implemented the vendoring, migrated 30 downstream packages, and ran all test suites. Design decisions and migration strategy were directed by fxhp; the agent executed.

Antti Haapala provided key technical guidance in the Pylons Discord: he identified that the import override system means you cannot simply use importlib.resources.files().joinpath() and expect it to work without breaking overrides. He triaged the downstream packages -- flagging deform's module-level resource_filename call as the worst pattern (fails at import time), substanced's EntryPoint as needing importlib.metadata, and weberror's working_set as needing importlib.metadata.distributions(). He also raised concerns about the threadlocal-based override mechanism: it doesn't cooperate with Pyramid's config/request lifecycle, and third-party libraries that cache resource_filename results can produce heisenbugs. His proposed direction: deprecate the old API and require explicit registry/request access for asset resolution.

raydeo contributed important perspective on backward compatibility: importlib.resources has a mechanism (the Traversable protocol) similar to what pkg_resources provided, pointing toward a possible future where Pyramid's override system is rebuilt on importlib.resources rather than vendored pkg_resources. He emphasized not breaking the override system "in any way" during the transition -- noting that PyPI/Warehouse famously used it to swap UI assets at runtime.

PRs #3748 and #3749 (by mmerickel) already migrated DottedNameResolver and the scripts (pshell, pdistreport) to importlib.metadata, clearing the path for this work to focus on the remaining hard part: the asset/resource provider system.

Solution

Pyramid Core

Vendor the minimal subset of pkg_resources directly into Pyramid, eliminating the setuptools runtime dependency while preserving the asset override architecture.

Two new modules:

• pyramid._pkg_resources (310 lines) -- Vendored from setuptools 80.x (MIT license, Jason R. Coombs). Contains only what Pyramid needs: _provider_factories, register_loader_type(), get_provider(), _find_adapter(), ResourceManager, NullProvider, DefaultProvider, and the module-level convenience functions (resource_filename, resource_exists, resource_isdir, resource_stream, resource_string, resource_listdir).

• pyramid.compat_resources (46 lines) -- Public API for the Pylons ecosystem. Re-exports the functions from _pkg_resources so downstream packages have a stable import path. Packages that depend on Pyramid should import from here rather than using importlib.resources directly, to preserve override compatibility.

Four existing source files updated (import swap only):

File	Change
src/pyramid/asset.py	import pkg_resources → from pyramid import _pkg_resources
src/pyramid/path.py	import pkg_resources → from pyramid import _pkg_resources
src/pyramid/static.py	from pkg_resources import ... → from pyramid._pkg_resources import ...
src/pyramid/config/assets.py	import pkg_resources → from pyramid import _pkg_resources

setuptools removed from install_requires in setup.py. The build-time from setuptools import setup stays (guaranteed by PEP 517).

New test file: tests/test__pkg_resources.py -- 67 tests covering unit, integration, and functional scenarios at 97.66% coverage.

Total pyramid core tests: 2593 existing + 67 new = 2660, all passing.

Downstream Packages

30 repositories across the Pylons organization were updated. The changes fall into categories based on the dependency graph:

1. Packages that depend on Pyramid (one-liner import swap)

These packages use pyramid.compat_resources -- Pyramid's public override-aware resource API. This is the right approach because these packages participate in Pyramid's asset override system (e.g., pyramid_chameleon templates can be overridden via config.override_asset()).

# Before
from pkg_resources import resource_filename

# After
from pyramid.compat_resources import resource_filename

Package	Files Changed
pyramid_chameleon	renderer.py (resource_filename, resource_exists)
pyramid_chameleon_genshi	__init__.py
pyramid_deform	__init__.py
pyramid_formish	__init__.py, zcml.py
pyramid_skins	src/pyramid_skins/configuration.py
akhet	akhet/static.py (resource_exists)
cartouche	cartouche/registration.py
pyramid_zcml	tests/test_units.py
substanced	file/views.py, form/__init__.py

2. Packages NOT depending on Pyramid (use stdlib directly)

These packages cannot import from pyramid.compat_resources because they don't depend on Pyramid. They use importlib.resources.files() from the stdlib (Python 3.9+), which is correct because they do not participate in Pyramid's asset override system.

deform (critical -- the module-level resource_filename call is the worst pattern because it runs at import time):

# Before (template.py line 7, 126-127)
from pkg_resources import resource_filename
default_dir = resource_filename('deform', 'templates/')

# After
from importlib.resources import files
default_dir = str(files('deform').joinpath('templates'))

weberror (two distinct patterns):

• evalexception.py: resource_filename → importlib.resources.files()
• formatter.py: pkg_resources.working_set → importlib.metadata.distributions()

3. EntryPoint replacement (substanced)

substanced's EntryPoint.parse().load() is really just importing a module by dotted name:

# Before (evolution/__init__.py)
from pkg_resources import EntryPoint
func = EntryPoint.parse('x=%s' % scriptname).load(False)

# After
from importlib import import_module
func = import_module(scriptname)

4. docs/conf.py files (19 repos)

# Before
import pkg_resources
version = pkg_resources.get_distribution('package_name').version

# After
from importlib.metadata import version as pkg_version
version = pkg_version('package_name')

Repos updated: colander, deform, hupper, hypatia, plaster_pastedeploy, pyramid_chameleon, pyramid_cookbook, pyramid_exclog, pyramid_jinja2, pyramid_mailer, pyramid_mako, pyramid_nacl_session, pyramid_retry, pyramid_rpc, pyramid_simpleform, pyramid_tm, pyramid_zcml, pyramid_zodbconn, venusian.

5. Test file fixes

Package	File	Change
plaster_pastedeploy	tests/conftest.py	pkg_resources.working_set.add_entry() → sys.path.insert()
pyramid_tm	tests/test_it.py	pkg_resources.get_distribution/parse_version → importlib.metadata.version + packaging.version.Version
pyramid_zcml	tests/test_units.py	pkg_resources.resource_filename → pyramid.compat_resources.resource_filename
shootout	tests/test_functional.py	pkg_resources.get_distribution().location → importlib.resources.files()
pyramid_skins	tests/test_doctests.py	pkg_resources.resource_filename → pyramid.compat_resources.resource_filename
pyramid_formish	tests/test_api.py	pkg_resources.resource_filename → pyramid.compat_resources.resource_filename
substanced	tests/test_evolution.py	Updated import_module mock target
deform	tests/test_template.py	pkg_resources.resource_filename → importlib.resources.files

Test Results

Pyramid Core

• 2660/2660 tests pass (2593 existing + 67 new)
• 97.66% coverage on _pkg_resources.py
• Verified: import pyramid does NOT trigger import pkg_resources from setuptools

Downstream Packages (full test suites)

Package	Tests	Pass	Fail	Notes
pyramid_chameleon	67	67	0
pyramid_deform	59	59	0
pyramid_zcml	107	107	0
pyramid_tm	61	61	0
plaster_pastedeploy	71	71	0
substanced	1549	1548	1	Pre-existing Pyramid 2.x security policy issue
deform	406	405	1	Test env missing pyramid (test dep, not runtime)
akhet	19	19	0
pyramid_skins	7	7	0
cartouche	25	19	6	Pre-existing deform HTML output changes
Total	2371	2363	8	0 migration-related failures

Not runnable (pre-existing Python 2 syntax)

• weberror -- migration code validated via API testing
• pyramid_formish -- import validation passes
• pyramid_chameleon_genshi -- import validation passes
• shootout -- uses removed Pyramid API (UnencryptedCookieSessionFactoryConfig)

Defect Found During Testing

NullProvider.__init__ crashed when module.__file__ is None (namespace packages). Fixed:

# Before
self.module_path = os.path.dirname(getattr(module, '__file__', ''))

# After
self.module_path = os.path.dirname(getattr(module, '__file__', '') or '')

getattr returns the attribute value None (not the default '') because __file__ exists but is set to None on namespace packages.

Migration Guide for Third-Party Packages

If your package depends on Pyramid

# Replace this:
from pkg_resources import resource_filename
# or
import pkg_resources
pkg_resources.resource_filename('mypackage', 'templates/')

# With this:
from pyramid.compat_resources import resource_filename
resource_filename('mypackage', 'templates/')

This preserves compatibility with config.override_asset().

If your package does NOT depend on Pyramid

# Replace this:
from pkg_resources import resource_filename
path = resource_filename('mypackage', 'data/file.txt')

# With this (Python 3.9+):
from importlib.resources import files
path = str(files('mypackage').joinpath('data/file.txt'))

For docs/conf.py version detection

# Replace this:
import pkg_resources
version = pkg_resources.get_distribution('mypackage').version

# With this:
from importlib.metadata import version as pkg_version
version = pkg_version('mypackage')

For EntryPoint.load() patterns

# Replace this:
from pkg_resources import EntryPoint
obj = EntryPoint.parse('x=dotted.module.name').load(False)

# With this:
from importlib import import_module
obj = import_module('dotted.module.name')

Architectural Notes and Future Direction

This vendoring approach is a pragmatic bridge fix that preserves exact backward compatibility. It does not attempt to redesign the asset override system. The community has been discussing a longer-term architectural direction, and there are important considerations for future work.

Why not just use importlib.resources.files() everywhere?

Pyramid's asset override system works by subclassing DefaultProvider and calling register_loader_type() to intercept resource lookups at the pkg_resources provider level. When you call config.override_asset('mypackage:templates/', 'otherpackage:templates/'), Pyramid installs an OverrideProvider that redirects resource lookups transparently.

If downstream packages switch to importlib.resources.files() directly, those calls bypass the override chain entirely. This is why packages that depend on Pyramid use pyramid.compat_resources (which routes through the vendored provider system) rather than importlib.resources.

Packages that do NOT depend on Pyramid (deform, weberror) can safely use importlib.resources.files() because they don't participate in the override system.

The threadlocal concern

The current override mechanism relies on threadlocals (get_current_registry()) to find the active registry and its configured overrides. As Antti Haapala pointed out, this creates potential issues:

• Code that caches resource_filename results on first use may get stale overrides
• The override is global to the thread, not scoped to a request or configuration
• It doesn't cooperate with Pyramid's config/request lifecycle

raydeo noted the design is "pretty slick" for transparently overriding resources in libraries like pyramid_chameleon and pyramid_jinja2 that use pkg_resources.resource_filename, but acknowledged the pitfalls.

Possible future directions

1. Deprecate the threadlocal-based override mechanism in favor of a new API that requires explicit registry/request access (e.g., request.resolve_asset('mypackage:templates/foo.pt') or config.resolve_asset(...)). This is the direction Antti Haapala advocated: a cleaner API where callers must have access to the registry or request, avoiding heisenbugs from cached resource lookups.

2. Build override support on importlib.resources using its Traversable protocol and custom resource readers. As raydeo noted, this mechanism is "more flexible than pkg_resources" and provides a more modern hook point than register_loader_type(). raydeo's optimistic goal: Pyramid patches importlib.resources (not pkg_resources) for overrides, so the system works natively without setuptools.

3. Keep vendored _pkg_resources as a fallback for the transition period, allowing packages to migrate at their own pace. raydeo emphasized not breaking the override system "in any way" during the transition.

The vendored module gives the ecosystem breathing room to plan this transition without being forced by a setuptools release.

What's NOT Included

• Egg/zip extraction -- not used by Pyramid
• Distribution/WorkingSet/Requirement -- already migrated to importlib.metadata in PRs remove pkg_resources from DottedNameResolver #3748/Remove pkg resources from scripts #3749
• Metadata methods (has_metadata, get_metadata) -- not used by Pyramid's asset system

Files Summary

Location	New	Modified
Pyramid core	3 (_pkg_resources.py, compat_resources.py, test__pkg_resources.py)	5 (asset.py, path.py, static.py, config/assets.py, setup.py)
Downstream packages	0	~45 files across 30 repos