Support arbitrary equality on arbitrary strings for Specifier and SpecifierSet's filter and contains method.

[notatallshaw]

Towards #943

According to the spec:

The primary use case for arbitrary equality is to allow for specifying a version which cannot otherwise be represented by this specification. This operator is special and acts as an escape hatch to allow someone using a tool which implements this specification to still install a legacy version which is otherwise incompatible with this specification.

An example would be ===foobar which would match a version of foobar.

This enables that main purpose of arbitrary equality for Specifier.contains, Specifier.filter, SpecifierSet.contains, and SpecifierSet.filter.

There are four design choices I have made that are either ambiguous in the spec or purely a question for the packaging API:

1. Empty SpecifierSet is satisfied by non-PEP 440 versions

This is a natural consequence of wanting the invariant:

For any version v and specifiers A and B, if v is not satisfied by A, then v must also not be satisfied by A and B.

Given this invariant the empty specifier MUST allow non-PEP 440 versions, or you end up with SpecifierSet("").contains("foobar") being False and then adding the specifier SpecifierSet("===foobar").contains("foobar") being True breaking the invariant.

While this invariant isn't part of the specification, it is very hard to do any kind of logical inference if it is not true, and it does not violate the specification.

2. Any operator other than === exclude non-PEP 440 versions

As the operators ~=, ==, !=, <=, >=, <, and > are only defined to work on PEP 440 style versions they exclude a non-PEP 440 version.

3. For Specifier.prereleases when a version is not PEP 440 complaint return None

This is an arbitrary choice, and technically changes the Specifier.prereleases API which previously guaranteed a bool return (though it's parent class BaseSpecifier already allows for a None return on this method), but I don't think either True or False make sense.

4. Explicitly setting prereleases does not affect the satisfiability of non-PEP 440 versions

This is an arbitrarily choice, and not determined by the spec but by these options being available in the packaging API.

Being that arbitrary strings have no concept of pre-releases to be consistent either:

1. Explicitly setting prereleases does not affect the satisfiability of non-PEP 440 versions
2. Or, setting prereleases does not allow any non-PEP 440 versions

My choice here was largely driven by the simplicity of the implementation, "1." is simple to implement, "2." is very complex and would likely require a major refactoring.

[notatallshaw]

Okay I've tweaked the semantics a little bit, and updated the PR description, based on the discussion with @Liam-DeVoe in #932 (comment).

Any standard operator, i.e. not ===, now excludes all non-PEP 440 arbitrary string version including !=V and !=V.*, so now:

>>> SpecifierSet('===foobar,!=1.0').contains('foobar')
False

This is because != is only defined on PEP 440 style versions and therefore the semantics do not define how it should handle arbitrary strings, so it's very justifiable to say it just excludes them. If the spec is clarified in the future it is also easier to go from being less permissive to more permissive than the other way around.

The empty specifier set still selects arbitrary string versions for the reasons I outline in the description.

I also realized there was an edge case around the empty specifier when passed arbitrary string versions and pre-release versions, I have updated the semantics so that non-PEP 440 arbitrary strings do not exclude pre-releases from being selected by the empty specifier e.g.:

>>> list(SpecifierSet('').filter(['foobar', '1.0.a1']))
['foobar', '1.0.a1']

Overall I think this is correct, and it reduces the semantic changes between this PR and what #932 implemented.

[notatallshaw]

Pinging @uranusjr @di and @pradyunsg based on #336 being the only significant discussion of arbitrary equality I could find, to check if anyone has any objections to this PR.

[notatallshaw]

I am going to wait one more week for objection, or reviews, and then merge this on the basis it is further completing PEP 440 implementation of Specifiers.

My only concern of semantics is case sensitivity, and I don't think that choice should be made in this PR, so I've broken out a separate issue to address that: #974

[henryiii]

I'd be happy to review the code, but I don't want to make a decision on changes, at least one like case insensitivity on ===, which I don't have a direct stake in. I've pinged the uv team, they might also have useful opinions, having a separate implementation.

[notatallshaw]

I've updated the PR and PR description to remove the concern on case insensitivity now that is decided:

• https://discuss.python.org/t/clarify-that-arbitrary-equality-is-supposed-to-be-case-insensitive/104885/57
• Specify arbitrary equality case insensitivity. packaging.python.org#1959

[henryiii]

I’ll try to review this by tomorrow. Ping me if I forget.

[henryiii]

Code looks good; I don't write resolvers, so don't weight my review of the idea too highly. On your choices:

1. The invariant seems quite useful, and logical inference on specifiers is quite important, sounds reasonable.
2. Reasonable.
3. Even without the base class, adding None, as long as it behaves close enough to False, is pretty safe. Very few uses compare with is False. Including the fact the base class allows None, I think it's quite safe.
4. I agree that option 1 sounds fine, simple is good, no strong opinion though.