Should the output from `packaging.tags.sys_tags()` take precedence over `distlib.wheel.compatible_tags()`?

[tm-jdelapuente]

Problem
I'm not downloading all the packages that I'm expecting. This is due to distlib.wheel.compatible_tags() returning a smaller set of tag combinations than packaging.tags.sys_tags().

Context
I can see that they're different implementations and from reading online packaging.tags.sys_tags() seems to be the reference implementation for PEP 425 (please correct me if I'm wrong).

Questions
Does this mean that the output that I get from packaging.tags.sys_tags() should take precedence than the one from distlib.wheel.compatible_tags()?

If so, why doesn't distlib import and use packaging.tags.sys_tags() to maintain consistency with packaging?

[vsajip]

If so, why doesn't distlib import

They are two separate, independent packages, and should deliver the same result. Both should support PEP425, but PEPs can change, and these are volunteer projects that proceed at different rates based on volunteer time available, It's possible that there is a bug in the distlib implementation, or that it hasn't kept pace with some change in the PEP.

from reading online packaging.tags.sys_tags() seems to be the reference implementation for PEP 425 (please correct me if I'm wrong).

Where exactly did you read this? It may be that certain people prefer certain libraries, but apart from that I'm not sure that there is a reference implementation - as far as I understand it, the point of a PEP is to allow multiple independent implementations to exist.

Can you give more details about your environment where the two APIs give different results?

[pfmoore]

I think the issue is that the tag spec doesn't actually define what tags must be supported in a given environment, so the actual result is implementation dependent (based on which library you use).

Most packaging tools use the packaging library, so that tends to be seen as the "correct" answer, but the reality is that there's no standard here 🙁

[tm-jdelapuente]

Thanks both! That's interesting and helpful.

If most packaging tools use the packaging library and both of these are from the PyPA, what's people's appetite for one of these changes to distlib?:

1. Refactor distlib.wheel.compatible_tags to get the tags from packaging.tags.sys_tags: this would require some processing of the output so that compatible_tags continues returning a list of (pyver, abi, arch) tuples
2. Keep distlib's implementation of tags but write unit tests so that it returns the same tags as distlib.wheel.compatible_tags
3. Keep distlib's implementation of tags but add documentation that's easier to find that the wheel tags from distlib may not match the implementation of other libraries and what to do if we want to use a Locator with packaging tags (e.g. we could add it to the docstring of Locator?)

I'd suggest the first to follow the principles outlined in PEP 20 ("There should be one-- and preferably only one --obvious way to do it").

[pfmoore]

As a matter of principle, I'd prefer it if the standards were improved to ensure that we documented the correct tags to use - if it matters for interoperability that two tools don't say different things about what wheels are compatible with any given platform, then the standards should reflect that. Doing this would be quite a lot of work, though.

As a practical solution, I'd be inclined to say that distlib could add some tests to check that it gives the same results as packaging. That would limit the dependency to only being in the tests. Obviously the distlib implementation would need to change to ensure those tests pass... Someone would need to create a PR for this, and it would potentially involve a certain amount of additional maintenance cost on distlib (I don't know how often packaging changes its tags algorithm - hopefully not often).

My least favourite solution would be to just use packaging. The whole point of having standards is to avoid behaviour being defined by a specific library (the uv installer, for example, is written in Rust, and might not even be able to use packaging).

But as the project maintainer, @vsajip may have a different (probably more practical 🙂) view, which should take precedence over the above.

[tm-jdelapuente]

Interesting - thanks! Would be curious to hear your take @vsajip - and happy to write the PR in the direction we agree to take it in.

[vsajip]

Before saying what the solution should be, I'd rather know more about the problem - namely, under what circumstances do the results of the two libraries differ? That would perhaps lead to answering the question of why/how they differ, and that in turn could lead to possible solutions. Adopting packaging in distlib's tests implies accepting that packaging is always right - how can we be sure of this if the spec is not all that well-defined? The standards should be improved where there are errors, omissions or ambiguities. If it can be established (independently of packaging - just by reference to the spec) that distlib does not follow PEP425 as written, then distlib should be updated to address that. If it can't be so established, then it seems a bit arbitrary to adopt packaging's algorithm as somehow "right", just because it's the more commonly used library.

In any case, I think more information on the differences in results is needed before deciding how best to proceed.

[tm-jdelapuente]

That's fair. I'll look into it when I have some time and loop back.