gh-88886: Remove excessive encoding name normalization

[serhiy-storchaka]

The codecs lookup function now performs only minimal normalization of the encoding name before passing it to the search functions: all ASCII letters are converted to lower case, spaces are replaced with hyphens.

Excessive normalization broke third-party codecs providers, like python-iconv.

Revert "bpo-37751: Fix codecs.lookup() normalization (GH-15092)"

This reverts commit 20f59fe.

• Issue: Codec name normalization breaks custom codecs #88886

---

📚 Documentation preview 📚: https://cpython-previews--137167.org.readthedocs.build/

[StanFromIreland]

You need to update normalize_encodings doc.

@malemburg Which order should the two PRs be merged in, switching it to the C implementation and simplifying the implementation?

[serhiy-storchaka]

This mainly restores the status quo prior to bpo-37751 and updates the documentation. But I am planning more changes.

I was not sure about replacing spaces with hyphens? Should we kept this here, on leave it to the search function? Should we convert spaces to underscores instead? Or should we remove any transformation?

I plan also to change _Py_normalize_encoding() -- only convert to lower case and replace spaces, hyphens and underscores, so encoding like "utf#$^(&8" will no longer be accepted. And change encodings.normalize_encoding() in a similar way. Names that are not valid module names after normalization should not be found.

[malemburg]

LGTM modulo the one change to the news entry.

[StanFromIreland]

The docs still need updating: https://docs.python.org/3.15/library/codecs.html#encodings.normalize_encoding

[vstinner]

LGTM

[malemburg]

Is there anything left to be done for this PR ?

After merging this one, I'd like to merge #136643

This will then give us a better story overall regarding encoding normalization in CPython: a central function which deals with normalization, avoiding duplicate normalizations as best as possible.

[serhiy-storchaka]

I do not think #136643 should be merged. Normalization is an internal affair of the search function. Different search functions can have different normalizations (including no normalization).

[malemburg]

I do not think #136643 should be merged. Normalization is an internal affair of the search function. Different search functions can have different normalizations (including no normalization).

Sure, but the encoding package uses the same normalization as the C implementation that's used internally (after all, the C implementation was crafted after the encoding package's normalization function), so it makes sense to reuse it that way.

Of course, other codec packages can have their own search functions and normalizations.

[serhiy-storchaka]

The C code only needs support for few variants of few builting encodings (like "ISO-8859-1", "ISO_8859-1" and "iso8859-1"), but it does not need to support normalization for all encodings or more exotic forms of builting encodings (like "ISO_8859-1:1987" or "iso88591"). It is needed because some encodings should be accessible by their standard names before importing the encodings package.

[malemburg]

The C code only needs support for few variants of few builting encodings (like "ISO-8859-1", "ISO_8859-1" and "iso8859-1"), but it does not need to support normalization for all encodings or more exotic forms of builting encodings (like "ISO_8859-1:1987" or "iso88591"). It is needed because some encodings should be accessible by their standard names before importing the encodings package.

Yes. I am well aware that it is internally only used for a few encodings, but since it is a C reimplementation of the encoding package's normalization function, it's a good idea to have this implemented in only one place. Given that the C version is faster, merging the PR is what I'd like to do.