Add public C API for partial "unicode-escape" and "raw-unicode-escape" decoding

[serhiy-storchaka]

BPO	45472
Nosy	@malemburg, @vstinner, @ezio-melotti, @serhiy-storchaka
PRs	gh-89635: Add PyUnicode_DecodeUnicodeEscapeStateful() and PyUnicode_DecodeRawUnicodeEscapeStateful() #28955

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2021-10-14.18:40:52.507>
labels = ['type-feature', 'expert-unicode', '3.11']
title = 'Add public C API for partial "unicode-escape" and "raw-unicode-escape" decoding'
updated_at = <Date 2021-10-14.19:00:35.314>
user = 'https://github.com/serhiy-storchaka'

bugs.python.org fields:

activity = <Date 2021-10-14.19:00:35.314>
actor = 'serhiy.storchaka'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Unicode']
creation = <Date 2021-10-14.18:40:52.507>
creator = 'serhiy.storchaka'
dependencies = []
files = []
hgrepos = []
issue_num = 45472
keywords = ['patch']
message_count = 1.0
messages = ['403932']
nosy_count = 4.0
nosy_names = ['lemburg', 'vstinner', 'ezio.melotti', 'serhiy.storchaka']
pr_nums = ['28955']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue45472'
versions = ['Python 3.11']

[serhiy-storchaka]

The proposed PR adds C API functions PyUnicode_DecodeUnicodeEscapeStateful() and PyUnicode_DecodeRawUnicodeEscapeStateful() for partially decoding with "unicode-escape" and "raw-unicode-escape" encodings. It were the only multibyte text codecs which did not provide such interface.

They are neccessary for correct implementation of increment decoders and stream readers (see bpo-45461 and bpo-45467).

[vstinner]

I'm not convinced that these encodings are common enough to justify adding new C API functions. You can already use them using the existing C API, no?

[malemburg]

These codecs are needed whenever you want to deal with Unicode escapes in strings - either to convert Unicode data to escaped versions or the other way around. pickle and the parser use the decoders, for example.

Many decode C API were made private in Python 3 for reasons I never understood (nor agreed with), since this breaks the symmetry of codecs having both an encoding and a decoding side to them. They are slowly being make public again, which is good, and this is just one of the necessary steps on this path.