Enhanced \N{} escapes for Unicode strings

[stevendaprano]

BPO	18614
Nosy	@terryjreedy, @ezio-melotti, @stevendaprano
Files	issue18614.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2013-08-01.13:54:05.093>
labels = ['type-feature', 'expert-unicode']
title = 'Enhanced \\N{} escapes for Unicode strings'
updated_at = <Date 2013-08-01.22:04:55.656>
user = 'https://github.com/stevendaprano'

bugs.python.org fields:

activity = <Date 2013-08-01.22:04:55.656>
actor = 'terry.reedy'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Unicode']
creation = <Date 2013-08-01.13:54:05.093>
creator = 'steven.daprano'
dependencies = []
files = ['31112']
hgrepos = []
issue_num = 18614
keywords = ['patch']
message_count = 3.0
messages = ['194075', '194087', '194123']
nosy_count = 4.0
nosy_names = ['terry.reedy', 'ezio.melotti', 'mrabarnett', 'steven.daprano']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue18614'
versions = ['Python 3.4']

Linked PRs

• gh-62814: Support U+XXXX notation in \N{...} and unicodedata.lookup #138598

[stevendaprano]

As per the discussion here:

http://mail.python.org/pipermail/python-ideas/2013-July/022419.html

\N{} escapes should support the Unicode code point notation U+xxxx (where there are four, five or six hex digits after the U+).

E.g. '\N{U+03BB}' => 'λ'

unicodedata.lookup should also support such numeric names, e.g.:

unicodedata.lookup('U+03BB') => 'λ'

As '+' is otherwise prohibited in Unicode character names, there should never be ambiguity between 'U+xxxx' as a code point and an actual name, and a single lookup function can handle both.

(See http://www.unicode.org/versions/Unicode6.2.0/ch04.pdf#G39 for details on characters allowed in names.)

Also add a function for the reverse

unicodedata.codepoint('λ') => 'U+03BB'

def codepoint(c):
    return 'U+{:04X}'.format(ord(c))

[mrabarnett]

I've attached a patch for this.

[terryjreedy]

I agree with the proposal.

Some of the code seems redundant with code we already have.
In Python, I would write

def codepoint_from_U_notation(name, namelen):
  if not (4 <= namelen <= 6): raise <wrong length>
  return chr(int(name, 16))

maybe with try-except to re-write error messages like
ValueError: invalid literal for int() with base 16: '99x3'
ValueError: chr() arg not in range(0x110000)

My point is that we already have code to convert hex strings to int; I presume PyUnicode_FromOrdinal(code) is the C version of 'chr' that already checks the max value.

[brianschubert]

I don't really see the need for this. What's the point of being able to write "\N{U+03BB}" when you can already write "\u03BB"?

I also don't see the need for a standard library function to parse U+XXXX notation. That can already be done as a trivial composition of chr and int, so I feel the old "not every 3-line function needs to be in the standard library" argument applies.

Even if added, I'm not sure that unicodedata.lookup() would be the right function to do it. That function is about accessing character name data stored in the Unicode Character Database. Overloading it to also parse U+XXXX notation feels weird.

[StanFromIreland]

I have a similar sentiment. @terryjreedy moving the question here, do you still support this?

[terryjreedy]

Re-reading the proposal from the top, I had the same thought as Brian:

What's the point of being able to write "\N{U+03BB}" when you can already write "\u03BB"?

Why did I support the proposal? Don't remember, but had little unicode experience then.

I checked the python-ideas link, which has Chris Warrick's refutation of Steven's rationale, the claim that U+ notation is 'standard'. Chris pointed out that while standard in the unicode docs and perhaps some other texts, it is not used in programming languages, which all use something like python. AFAIK, this is still more or less true. I credit Steven for linking to this rather than only his proposal, without the counter-argument that I now find persuasive.

In any case, this would now need discussion and support on discuss.python.org (which I consider unlikely).

[serhiy-storchaka]

This was a part of the long thread. The drawback of the current syntax is that \UHHHHHHHH is too long, and first 2-3 digits are always 0. The original proposition was \U+HHH..., with variable number of hexadecimal digits. It has an obvious flaw -- the lack of explicit separator between it and the following hexadecimal digit. Down the thread, some obscure programming languages ​​that use the \U+HHH... syntax were listed. Some programming languages support non-ambiguous syntax \x{HHH...} and \U{HHH...}. I didn't read the whole thread, but I think someone noticed that support for variable number of hexadecimal digits can be added without introducing new syntax -- just by making unicodedata.lookup() supporting U+HHH.... This is a clever idea that significantly lowers the bar.

The proposition is still of questionable value. \N{U+1F40D} is not shorter than \U0001f40d, and for smaller codes we have even more compact syntax \uHHHH, \xHH and \OOO.