Parser gives UnicodeDecodeError on what should be good code

[tom-pytel]

Bug report

Bug description:

In 3.12 and below parses fine. Note the special unicode character in the inner string, its '\u3001' (utf8 b'\xe3\x80\x81'), it goes from good to fail by removing a space or parenthesizing the whole expression, so positional?

>>> from io import BytesIO
>>> from tokenize import tokenize
>>> 
>>> src_good = '''f"{f(a=lambda: '、' \n)}"'''
>>> src_bad1 = '''f"{f(a=lambda: '、'\n)}"'''
>>> src_bad2 = '''(f"{f(a=lambda: '、' \n)}")'''
>>>
>>> for token in tokenize(BytesIO(src_good.encode()).readline): print(token)
... 
TokenInfo(type=65 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='')
TokenInfo(type=59 (FSTRING_START), string='f"', start=(1, 0), end=(1, 2), line='f"{f(a=lambda: \'、\' \n')
TokenInfo(type=55 (OP), string='{', start=(1, 2), end=(1, 3), line='f"{f(a=lambda: \'、\' \n')
TokenInfo(type=1 (NAME), string='f', start=(1, 3), end=(1, 4), line='f"{f(a=lambda: \'、\' \n')
TokenInfo(type=55 (OP), string='(', start=(1, 4), end=(1, 5), line='f"{f(a=lambda: \'、\' \n')
TokenInfo(type=1 (NAME), string='a', start=(1, 5), end=(1, 6), line='f"{f(a=lambda: \'、\' \n')
TokenInfo(type=55 (OP), string='=', start=(1, 6), end=(1, 7), line='f"{f(a=lambda: \'、\' \n')
TokenInfo(type=1 (NAME), string='lambda', start=(1, 7), end=(1, 13), line='f"{f(a=lambda: \'、\' \n')
TokenInfo(type=55 (OP), string=':', start=(1, 13), end=(1, 14), line='f"{f(a=lambda: \'、\' \n')
TokenInfo(type=3 (STRING), string="'、'", start=(1, 15), end=(1, 18), line='f"{f(a=lambda: \'、\' \n')
TokenInfo(type=63 (NL), string='\n', start=(1, 19), end=(1, 20), line='f"{f(a=lambda: \'、\' \n')
TokenInfo(type=55 (OP), string=')', start=(2, 0), end=(2, 1), line=')}"')
TokenInfo(type=55 (OP), string='}', start=(2, 1), end=(2, 2), line=')}"')
TokenInfo(type=61 (FSTRING_END), string='"', start=(2, 2), end=(2, 3), line=')}"')
TokenInfo(type=4 (NEWLINE), string='', start=(2, 3), end=(2, 4), line=')}"')
TokenInfo(type=0 (ENDMARKER), string='', start=(3, 0), end=(3, 0), line='')

>>> for token in tokenize(BytesIO(src_bad1.encode()).readline): print(token)
... 
TokenInfo(type=65 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='')
TokenInfo(type=59 (FSTRING_START), string='f"', start=(1, 0), end=(1, 2), line='f"{f(a=lambda: \'、\'\n')
TokenInfo(type=55 (OP), string='{', start=(1, 2), end=(1, 3), line='f"{f(a=lambda: \'、\'\n')
TokenInfo(type=1 (NAME), string='f', start=(1, 3), end=(1, 4), line='f"{f(a=lambda: \'、\'\n')
TokenInfo(type=55 (OP), string='(', start=(1, 4), end=(1, 5), line='f"{f(a=lambda: \'、\'\n')
TokenInfo(type=1 (NAME), string='a', start=(1, 5), end=(1, 6), line='f"{f(a=lambda: \'、\'\n')
TokenInfo(type=55 (OP), string='=', start=(1, 6), end=(1, 7), line='f"{f(a=lambda: \'、\'\n')
TokenInfo(type=1 (NAME), string='lambda', start=(1, 7), end=(1, 13), line='f"{f(a=lambda: \'、\'\n')
TokenInfo(type=55 (OP), string=':', start=(1, 13), end=(1, 14), line='f"{f(a=lambda: \'、\'\n')
TokenInfo(type=3 (STRING), string="'、'", start=(1, 15), end=(1, 18), line='f"{f(a=lambda: \'、\'\n')
TokenInfo(type=63 (NL), string='\n', start=(1, 18), end=(1, 19), line='f"{f(a=lambda: \'、\'\n')
TokenInfo(type=55 (OP), string=')', start=(2, 0), end=(2, 1), line=')}"')
Traceback (most recent call last):
  File "<python-input-8>", line 1, in <module>
    for token in tokenize(BytesIO(src_bad1.encode()).readline): print(token)
                 ~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/tokenize.py", line 492, in tokenize
    yield from _generate_tokens_from_c_tokenizer(rl_gen.__next__, encoding, extra_tokens=True)
  File "/usr/local/lib/python3.13/tokenize.py", line 582, in _generate_tokens_from_c_tokenizer
    for info in it:
                ^^
UnicodeDecodeError: 'utf-8' codec can't decode bytes in position 13-14: unexpected end of data

The other bad source and other permutations give same error. You get immediate error typing the bad src interactively.

CPython versions tested on:

3.13, 3.14, 3.15

Operating systems tested on:

Linux

Linked PRs

• gh-139516: Fix lambda colon start format spec in f-string in tokenizer #139657

[terryjreedy]

On Windows with 3.13.7, 3.14.0rc3 (3.14.0 due maybe today in a week), and fresh local 3.15.0a0, the 'bad' expressions run like the good ones. Did you use the most recent versions on Linux?

[tom-pytel]

On Windows with 3.13.7, 3.14.0rc3 (3.14.0 due maybe today in a week), and fresh local 3.15.0a0, the 'bad' expressions run like the good ones. Did you use the most recent versions on Linux?

Confirm problem exists on Ubuntu 22.04.1 with: Python 3.13.7 (main, Sep 30 2025, 10:24:57) [GCC 14.2.0] on linux (docker image).

Also on latest main: Python 3.15.0a0 (heads/main:af01b468661, Oct 2 2025, 15:03:33) [GCC 11.4.0] on linux (my VM).

Didn't try with latest 3.14 but given these fail I assume that will too (that one I confirmed Ubuntu native on not latest 3.14). Though observation that it works on Windows is probably worth something.

[StanFromIreland]

I can also reproduce on Linux with latest main.

[StanFromIreland]

Note that this can be reproduced with non-ascii character, e.g.:

>>> src_bad1 = '''f"{f(a=lambda: 'à'\n)}"'''
>>> for token in tokenize(BytesIO(src_bad1.encode()).readline): print(token)
[...]
Traceback (most recent call last):
  File "<python-input-15>", line 1, in <module>
    for token in tokenize(BytesIO(src_bad1.encode()).readline): print(token)
                 ~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.15/tokenize.py", line 492, in tokenize
    yield from _generate_tokens_from_c_tokenizer(rl_gen.__next__, encoding, extra_tokens=True)
  File "/usr/local/lib/python3.15/tokenize.py", line 582, in _generate_tokens_from_c_tokenizer
    for info in it:
                ^^
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc3 in position 13: unexpected end of data

It seems like characters are getting split?

[tom-pytel]

It seems like characters are getting split?

Note the token dump above, the error occurs on the source line after (and two tokens past) the unicode part has already been returned successfully, so internal parser state corruption?

[tom-pytel]

Its a little screwy, the colon : in the lambda seems to confuse f-string tokenizing (since : is a special symbol in f-strings) because by the time it gets to this loop on the closing } the state is wrong:

cpython/Parser/lexer/lexer.c

Line 129 in 4e7e2dd

for (Py_ssize_t i = 0; i < tok_mode->last_expr_size - tok_mode->last_expr_end; i++) {

last_expr_buffer only contains the expression up to the first explicit \n and the last_expr_size as well but at that point the last_expr_end contains the length from the next line } to the end (which is 3, and this part is not in the buffer or included in size). So this in combination with the perfectly placed unicode character causes a partial copy and a split encoding and the decode error here:

cpython/Parser/lexer/lexer.c

Line 212 in 4e7e2dd

res = PyUnicode_DecodeUTF8(

Confirmed by replacing the : with a + and it parses fine.

[tom-pytel]

Well, found it. The = in the f(a=... sets the in_debug flag which causes the : from the lambda to think its starting a format spec. The unicode in the lambda just highlights the error, but it is there regardless of if there is a decode error thrown or not (by a truly unlikely concurrence of elements), it will set wrong token->metadata if this condition occurs anyway.

Here is a simple fix which passes all tests, do you guys want as a PR?

$ git diff main
diff --git a/Parser/lexer/lexer.c b/Parser/lexer/lexer.c
index 81363cf8e81..a69994e9b3d 100644
--- a/Parser/lexer/lexer.c
+++ b/Parser/lexer/lexer.c
@@ -1376,7 +1376,7 @@ tok_get_normal_mode(struct tok_state *tok, tokenizer_mode* current_tok, struct t
         return MAKE_TOKEN(_PyTokenizer_syntaxerror(tok, "invalid non-printable character U+%04X", c));
     }
 
-    if( c == '=' && INSIDE_FSTRING_EXPR(current_tok)) {
+    if( c == '=' && INSIDE_FSTRING_EXPR_AT_TOP(current_tok)) {
         current_tok->in_debug = 1;
     }
 
diff --git a/Parser/lexer/state.h b/Parser/lexer/state.h
index 5e8cac7249b..877127125a7 100644
--- a/Parser/lexer/state.h
+++ b/Parser/lexer/state.h
@@ -9,6 +9,8 @@
 
 #define INSIDE_FSTRING(tok) (tok->tok_mode_stack_index > 0)
 #define INSIDE_FSTRING_EXPR(tok) (tok->curly_bracket_expr_start_depth >= 0)
+#define INSIDE_FSTRING_EXPR_AT_TOP(tok) \
+    (tok->curly_bracket_depth - tok->curly_bracket_expr_start_depth == 1)
 
 enum decoding_state {
     STATE_INIT,

[tom-pytel]

From a previous issue I understand you guys own this part? @pablogsal @lysnikolaou

[pablogsal]

Here is a simple fix which passes all tests, do you guys want as a PR?

Thanks a lot for the issue and the reasearch! A PR would be great! Please tag me in and ensure to add a test to test_fstring.py