[not crashing] runtime error in tokenizer (pointer index expression overflow)

[qexat]

Bug report

Bug description:

Hello! I am not really sure how to qualify this. It is technically a bug but I assume it's only visible with pydebug builds.

I searched in the issue tracker related issues with keywords like tokenizer and runtime error but I could not find anything.

configure flags

--enable-optimizations --with-valgrind --enable-profiling \
    --enable-pystats --with-pydebug --with-trace-refs \
    --with-assertions --with-lto=full --with-undefined-behavior-sanitizer

Steps to reproduce

Assuming that your build was made with the same flags as mine:

1. Open the REPL
2. Type any function name followed by an opening parenthesis and a backslash, e.g. sum(\ and press enter
3. On the next line, type any expression and press enter

A message in the following form has now been printed:

Parser/tokenizer/file_tokenizer.c:255: runtime error: pointer index expression with base <address> overflowed to <address>

with <address> being replaced by some actual addresses.

Few notes:

• it only happens once ; re-doing the steps will not print the message again (until the REPL is exited and re-opened)
• it does not crash the interpreter

Config

Python version: 3.13.0a2+ (heads/main:c2e8298eba)
OS: EndeavourOS 2022.08.28 64-bit (Linux 6.6.9-arch1-1)
Architecture: x86_64

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

[qexat]

Turns out the function is not required: the key seems to be the combination (\ followed by a newline (and an expression afterwards).

[StanFromIreland]

Unable to reproduce, all I get is SyntaxError: '(' was never closed.

[sergey-miryanov]

I reproduced it, but in a different place on the current main:

./python -X dev -m test test_import                                                             
Using random seed: 3891375552
0:00:00 load avg: 0.00 Run 1 test sequentially in a single process
0:00:00 load avg: 0.00 [1/1] test_import
Parser/lexer/buffer.c:30:32: runtime error: pointer index expression with base 0x564349cfed00 overflowed to 0xfffffffffff4fb00
Parser/lexer/buffer.c:31:43: runtime error: pointer index expression with base 0x564349cfed00 overflowed to 0xfffffffffff4fb00
0:00:08 load avg: 0.16 [1/1] test_import passed

== Tests result: SUCCESS ==

1 test OK.

Total duration: 8.7 sec
Total tests: run=117 skipped=3
Total test files: run=1/1
Result: SUCCESS

Built with --with-pydebug --with-undefined-behavior-sanitizer

cc @efimov-mikhail