Unexpected behavior when `IndentationError` raise

[F-park]

Bug report

Bug description:

In python 3.8

>>>     0
  File "<stdin>", line 1
    0
    ^
IndentationError: unexpected indent
>>> import sys, traceback
>>> sys.excepthook = lambda *exc_info: traceback.print_exception(*exc_info)
>>>     0
  File "<stdin>", line 1
    0
    ^
IndentationError: unexpected indent

In python 3.9 ... 3.13.0a3#e7331365b488382d906ce6733ab1349ded49c928^

>>>     0
  File "<stdin>", line 1
    0
IndentationError: unexpected indent
>>> import sys, traceback
>>> sys.excepthook = lambda *exc_info: traceback.print_exception(*exc_info)
>>>     0
  File "<stdin>", line 1
        0
    ^
IndentationError: unexpected indent

In python 3.13.0a3#e7331365b488382d906ce6733ab1349ded49c928 ... main

>>>     0
  File "<stdin>", line 1
        0
    ^
IndentationError: unexpected indent

CPython versions tested on:

3.9 ... 3.13.0a3#e7331365b488382d906ce6733ab1349ded49c928^

Operating systems tested on:

Windows

[gaogaotiantian]

I'm a bit confused here. The exception looks normal to me. Could you post what you think is incorrect?

[cyx2015s]

The underlying up arrow mark points at different location between different versions.

---

Edit:

He might meant that the default error message and trace_back.print_execption results are different

[cyx2015s]

Python 3.8:

Python 3.9+:

These two images are from @F-park .

[terryjreedy]

@pablogsal The issue is the '^' location after an IndentationError. In 3.13.0a2 and current main, there is no caret. Is this an intentional fix of it being misplaced? Or an intentional change due to color coding syntax errors in the code line itself? For the latter, coloring the improper space before the first character, rather than the first character itself, would seem appropriate.

@F-park You linked the place where a parser indentation error is raised. More relevant is the code that calculated whether and how to generate an error location symbol line.

[F-park]

I'm a bit confused here. The exception looks normal to me. Could you post what you think is incorrect?

In python 3.8

>>>     0
  File "<stdin>", line 1
    0
    ^
IndentationError: unexpected indent
>>> import sys, traceback
>>> sys.excepthook = lambda *exc_info: traceback.print_exception(*exc_info)
>>>     0
  File "<stdin>", line 1
    0
    ^
IndentationError: unexpected indent

In python 3.9+

>>>     0
  File "<stdin>", line 1
    0
IndentationError: unexpected indent
>>> import sys, traceback
>>> sys.excepthook = lambda *exc_info: traceback.print_exception(*exc_info)
>>>     0
  File "<stdin>", line 1
        0
    ^
IndentationError: unexpected indent

I consider in python3.9+, the correct output should be

>>>     0
  File "<stdin>", line 1
    0
^
IndentationError: unexpected indent

[gaogaotiantian]

The direct cause of this is the parser change in 3.9, which determines the column number of the actual error. In 3.8, the parser thinks the error is at the first character, which works fine with the carets. From 3.9, the parser believes the error is at the space before the first character, but the exception formatter will dedent the line with the syntax error, which leaves no space for carets and therefore no carets are printed. In repl I believe there are extra spaces so the carets are available again.

I guess the question here is - what's the expected behavior?

  File "/home/gaogaotiantian/programs/mycpython/scrabble.py", line 3
     b = 2
    ^
IndentationError: unexpected indent

Is this better? Notice that the source line b = 2 is indented by an extra space for the caret.

[gaogaotiantian]

The current behavior is intentional from 15bc9ab and maybe @gvanrossum can share thoughts on this? Do you believe suppressing the caret for the IndentationError is the preferred behavior?

[F-park]

The bug fixed in e733136 , but it not backport to 3.9 ... 3.12.

[gvanrossum]

Assuming the fix won't be backported (too fiddly, not very important), do you still need my opinion? I have to admit that I don't recall the details around 15bc9ab, so it would take considerable time for me to reconstruct what's going on here and form another opinion...

[gaogaotiantian]

It's not "fixed". It happens to not appear on REPL, but the same issue happens with scripts.

It's actually a pretty simple choice (to save you from recollections):

    print("line with unexpected indent")

Do you think we should do:

  File "/home/gaogaotiantian/programs/mycpython/example.py", line 1
    print("hello")
IndentationError: unexpected indent

Or

  File "/home/gaogaotiantian/programs/mycpython/example.py", line 1
    print("hello")
   ^
IndentationError: unexpected indent

We could have other possible indentations for the actual line in the traceback, the core question is - do you think there should be a caret pointing to the space before the print (or maybe to the print)?

[gvanrossum]

Comparing various indentation errors in the REPL:

(.venv) ~/cpython$ python3.13
Python 3.13.0a4 (v3.13.0a4:9d34f60783, Feb 15 2024, 10:52:45) [Clang 13.0.0 (clang-1300.0.29.30)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> def f():
... pass
  File "<stdin>", line 2
    pass
    ^^^^
IndentationError: expected an indented block after function definition on line 1
>>> def f():
...     pass
...   pass
  File "<stdin>", line 3
    pass
        ^
IndentationError: unindent does not match any outer indentation level
>>> def f():
...     pass
...       pass
  File "<stdin>", line 3
    pass
IndentationError: unexpected indent
>>> def f():
...     pass
... pass
  File "<stdin>", line 3
    pass
    ^^^^
SyntaxError: invalid syntax
>>> 

It looks like the "unexpected indent" one is the odd one out that doesn't print the caret. So it looks like a mistake.

But probably we should ask @pablogsal, who I'm sure will have an opinion. (Hi, Pablo!)

[gaogaotiantian]

Yes, the reason "unexpected indent" is the odd one is the caret is before the actual statement. The statement is dedented in the traceback so the "supposed position" of the caret is negative, which will be suppressed.

If we want to change the behavior, we need to decide where we should put the caret - from the bug picture, there are extra spaces for the caret, but the traceback formatter probably does not know that. The indentation is inserted by multiple layers and when traceback formatter calculates the position of the caret, it believes there's no position for it.

[gvanrossum]

It should be put at the start of the line, I.e. below the first token. I have no idea where that code is any more, sorry. (It’s always been fiddly, and has probably only gotten more so with various improvements over the past 5 years.)

[gaogaotiantian]

No worries, I found the code that produces this error and kind of figured out what happened, so no code tracking work for you. However, the new parser (we changed to in 3.9) believes the parsing error is before the first token - on the last space before the character, that's why the behavior changed in 3.9. If we want to put the caret on the first token, we need to set special rule for this, overriding what the parser generates.