Incorrect SyntaxError attributes when raised by `compile`

[mrigankpawagi]

Bug report

Bug description:

This is probably a problem with compile, and I will demonstrate it through ast.parse. Consider the following code in a file f.py.

import ast

code = "x y"
ast.parse(code, filename='f.py')

This correctly raises a SyntaxError with lineno = 1 and offset = 3 since the error is at the beginning of y in the code. However, the traceback points into the file f.py itself, instead of the string code. Notice that it points to the third column of the first line of f.py.

  File "f.py", line 1
    import ast
      ^
SyntaxError: invalid syntax

This does not happen for any other string given as filename to ast.parse. In general, the traceback seems to point with the "correct" lineno and offset to the code in the file itself instead of code given to ast.parse. However, the problem is slightly more nuanced. If the offset in the string given to ast.parse is more than the length of the corresponding line in f.py, the offset attribute of the SyntaxError raised is incorrectly set to the length of the corresponding line in f.py plus two. Consider the following code in f.py.

import ast

code = "                                     x y"
ast.parse(code, filename='f.py')

This raises the following exception.

  File "f.py", line 1
    import ast
              ^
IndentationError: unexpected indent

On closer inspection, it turns out that the offset of the exception raised is 12 (note that the length of "import ast" is 10) instead of the correct 37 which is observed when filename is set to anything other than f.py. However, something more interesting happens when the number of lines in the code string is more than the number of lines in the file f.py.

import ast

code = "\n\n\n\nx y"
ast.parse(code, filename='f.py')

The problem goes away!

  File "f.py", line 5
    x y
      ^
SyntaxError: invalid syntax

We even see the correct offset.

CPython versions tested on:

3.11, 3.12, 3.13

Operating systems tested on:

Linux

(Edit) Incorrect text attribute

There is something more going on here. The following issue can be seen in 3.11 and 3.12 but not in 3.13. Consider the following f.py.

import ast

with open('temp.py') as f:
    code = f.read()
ast.parse(code, filename='temp.py')

temp.py is as follows.

"""                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             """; x """                                                                                                                                                         """; foobar

This leads to the following traceback.

  File "temp.py", line 1
    foobar
          ^
SyntaxError: invalid syntax

Besides the fact that the offset is incorrectly set to 8 (instead of 840), the text is incorrectly set to foobar (on 3.13, the entire file contents of temp.py are present in text). This problem goes away if filename is set to anything else. The problem also goes away if the contents of temp.py are shorter!

[terryjreedy]

In general, exceptions include a 'traceback' object which constitutes a linked list of stack frames. The only connection from stackframe to source is a file name. An exception not caught by code is caught by the Python runtime, which prints a default traceback display. The traceback printer uses filenames to get source lines when available.

Your example mis-represents the content of file.py. Mis-information in, mis-information out. We could close as "won't fix".

Could we do better? SyntaxErrors include a .text attribute. Could the default traceback printer ignore the filename when attempting to retrieve the error source line and get e.text instead? Could this simplify any of the current workarounds for getting SyntaxError source lines?

[mrigankpawagi]

Thanks for your comment, @terryjreedy. Please correct me here: since I set filename='f.py', the traceback printer reads f.py and supplies that source to the exception's text field. How does the string code return after the end of f.py in the case where I added newlines?

The documentation has little information on what filename does (from what I can see, it says that filename should have the name of the source of the code — this will make traceback easier to understand).

[mrigankpawagi]

Also even if the traceback shows mis-information, it is not immediately clear to me why sometimes the offset value is incorrect.

[picnixz]

IIRC, the "filename" is not the name of the file containing the compile call but the "file" that contains the code being compiled. In your case, the code that is being compiled is "x y" (there is no "file" in this case). As you observed, changing the name of the file makes it work.

Now, if you use a filename that exists and that has been somehow cached by linecache, I think you'll also get an incorrect traceback report (but I didn't check this).

[mrigankpawagi]

I see. I understand the misinformation in, misinformation out argument, but I think that even if the traceback is not right, the exception object should provide correct offset values. This information has downstream effects, for example, in code quality tools like pyflakes.

[mrigankpawagi]

@picnixz @terryjreedy I have updated the issue with the following additions. Again, your insights will be very helpful.

Incorrect text attribute

There is something more going on here. The following issue can be seen in 3.11 and 3.12 but not in 3.13. Consider the following f.py.

import ast

with open('temp.py') as f:
    code = f.read()
ast.parse(code, filename='temp.py')

temp.py is as follows.

"""                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             """; x """                                                                                                                                                         """; foobar

This leads to the following traceback.

  File "temp.py", line 1
    foobar
          ^
SyntaxError: invalid syntax

Besides the fact that the offset is incorrectly set to 8 (instead of 840), the text is incorrectly set to foobar (on 3.13, the entire file contents of temp.py are present in text). This problem goes away if filename is set to anything else. The problem also goes away if the contents of temp.py are shorter (by at least 8 characters)!

I am not sure if this is also misinformation (after all the code really is from temp.py) — and the fact that this is fixed in 3.13 is also very interesting... I find this behaviour very unexpected particularly because it comes up only for long strings.

[picnixz]

Let's start with something simple. Assume that we have:

"""..."""; x """..."""; foobar

Besides the fact that the offset is incorrectly set to 8 (instead of 840), the text is incorrectly set to foobar (on 3.13, the entire file contents of temp.py are present in text).

Now, it appears that the offset entirely depends on the filename that is being passed, which is something I find a bit surprising though. So I can try to investigate. For instance, on 3.11.3 (Windows), in the REPL:

import ast

code = '"""..."""; x """abc"""; foobar'

try:
	ast.parse(code)
except SyntaxError as e:
	assert e.offset == 14

versus:

import ast
import fnmatch

code = '"""..."""; x """..."""; foobar'

try:
	ast.parse(code, filename=fnmatch.__file__)
except SyntaxError as e:
	assert e.offset == 14

Both report the message identitically. In the first case, it's """..."""; x ["""abc"""]; foobar being highlighted while in the second it's """Filename m[atching w]ith shell patterns. (this is the first docstring of fnmatch.py) (the part with ^^^^^ below is in brackets).

Now, I tried replacing my ... with 800 whitespaces and the offset changed to 811 in the first case (which is what expected) and 43 in the second case (which is not what I expected). Note that 43 in this case is the minimum between the line length of the reported error (first line """Filename matching with shell patterns. of fnmatch.__doc__ has 43 characters) and the actual "error" that we obtained without the filename (so min(43, 811) = 43 I think). Now, in 3.13+, we might perhaps not compute that minimum anymore, but I don't have access to a machine for testing this now.

@pablogsal @JelleZijlstra can you confirm the analysis? or say something on this topic.

---

The fact that this is fixed in 3.13 is also very interesting

If you can, can you bisect the commit for that? if not I have some guesses:

• It was fixed because of the new REPL.
• It was fixed because we fixed something in the AST. Or because how we parse stuff.
• It was fixed because we fixed linecache recently (I know we fixed something but I don't know if it could have helped).
• Did we change tokenization in some way that made it behave differently? and report offsets differently?

[picnixz]

I'm removing the pending label because I would like to know whether it's really an issue or not and whether 3.12 should or can be fixed.

[mrigankpawagi]

Thanks for the analysis, @picnixz. I will try to look into the commits and share on this thread if I find something.

In my example, I set filename to the actual temp.py (in your example, you point to a different file altogether) which does contain the same number of characters on its first line (after all the code is from that very file itself). Your guess regarding the minimum may not directly explain this. What do you think?

[picnixz]

On Python 3.9, I also have an offset of 14 (or 808 depending on the number of whitespaces I put) when I use the real filename. When I use the fake filename (fnmatch.__file__), I get 808 on Python 3.9 but 43 on Python 3.12. So something is happening indeed.