You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
"""This function is just a convenience wrapper around ast.parse, with default flags useful to Mypy.
143
-
It also incorporates a hack to accomodate `# mypy: ignore` comments, which are treated by mypy as `# type: ignore` comments.
143
+
It also handles `# mypy: ignore` comments, by turning them into `# type: ignore` comments.
144
144
"""
145
145
# Hack to support "mypy: ignore" comments until the builtin compile function changes to allow us to detect it otherwise:
146
-
# (Note: completely distinct from https://mypy.readthedocs.io/en/stable/inline_config.html ; see also, util.get_mypy_comments in this codebase)
146
+
# (Note: completely distinct from https://mypy.readthedocs.io/en/stable/inline_config.html ; see also, util.get_mypy_comments in this codebase.)
147
147
148
148
# We make the substitution in comments, and to find those comments we use Python's `tokenize`.
149
149
# https://docs.python.org/3/library/tokenize.html has a big red **Warning:**
150
-
# Note that the functions in this module are only designed to parse syntactically valid Python code (code that does not raise when parsed using ast.parse()). The behavior of the functions in this module is **undefined** when providing invalid Python code and it can change at any point.
150
+
# Note that the functions in this module are only designed to parse syntactically valid Python
151
+
# code (code that does not raise when parsed using ast.parse()). The behavior of the functions
152
+
# in this module is **undefined** when providing invalid Python code and it can change at any point.
151
153
# So, we cannot rely on roundtrip behavior in tokenize iff ast.parse would throw when given `source`.
152
-
# The simplest way to deal with that is just to call ast.parse twice, once before and once after. So, we do that.
154
+
# The simplest way to deal with that is just to call ast.parse twice, once before and once after!
# We do a workaround for a roundtripping error https://github.com/python/cpython/issues/125008
164
-
# An error that was introduced in 3.12.3 (https://github.com/python/cpython/blob/v3.12.3/Lib/tokenize.py#L205) and fixed in 3.12.8 (not out yet at time of writing but it's probably https://github.com/python/cpython/blob/v3.12.8/Lib/tokenize.py#L203 or thereabouts).
165
+
# We do a workaround for a roundtripping error (https://github.com/python/cpython/issues/125008)
166
+
# that was introduced in 3.12.3 (https://github.com/python/cpython/blob/v3.12.3/Lib/tokenize.py#L205)
167
+
# and fixed in 3.12.8 (https://github.com/python/cpython/blob/v3.12.8/Lib/tokenize.py#L205).
165
168
# Luckily, it was caught before the first official (non-rc) release of python 3.13.
# Could the workaround ever come back to bite us? I confess I don't know enough about FSTRING_MIDDLE to say no for certain, even though I have tried to examine all relevant cases.
170
+
# This is a gnarly list comprehension, but that's basically just how untokenize is supposed to work.
168
171
source=tokenize.untokenize(
169
172
(
170
173
t,
@@ -182,11 +185,11 @@ def p() -> AST:
182
185
elses
183
186
)
184
187
),
185
-
*_, # this improves whitespace roundtripping (possibly always making it perfect, for this usecase?)
188
+
*_, # Including this bit improves whitespace roundtripping (possibly always making it perfect).
186
189
)
187
190
fort, s, *_intokens
188
191
)
189
-
returnp()
192
+
returnp()# `source` has changed, so this result is different than the first call.
0 commit comments