Skip to content

gh-113547 cases generator lexer: dedent > 0 + escape sequence #113546

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 7 commits into
base: main
Choose a base branch
from
Open

gh-113547 cases generator lexer: dedent > 0 + escape sequence #113546

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
92ecf9c
dedent > 0 + escape sequence
ksfi Dec 28, 2023
0871ee8
typing
ksfi Dec 28, 2023
a041e52
less typing
ksfi Dec 28, 2023
0ad22c3
📜🤖 Added by blurb_it.
blurb-it[bot] Dec 28, 2023
33cb7a0
simpler indent expression + tests
ksfi Jan 7, 2024
452a167
Merge branch 'main' of https://github.com/ksfi/cpython
ksfi Jan 7, 2024
d7e0295
change name test_lexer.py + dead code
ksfi Oct 3, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions Misc/NEWS.d/next/Tools-Demos/2023-12-28-18-12-24.gh-issue-113547.TIwp20.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
TODOs in the cases generator lexer: dedent > 0 case in the to_text function + escape sequence handling
13 changes: 11 additions & 2 deletions Tools/cases_generator/lexer.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -109,7 +109,7 @@ def choice(*opts: str) -> str:
string_char = r"""([^"\\\n]|""" + escape_sequence + ")"
str_re = '"' + string_char + '*"'
STRING = "STRING"
char = r"\'.\'" # TODO: escape sequence
char = r"\'([^'\\]|\\[0-7]{1,3}|\\x[0-9a-fA-F]+|\\.|\\\\)\'"
CHARACTER = "CHARACTER"

comment_re = r"(//.*)|/\*([^*]|\*[^/])*\*/"
Expand Down Expand Up @@ -344,7 +344,16 @@ def to_text(tkns: list[Token], dedent: int = 0) -> str:
if dedent != 0 and tkn.kind == "COMMENT" and "\n" in text:
if dedent < 0:
text = text.replace("\n", "\n" + " " * -dedent)
# TODO: dedent > 0
elif dedent > 0:
temp: list[str] = []
for line in text.split("\n"):
leading_space = len(line) - len(line.lstrip())
if leading_space > dedent:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the regex necessary? Would line = line[min(leading_space, dedent):] work?

line = re.sub(r'(?m)^[ \t]{' + str(dedent) + r'}', '', line)
else:
line = re.sub(r'(?m)^[ \t]{' + str(leading_space) + r'}', '', line)
temp.append(line)
text = "\n".join(temp)
res.append(text)
line, col = tkn.end
return "".join(res)
Expand Down