Skip to content

Commit 8b9ab48

Browse files
ntBrentBre
authored
Fix syntax error false positives for escapes and quotes in f-strings (#20867)
Summary -- Fixes #20844 by refining the unsupported syntax error check for [PEP 701] f-strings before Python 3.12 to allow backslash escapes and escaped outer quotes in the format spec part of f-strings. These are only disallowed within the f-string expression part on earlier versions. Using the examples from the PR: ```pycon >>> f"{1:\x64}" '1' >>> f"{1:\"d\"}" Traceback (most recent call last): File "<stdin>", line 1, in <module> ValueError: Invalid format specifier '"d"' for object of type 'int' ``` Note that the second case is a runtime error, but this is actually avoidable if you override `__format__`, so despite being pretty weird, this could actually be a valid use case. ```pycon >>> class C: ... def __format__(*args, **kwargs): return "<C>" ... >>> f"{C():\"d\"}" '<C>' ``` At first I thought narrowing the range we check to exclude the format spec would only work for escapes, but it turns out that cases like `f"{1:""}"` are already covered by an existing `ParseError`, so we can just narrow the range of both our escape and quote checks. Our comment check also seems to be working correctly because it's based on the actual tokens. A case like [this](https://play.ruff.rs/9f1c2ff2-cd8e-4ad7-9f40-56c0a524209f): ```python f"""{1:# }""" ``` doesn't include a comment token, instead the `#` is part of an `InterpolatedStringLiteralElement`. Test Plan -- New inline parser tests [PEP 701]: https://peps.python.org/pep-0701/
1 parent 8817ea5 commit 8b9ab48

File tree

9 files changed

+317
-62
lines changed

9 files changed

+317
-62
lines changed

‎crates/ruff_python_formatter/resources/test/fixtures/ruff/expression/fstring.py‎

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -706,8 +706,6 @@
706706
f'{1: abcd "{1}" }'
707707
f'{1: abcd "{'aa'}" }'
708708
f'{1=: "abcd {'aa'}}'
709-
# FIXME(brent) This should not be a syntax error on output. The escaped quotes are in the format
710-
# spec, which is valid even before 3.12.
711709
f'{x:a{z:hy "user"}} \'\'\''
712710

713711
# Changing the outer quotes is fine because the format-spec is in a nested expression.

‎crates/ruff_python_formatter/tests/snapshots/format@expression__fstring.py.snap‎

Lines changed: 0 additions & 54 deletions
Original file line numberDiff line numberDiff line change
@@ -712,8 +712,6 @@ f'{1:hy "user"}'
712712
f'{1: abcd "{1}" }'
713713
f'{1: abcd "{'aa'}" }'
714714
f'{1=: "abcd {'aa'}}'
715-
# FIXME(brent) This should not be a syntax error on output. The escaped quotes are in the format
716-
# spec, which is valid even before 3.12.
717715
f'{x:a{z:hy "user"}} \'\'\''
718716

719717
# Changing the outer quotes is fine because the format-spec is in a nested expression.
@@ -1536,8 +1534,6 @@ f'{1:hy "user"}'
15361534
f'{1: abcd "{1}" }'
15371535
f'{1: abcd "{"aa"}" }'
15381536
f'{1=: "abcd {'aa'}}'
1539-
# FIXME(brent) This should not be a syntax error on output. The escaped quotes are in the format
1540-
# spec, which is valid even before 3.12.
15411537
f"{x:a{z:hy \"user\"}} '''"
15421538

15431539
# Changing the outer quotes is fine because the format-spec is in a nested expression.
@@ -2365,8 +2361,6 @@ f'{1:hy "user"}'
23652361
f'{1: abcd "{1}" }'
23662362
f'{1: abcd "{"aa"}" }'
23672363
f'{1=: "abcd {'aa'}}'
2368-
# FIXME(brent) This should not be a syntax error on output. The escaped quotes are in the format
2369-
# spec, which is valid even before 3.12.
23702364
f"{x:a{z:hy \"user\"}} '''"
23712365

23722366
# Changing the outer quotes is fine because the format-spec is in a nested expression.
@@ -2418,30 +2412,6 @@ print(f"{ {}, 1 }")
24182412
24192413
24202414
### Unsupported Syntax Errors
2421-
error[invalid-syntax]: Cannot use an escape sequence (backslash) in f-strings on Python 3.10 (syntax was added in Python 3.12)
2422-
--> fstring.py:764:19
2423-
|
2424-
762 | # FIXME(brent) This should not be a syntax error on output. The escaped quotes are in the format
2425-
763 | # spec, which is valid even before 3.12.
2426-
764 | f"{x:a{z:hy \"user\"}} '''"
2427-
| ^
2428-
765 |
2429-
766 | # Changing the outer quotes is fine because the format-spec is in a nested expression.
2430-
|
2431-
warning: Only accept new syntax errors if they are also present in the input. The formatter should not introduce syntax errors.
2432-
2433-
error[invalid-syntax]: Cannot use an escape sequence (backslash) in f-strings on Python 3.10 (syntax was added in Python 3.12)
2434-
--> fstring.py:764:13
2435-
|
2436-
762 | # FIXME(brent) This should not be a syntax error on output. The escaped quotes are in the format
2437-
763 | # spec, which is valid even before 3.12.
2438-
764 | f"{x:a{z:hy \"user\"}} '''"
2439-
| ^
2440-
765 |
2441-
766 | # Changing the outer quotes is fine because the format-spec is in a nested expression.
2442-
|
2443-
warning: Only accept new syntax errors if they are also present in the input. The formatter should not introduce syntax errors.
2444-
24452415
error[invalid-syntax]: Cannot reuse outer quote character in f-strings on Python 3.10 (syntax was added in Python 3.12)
24462416
--> fstring.py:178:8
24472417
|
@@ -2452,27 +2422,3 @@ error[invalid-syntax]: Cannot reuse outer quote character in f-strings on Python
24522422
179 | f"foo {'"bar"'}"
24532423
|
24542424
warning: Only accept new syntax errors if they are also present in the input. The formatter should not introduce syntax errors.
2455-
2456-
error[invalid-syntax]: Cannot reuse outer quote character in f-strings on Python 3.10 (syntax was added in Python 3.12)
2457-
--> fstring.py:773:14
2458-
|
2459-
771 | f'{1=: "abcd \'\'}' # Don't change the outer quotes, or it results in a syntax error
2460-
772 | f"{1=: abcd \'\'}" # Changing the quotes here is fine because the inner quotes aren't the opposite quotes
2461-
773 | f"{1=: abcd \"\"}" # Changing the quotes here is fine because the inner quotes are escaped
2462-
| ^
2463-
774 | # Don't change the quotes in the following cases:
2464-
775 | f'{x=:hy "user"} \'\'\''
2465-
|
2466-
warning: Only accept new syntax errors if they are also present in the input. The formatter should not introduce syntax errors.
2467-
2468-
error[invalid-syntax]: Cannot reuse outer quote character in f-strings on Python 3.10 (syntax was added in Python 3.12)
2469-
--> fstring.py:764:14
2470-
|
2471-
762 | # FIXME(brent) This should not be a syntax error on output. The escaped quotes are in the format
2472-
763 | # spec, which is valid even before 3.12.
2473-
764 | f"{x:a{z:hy \"user\"}} '''"
2474-
| ^
2475-
765 |
2476-
766 | # Changing the outer quotes is fine because the format-spec is in a nested expression.
2477-
|
2478-
warning: Only accept new syntax errors if they are also present in the input. The formatter should not introduce syntax errors.

‎crates/ruff_python_parser/resources/inline/err/nested_quote_in_format_spec_py312.py‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
# parse_options: {"target-version": "3.12"}
2+
f"{1:""}" # this is a ParseError on all versions

‎crates/ruff_python_parser/resources/inline/ok/non_nested_quote_in_format_spec_py311.py‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
# parse_options: {"target-version": "3.11"}
2+
f"{1:''}" # but this is okay on all versions

‎crates/ruff_python_parser/resources/inline/ok/pep701_f_string_py311.py‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,3 +5,5 @@
55
f"""{f'''# before expression {f'# aro{f"#{1+1}#"}und #'}'''} # after expression"""
66
f"escape outside of \t {expr}\n"
77
f"test\"abcd"
8+
f"{1:\x64}" # escapes are valid in the format spec
9+
f"{1:\"d\"}" # this also means that escaped outer quotes are valid

‎crates/ruff_python_parser/src/parser/expression.rs‎

Lines changed: 21 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1571,6 +1571,8 @@ impl<'src> Parser<'src> {
15711571
// f"""{f'''# before expression {f'# aro{f"#{1+1}#"}und #'}'''} # after expression"""
15721572
// f"escape outside of \t {expr}\n"
15731573
// f"test\"abcd"
1574+
// f"{1:\x64}" # escapes are valid in the format spec
1575+
// f"{1:\"d\"}" # this also means that escaped outer quotes are valid
15741576

15751577
// test_err pep701_f_string_py311
15761578
// # parse_options: {"target-version": "3.11"}
@@ -1586,6 +1588,13 @@ impl<'src> Parser<'src> {
15861588
// f"""{f"""{x}"""}""" # mark the whole triple quote
15871589
// f"{'\n'.join(['\t', '\v', '\r'])}" # multiple escape sequences, multiple errors
15881590

1591+
// test_err nested_quote_in_format_spec_py312
1592+
// # parse_options: {"target-version": "3.12"}
1593+
// f"{1:""}" # this is a ParseError on all versions
1594+
1595+
// test_ok non_nested_quote_in_format_spec_py311
1596+
// # parse_options: {"target-version": "3.11"}
1597+
// f"{1:''}" # but this is okay on all versions
15891598
let range = self.node_range(start);
15901599

15911600
if !self.options.target_version.supports_pep_701()
@@ -1594,22 +1603,29 @@ impl<'src> Parser<'src> {
15941603
let quote_bytes = flags.quote_str().as_bytes();
15951604
let quote_len = flags.quote_len();
15961605
for expr in elements.interpolations() {
1597-
for slash_position in memchr::memchr_iter(b'\\', self.source[expr.range].as_bytes())
1598-
{
1606+
// We need to check the whole expression range, including any leading or trailing
1607+
// debug text, but exclude the format spec, where escapes and escaped, reused quotes
1608+
// are allowed.
1609+
let range = expr
1610+
.format_spec
1611+
.as_ref()
1612+
.map(|format_spec| TextRange::new(expr.start(), format_spec.start()))
1613+
.unwrap_or(expr.range);
1614+
for slash_position in memchr::memchr_iter(b'\\', self.source[range].as_bytes()) {
15991615
let slash_position = TextSize::try_from(slash_position).unwrap();
16001616
self.add_unsupported_syntax_error(
16011617
UnsupportedSyntaxErrorKind::Pep701FString(FStringKind::Backslash),
1602-
TextRange::at(expr.range.start() + slash_position, '\\'.text_len()),
1618+
TextRange::at(range.start() + slash_position, '\\'.text_len()),
16031619
);
16041620
}
16051621

16061622
if let Some(quote_position) =
1607-
memchr::memmem::find(self.source[expr.range].as_bytes(), quote_bytes)
1623+
memchr::memmem::find(self.source[range].as_bytes(), quote_bytes)
16081624
{
16091625
let quote_position = TextSize::try_from(quote_position).unwrap();
16101626
self.add_unsupported_syntax_error(
16111627
UnsupportedSyntaxErrorKind::Pep701FString(FStringKind::NestedQuote),
1612-
TextRange::at(expr.range.start() + quote_position, quote_len),
1628+
TextRange::at(range.start() + quote_position, quote_len),
16131629
);
16141630
}
16151631
}

‎crates/ruff_python_parser/tests/snapshots/invalid_syntax@nested_quote_in_format_spec_py312.py.snap‎

Lines changed: 90 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,90 @@
1+
---
2+
source: crates/ruff_python_parser/tests/fixtures.rs
3+
input_file: crates/ruff_python_parser/resources/inline/err/nested_quote_in_format_spec_py312.py
4+
---
5+
## AST
6+
7+
```
8+
Module(
9+
ModModule {
10+
node_index: NodeIndex(None),
11+
range: 0..94,
12+
body: [
13+
Expr(
14+
StmtExpr {
15+
node_index: NodeIndex(None),
16+
range: 44..53,
17+
value: FString(
18+
ExprFString {
19+
node_index: NodeIndex(None),
20+
range: 44..53,
21+
value: FStringValue {
22+
inner: Concatenated(
23+
[
24+
FString(
25+
FString {
26+
range: 44..50,
27+
node_index: NodeIndex(None),
28+
elements: [
29+
Interpolation(
30+
InterpolatedElement {
31+
range: 46..49,
32+
node_index: NodeIndex(None),
33+
expression: NumberLiteral(
34+
ExprNumberLiteral {
35+
node_index: NodeIndex(None),
36+
range: 47..48,
37+
value: Int(
38+
1,
39+
),
40+
},
41+
),
42+
debug_text: None,
43+
conversion: None,
44+
format_spec: Some(
45+
InterpolatedStringFormatSpec {
46+
range: 49..49,
47+
node_index: NodeIndex(None),
48+
elements: [],
49+
},
50+
),
51+
},
52+
),
53+
],
54+
flags: FStringFlags {
55+
quote_style: Double,
56+
prefix: Regular,
57+
triple_quoted: false,
58+
},
59+
},
60+
),
61+
Literal(
62+
StringLiteral {
63+
range: 50..53,
64+
node_index: NodeIndex(None),
65+
value: "}",
66+
flags: StringLiteralFlags {
67+
quote_style: Double,
68+
prefix: Empty,
69+
triple_quoted: false,
70+
},
71+
},
72+
),
73+
],
74+
),
75+
},
76+
},
77+
),
78+
},
79+
),
80+
],
81+
},
82+
)
83+
```
84+
## Errors
85+
86+
|
87+
1 | # parse_options: {"target-version": "3.12"}
88+
2 | f"{1:""}" # this is a ParseError on all versions
89+
| ^ Syntax Error: f-string: expecting '}'
90+
|

‎crates/ruff_python_parser/tests/snapshots/valid_syntax@non_nested_quote_in_format_spec_py311.py.snap‎

Lines changed: 77 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,77 @@
1+
---
2+
source: crates/ruff_python_parser/tests/fixtures.rs
3+
input_file: crates/ruff_python_parser/resources/inline/ok/non_nested_quote_in_format_spec_py311.py
4+
---
5+
## AST
6+
7+
```
8+
Module(
9+
ModModule {
10+
node_index: NodeIndex(None),
11+
range: 0..90,
12+
body: [
13+
Expr(
14+
StmtExpr {
15+
node_index: NodeIndex(None),
16+
range: 44..53,
17+
value: FString(
18+
ExprFString {
19+
node_index: NodeIndex(None),
20+
range: 44..53,
21+
value: FStringValue {
22+
inner: Single(
23+
FString(
24+
FString {
25+
range: 44..53,
26+
node_index: NodeIndex(None),
27+
elements: [
28+
Interpolation(
29+
InterpolatedElement {
30+
range: 46..52,
31+
node_index: NodeIndex(None),
32+
expression: NumberLiteral(
33+
ExprNumberLiteral {
34+
node_index: NodeIndex(None),
35+
range: 47..48,
36+
value: Int(
37+
1,
38+
),
39+
},
40+
),
41+
debug_text: None,
42+
conversion: None,
43+
format_spec: Some(
44+
InterpolatedStringFormatSpec {
45+
range: 49..51,
46+
node_index: NodeIndex(None),
47+
elements: [
48+
Literal(
49+
InterpolatedStringLiteralElement {
50+
range: 49..51,
51+
node_index: NodeIndex(None),
52+
value: "''",
53+
},
54+
),
55+
],
56+
},
57+
),
58+
},
59+
),
60+
],
61+
flags: FStringFlags {
62+
quote_style: Double,
63+
prefix: Regular,
64+
triple_quoted: false,
65+
},
66+
},
67+
),
68+
),
69+
},
70+
},
71+
),
72+
},
73+
),
74+
],
75+
},
76+
)
77+
```

0 commit comments

Comments
 (0)