Skip to content

Commit b6554ad

Browse files
committed
Fix parser translator tokens for backslashes in single-quoted strings and word arrays
These are not line continuations. They either should be taken literally, or allow the word array to contain the following whitespace (newlines in this case) Before: ``` 0...1: tSTRING_BEG => "'" 1...12: tSTRING_CONTENT => "foobar\\\n" 12...16: tSTRING_CONTENT => "baz\n" 16...17: tSTRING_END => "'" 17...18: tNL => nil ``` After: ``` 0...1: tSTRING_BEG => "'" 1...6: tSTRING_CONTENT => "foo\\\n" 6...12: tSTRING_CONTENT => "bar\\\n" 12...16: tSTRING_CONTENT => "baz\n" 16...17: tSTRING_END => "'" 17...18: tNL => nil ```
1 parent 6bfe3a5 commit b6554ad

File tree

4 files changed

+123
-95
lines changed

4 files changed

+123
-95
lines changed

lib/prism/translation/parser/lexer.rb

Lines changed: 31 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -377,12 +377,20 @@ def to_a
377377

378378
lines.each.with_index do |line, index|
379379
chomped_line = line.chomp
380-
381-
# When the line ends with an odd number of backslashes, it must be a line continuation.
382-
if chomped_line[/\\{1,}\z/]&.length&.odd?
383-
chomped_line.delete_suffix!("\\")
384-
current_line << chomped_line
385-
adjustment += 2
380+
backslash_count = chomped_line[/\\{1,}\z/]&.length || 0
381+
is_interpolation = interpolation?(quote_stack.last)
382+
is_percent_array = percent_array?(quote_stack.last)
383+
384+
if backslash_count.odd? && (is_interpolation || is_percent_array)
385+
if is_percent_array
386+
# Remove the last backslash, keep potential newlines
387+
current_line << line.sub(/(\\)(\r?\n)\z/, '\2')
388+
adjustment += 1
389+
else
390+
chomped_line.delete_suffix!("\\")
391+
current_line << chomped_line
392+
adjustment += 2
393+
end
386394
# If the string ends with a line continuation emit the remainder
387395
emit = index == lines.count - 1
388396
else
@@ -577,7 +585,13 @@ def unescape_string(string, quote)
577585
# TODO: Implement regexp escaping
578586
return string if quote == "/" || quote.start_with?("%r")
579587

580-
if quote == "'" || quote.start_with?("%q") || quote.start_with?("%w") || quote.start_with?("%i")
588+
if interpolation?(quote)
589+
# In interpolation, escape sequences can be written literally. For example, "\\n" becomes "\n",
590+
# and "\\\\n" becomes "\\n". Unknown escapes sequences, like "\\o" simply become "o".
591+
string.gsub(/\\./) do |match|
592+
ESCAPES[match[1]] || match[1]
593+
end
594+
else
581595
if quote == "'"
582596
delimiter = "'"
583597
else
@@ -586,14 +600,18 @@ def unescape_string(string, quote)
586600

587601
delimiters = Regexp.escape("#{delimiter}#{DELIMITER_SYMETRY[delimiter]}")
588602
string.gsub(/\\([\\#{delimiters}])/, '\1')
589-
else
590-
# When double-quoted, escape sequences can be written literally. For example, "\\n" becomes "\n",
591-
# and "\\\\n" becomes "\\n". Unknown escapes sequences, like "\\o" simply become "o".
592-
string.gsub(/\\./) do |match|
593-
ESCAPES[match[1]] || match[1]
594-
end
595603
end
596604
end
605+
606+
# Determine if characters preceeded by a backslash should be escaped or not
607+
def interpolation?(quote)
608+
quote != "'" && !quote.start_with?("%q", "%w", "%i")
609+
end
610+
611+
# Determine if the string is part of a %-style array.
612+
def percent_array?(quote)
613+
quote.start_with?("%w", "%W", "%i", "%I")
614+
end
597615
end
598616
end
599617
end

test/prism/fixtures/strings.txt

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -83,6 +83,11 @@ b\nar
8383

8484
'\\ foo \\ bar'
8585

86+
'foo\
87+
bar\\
88+
baz
89+
'
90+
8691
"#$foo"
8792

8893
"#@foo"

test/prism/ruby/parser_test.rb

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -83,7 +83,6 @@ class ParserTest < TestCase
8383
skip_tokens = [
8484
"comments.txt",
8585
"dash_heredocs.txt",
86-
"dos_endings.txt",
8786
"embdoc_no_newline_at_end.txt",
8887
"heredoc_with_comment.txt",
8988
"heredocs_with_ignored_newlines.txt",

test/prism/snapshots/strings.txt

Lines changed: 87 additions & 81 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

0 commit comments

Comments
 (0)