Skip to content

Commit 970813d

Browse files
Earlopainmatzbot
authored andcommitted
[ruby/prism] Fix parser translator during string escaping with invalid utf-8
Instead, prefer `scan_byte` over `get_byte` since that already returns the byte as an integer, sidestepping conversion issues. Fixes ruby/prism#3582 ruby/prism@7f3008b2b5
1 parent 9520129 commit 970813d

File tree

4 files changed

+23
-3
lines changed

4 files changed

+23
-3
lines changed

lib/prism/polyfill/scan_byte.rb

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
# frozen_string_literal: true
2+
3+
require "strscan"
4+
5+
# Polyfill for StringScanner#scan_byte, which didn't exist until Ruby 3.4.
6+
if !(StringScanner.instance_methods.include?(:scan_byte))
7+
StringScanner.include(
8+
Module.new {
9+
def scan_byte # :nodoc:
10+
get_byte&.b&.ord
11+
end
12+
}
13+
)
14+
end

lib/prism/prism.gemspec

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -88,6 +88,7 @@ Gem::Specification.new do |spec|
8888
"lib/prism/pattern.rb",
8989
"lib/prism/polyfill/append_as_bytes.rb",
9090
"lib/prism/polyfill/byteindex.rb",
91+
"lib/prism/polyfill/scan_byte.rb",
9192
"lib/prism/polyfill/unpack1.rb",
9293
"lib/prism/polyfill/warn.rb",
9394
"lib/prism/reflection.rb",

lib/prism/translation/parser/lexer.rb

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@
33

44
require "strscan"
55
require_relative "../../polyfill/append_as_bytes"
6+
require_relative "../../polyfill/scan_byte"
67

78
module Prism
89
module Translation
@@ -762,12 +763,12 @@ def escape_read(result, scanner, control, meta)
762763
elsif (value = scanner.scan(/M-\\?(?=[[:print:]])/))
763764
# \M-x where x is an ASCII printable character
764765
escape_read(result, scanner, control, true)
765-
elsif (byte = scanner.get_byte)
766+
elsif (byte = scanner.scan_byte)
766767
# Something else after an escape.
767-
if control && byte == "?"
768+
if control && byte == 0x3f # ASCII '?'
768769
result.append_as_bytes(escape_build(0x7f, false, meta))
769770
else
770-
result.append_as_bytes(escape_build(byte.ord, control, meta))
771+
result.append_as_bytes(escape_build(byte, control, meta))
771772
end
772773
end
773774
end

test/prism/fixtures/strings.txt

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -146,6 +146,10 @@ baz
146146

147147
%Q{abc}
148148

149+
%Q(\«)
150+
151+
%q(\«)
152+
149153
%^#$^#
150154

151155
%@#@#

0 commit comments

Comments
 (0)