Skip to content

Commit b1cb0f2

Browse files
committed
Use UInt32 for stored character indices
This greatly reduces the size of the TaggedRange data structure on 64 bit systems, at the cost of not being able to parse files larger than 4GiB. But that seems like a reasonable tradeoff, for now. (We can parameterize later, if really necessary) It apperas that this one change makes parsing ~10% faster.
1 parent d95f6d2 commit b1cb0f2

File tree

1 file changed

+6
-6
lines changed

1 file changed

+6
-6
lines changed

src/parse_stream.jl

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -79,8 +79,8 @@ Information about preceding whitespace is added for use by the parser.
7979
"""
8080
struct SyntaxToken
8181
kind::Kind
82-
first_byte::Int
83-
last_byte::Int
82+
first_byte::UInt32
83+
last_byte::UInt32
8484
# Flags for leading whitespace
8585
is_dotted::Bool
8686
is_suffixed::Bool
@@ -122,17 +122,17 @@ TODO: Optimize this data structure? It's very large at the moment.
122122
struct TaggedRange
123123
head::SyntaxHead # Kind,flags
124124
orig_kind::Kind # Kind of the original token for leaf tokens, or K"Nothing"
125-
first_byte::Int # First byte in the input text
126-
last_byte::Int # Last byte in the input text
127-
start_mark::Int # Index of first emitted range which this range covers
125+
first_byte::UInt32 # First byte in the input text
126+
last_byte::UInt32 # Last byte in the input text
127+
start_mark::UInt32 # Index of first emitted range which this range covers
128128
end
129129

130130
head(range::TaggedRange) = range.head
131131
kind(range::TaggedRange) = kind(range.head)
132132
flags(range::TaggedRange) = flags(range.head)
133133
first_byte(range::TaggedRange) = range.first_byte
134134
last_byte(range::TaggedRange) = range.last_byte
135-
span(range::TaggedRange) = last_byte(range) - first_byte(range) + 1
135+
span(range::TaggedRange) = 1 + last_byte(range) - first_byte(range)
136136

137137
#-------------------------------------------------------------------------------
138138
struct ParseStreamPosition

0 commit comments

Comments
 (0)