Process triple quoted string trivia during parsing

Triple quoted strings are de-indented with fairly complicated rules
* Based on string content
* The position of interpolations across multiple tokens

Because indentation isn't part of the string data it should also ideally
be excluded from the string content within the green tree. That is, it
should be treated as separate whitespace trivia tokens. With this
separation things like formatting should be much easier. The same
reasoning goes for escaping newlines and following whitespace with
backslashes in normal strings.

The downside of detecting string trivia during parsing is that string
content is split over several tokens. Here we wrap these in the K"string"
kind (as is already used for interpolations). The individual chunks can
then be reassembled during Expr construction. A possible alternative
might be to reuse the K"String" and K"CmdString" kinds for groups of
string chunks (without interpolation).

Author: c42f

README.md

@@ -458,6 +458,18 @@ use a flattened structure:
   to be the module name after the `.` is parsed. But `$` can never be a valid
   module name in normal Julia code so this makes no sense.
 
+* Triple quoted `var"""##"""` identifiers are allowed. But it's not clear these
+  are required or desired given that they come with the complex triple-quoted
+  string deindentation rules.
+
+* Deindentation of triple quoted strings with mismatched whitespace is weird
+  when there's nothing but whitespace. For example, we have
+  `"\"\"\"\n  \n \n  \"\"\"" ==> "\n \n"` so the middle line of whitespace
+  here isn't dedented but the other two longer lines are?? Here it seems more
+  consistent that either (a) the middle line should be deindented completely,
+  or (b) all lines should be dedented only one character, as that's the
+  matching prefix.
+
 # Comparisons to other packages
 
 ### Official Julia compiler

src/parse_stream.jl

@@ -130,8 +130,8 @@ end
 head(range::TaggedRange)       = range.head
 kind(range::TaggedRange)       = kind(range.head)
 flags(range::TaggedRange)      = flags(range.head)
-first_byte(range::TaggedRange) = range.first_byte
-last_byte(range::TaggedRange)  = range.last_byte
+first_byte(range::TaggedRange) = Int(range.first_byte)
+last_byte(range::TaggedRange)  = Int(range.last_byte)
 span(range::TaggedRange)       = 1 + last_byte(range) - first_byte(range)
 
 #-------------------------------------------------------------------------------
@@ -492,16 +492,42 @@ the kind or flags of a token in a way which would require unbounded lookahead
 in a recursive descent parser. Modifying the output with reset_node! is useful
 in those cases.
 """
-function reset_node!(stream::ParseStream, mark::ParseStreamPosition;
+function reset_node!(stream::ParseStream, pos::ParseStreamPosition;
                      kind=nothing, flags=nothing)
-    range = stream.ranges[mark.output_index]
+    range = stream.ranges[pos.output_index]
     k = isnothing(kind)  ? (@__MODULE__).kind(range)  : kind
     f = isnothing(flags) ? (@__MODULE__).flags(range) : flags
-    stream.ranges[mark.output_index] =
+    stream.ranges[pos.output_index] =
         TaggedRange(SyntaxHead(k, f), range.orig_kind,
                     first_byte(range), last_byte(range), range.start_mark)
 end
 
+"""
+Move `numbytes` from the range at output position `pos+1` to the output
+position `pos`. If the donor range becomes empty, mark it dead with
+K"TOMBSTONE" and return `true`, otherwise return `false`.
+
+Hack alert! This is used only for managing the complicated rules related to
+dedenting triple quoted strings.
+"""
+function steal_node_bytes!(stream::ParseStream, pos::ParseStreamPosition, numbytes)
+    i = pos.output_index
+    r1 = stream.ranges[i]
+    r2 = stream.ranges[i+1]
+    @assert span(r1) == 0
+    @assert numbytes <= span(r2)
+    fb2 = r2.first_byte + numbytes
+    rhs_empty = fb2 > last_byte(r2)
+    head2 = rhs_empty ? SyntaxHead(K"TOMBSTONE", EMPTY_FLAGS) : r2.head
+    stream.ranges[i]   = TaggedRange(r1.head, r1.orig_kind,
+                                     r2.first_byte, fb2 - 1,
+                                     r1.start_mark)
+    stream.ranges[i+1] = TaggedRange(head2, r2.orig_kind,
+                                     fb2, r2.last_byte,
+                                     r2.start_mark)
+    return rhs_empty
+end
+
 function Base.position(stream::ParseStream)
     ParseStreamPosition(stream.next_byte, lastindex(stream.ranges))
 end