Scohen/encapsulate source file (#743)

* SourceFile experimental replacement

This PR represents a suggestion on how elixir-ls can evolve the concept
of the source file. The current implementation is a group of
semi-related functions that operate on many different things, leading
to logic that's spread around the codebase, with each user of SourceFile
treating things differently.

This module, however always operates on the data structure it defines,
and provides a consistent and simplified interface for its users.
Among other things, it:

  * Builds changes with iolists, and doesn't need to do any reversing
    options
  * Introduces a data structure representing a document, with 0(1)
    random access.
  * Introduces a line record that separates the ending from the text
  * Introduces a new line parser that is significantly faster and more
    efficient than SourceFile.lines_with_endings
  * Applies edits in a single loop over the document

* Added benchmarks and line parser test

* Optimized slice, removed dead code

* experimental; changed text to to_string

* Made an experimental top-level package

This will help isolate the experimental stuff from the mainline more
cleanly, and also allows swapping of implementations via aliasing

* Added benchmark for document access

While playing around with the benchmark, I noticed that we weren't
consolidating protocols. This had a dramatic effect on the fetch
performance of the fetch_line call.

* Converted document backing store to tuples

* Added benchmark to test the speeds of various backing stores for Documents

* Ported ASCII detection / conversion from protocol branch

When parsing a file, we check if each line is ascii or not. If it is,
we can skip the conversions from utf-8 -> utf-16 -> utf8 when we apply
changes from the client. This means we reduce a ton of allocations and
simplify our code.

I also made a conversions module and some types representing Positions
and Ranges. Don't read too much into them, because I have a more
extensive data model that I'd like to discuss.

* Re-added Enum.at

* Fixed package name

* Performance improvements

* Changed document implementation to use tuples
* Enabled protocol consolidation outside of tests
* Change LineParser to use an indexed based approach

* Ported more improvements from the protocol branch

* Slightly simplified line parsing code

* Fixed utf conversion bugs

LineParser was not correctly setting the ascii? flag.
Also added an optimization to SourceFile to not convert the prefixes
and suffixes to utf16 if the lines are ascii.

* Removed utf16 conversions during document changes

Prior to this change, document tracking required doing utf-8 to utf-16
conversion on every change. Now all conversion is done on the
positions that are sent to us. We still need to do conversions on
non-ascii lines, but this should greatly cut down on the number of
conversions that elixir-ls has to do.

There was also an off-by-one error in the `conversions.ex` file where
we looked up the wrong line to do conversions. When utf16 conversions
were removed from `source_file.ex`, the bug became apparent with
several unit tests failing.

This also includes a small change to heuristically detect utf16 text.

* Removed utf16 conversions during document changes

Prior to this change, document tracking required doing utf-8 to utf-16
conversion on every change. Now all conversion is done on the
positions that are sent to us. We still need to do conversions on
non-ascii lines, but this should greatly cut down on the number of
conversions that elixir-ls has to do.

There was also an off-by-one error in the `conversions.ex` file where
we looked up the wrong line to do conversions. When utf16 conversions
were removed from `source_file.ex`, the bug became apparent with
several unit tests failing.

This also includes a small change to heuristically detect utf16 text.

* Removed protocol type

* Simplified prefix/suffix functions

* fixed import

* Fixed benchamrk off-by-one error

* Complied with out-of-spec unit tests

Some of the microsoft node tests deal with out of spec-lsp
messages. Prior, I was returning an error if an out-of-spec position
was encountered, but this change allows us to accept these messages
and handle them like the node client does.

* fixed merge error

* Revert "Complied with out-of-spec unit tests"

This reverts commit e07e0af.

* Added additional tests

Author: scohen

apps/language_server/benchmarks/document_backing_store_bench.exs

@@ -0,0 +1,38 @@
+file_generator = StreamData.string(:alphanumeric, min_size: 10, max_size: 120)
+
+line_endings = ["\r", "\n", "\r\n"]
+
+generate_lines = fn line_count ->
+  :alphanumeric
+  |> StreamData.string(min_size: 10, max_size: 120)
+  |> Enum.take(line_count)
+end
+
+Benchee.run(
+  %{
+    ":array.get(count - 1, array)" => fn %{array: array, count: count} ->
+      :array.get(count - 1, array)
+    end,
+    "Enum.at(lines, count - 1)" => fn %{lines: lines, count: count} ->
+      Enum.at(lines, count - 1)
+    end,
+    "list |> List.to_tuple() |> elem(count - 1)" => fn %{lines: lines, count: count} ->
+      lines |> List.to_tuple() |> elem(count - 1)
+    end,
+    "tuple" => fn %{tuple: tuple, count: count} ->
+      elem(tuple, count - 1)
+    end
+  },
+  inputs:
+    Map.new([80, 500, 1500], fn count ->
+      lines = generate_lines.(count)
+
+      {"#{count} lines",
+       %{
+         lines: lines,
+         array: :array.from_list(lines),
+         tuple: List.to_tuple(lines),
+         count: count
+       }}
+    end)
+)

apps/language_server/benchmarks/document_bench.exs

@@ -0,0 +1,44 @@
+alias ElixirLS.LanguageServer.Experimental.SourceFile.Document
+file_generator = StreamData.string(:alphanumeric, min_size: 10, max_size: 120)
+
+line_endings = ["\r", "\n", "\r\n"]
+
+generate_lines = fn line_count ->
+  :alphanumeric
+  |> StreamData.string(min_size: 10, max_size: 120)
+  |> Enum.take(line_count)
+end
+
+Benchee.run(
+  %{
+    "String.split |> Enum.at" => fn %{text: text, count: count} ->
+      text
+      |> String.split(line_endings)
+      |> Enum.at(count - 1)
+    end,
+    "Enum.at" => fn %{lines: lines, count: count} ->
+      Enum.at(lines, count - 1)
+    end,
+    "Document" => fn %{document: doc, count: count} ->
+      {:ok, _} = Document.fetch_line(doc, count - 1)
+    end,
+    "Document.new |> Document.fetch_line" => fn %{text: text, count: count} ->
+      text
+      |> Document.new()
+      |> Document.fetch_line(count)
+    end
+  },
+  inputs:
+    Map.new([80, 500, 1500], fn count ->
+      lines = generate_lines.(count)
+      text = Enum.join(lines, Enum.random(line_endings))
+
+      {"#{count} lines",
+       %{
+         lines: lines,
+         document: Document.new(text),
+         text: text,
+         count: count
+       }}
+    end)
+)

apps/language_server/benchmarks/line_parser_bench.exs

@@ -0,0 +1,26 @@
+alias ElixirLS.LanguageServer.SourceFile
+alias ElixirLS.LanguageServer.Experimental.SourceFile.LineParser
+
+line_endings = ["\r", "\n", "\r\n"]
+
+generate_file = fn line_count ->
+  :alphanumeric
+  |> StreamData.string(min_size: 10, max_size: 120)
+  |> Enum.take(line_count)
+  |> Enum.join(Enum.random(line_endings))
+end
+
+large_file = generate_file.(500)
+
+Benchee.run(
+  %{
+    "SourceFile.lines" => &SourceFile.lines/1,
+    "SourceFile.lines_with_endings/1" => &SourceFile.lines_with_endings/1,
+    "LineParser.parse" => &LineParser.parse(&1, 1)
+  },
+  inputs: %{
+    "80 lines" => generate_file.(80),
+    "500 lines" => generate_file.(500),
+    "1500 lines" => generate_file.(1500)
+  }
+)

apps/language_server/lib/language_server/experimental/protocol/types.ex

@@ -0,0 +1,21 @@
+defmodule ElixirLS.LanguageServer.Experimental.Protocol.Types do
+  defmodule Position do
+    defstruct [:line, :character]
+
+    def new(opts \\ []) do
+      line = Keyword.get(opts, :line, 0)
+      character = Keyword.get(opts, :character, 0)
+      %__MODULE__{line: line, character: character}
+    end
+  end
+
+  defmodule Range do
+    defstruct [:start, :end]
+
+    def new(opts \\ []) do
+      start_pos = Keyword.get(opts, :start, Position.new())
+      end_pos = Keyword.get(opts, :end, Position.new())
+      %__MODULE__{start: start_pos, end: end_pos}
+    end
+  end
+end

apps/language_server/lib/language_server/experimental/source_file.ex

@@ -0,0 +1,230 @@
+defmodule ElixirLS.LanguageServer.Experimental.SourceFile do
+  alias ElixirLS.LanguageServer.Experimental.SourceFile.Conversions
+  alias ElixirLS.LanguageServer.Experimental.SourceFile.Document
+  alias ElixirLS.LanguageServer.Experimental.SourceFile.Position
+  alias ElixirLS.LanguageServer.Experimental.SourceFile.Range
+  alias ElixirLS.LanguageServer.SourceFile
+  import ElixirLS.LanguageServer.Protocol, only: [range: 4]
+  import ElixirLS.LanguageServer.Experimental.SourceFile.Line
+
+  defstruct [:uri, :path, :version, dirty?: false, document: nil]
+
+  @type t :: %__MODULE__{
+          uri: String.t(),
+          version: pos_integer(),
+          dirty?: boolean,
+          document: Document.t(),
+          path: String.t()
+        }
+
+  @type version :: pos_integer()
+  @type change_application_error :: {:error, {:invalid_range, map()}}
+  # public
+  @spec new(URI.t(), String.t(), pos_integer()) :: t
+  def new(uri, text, version) do
+    %__MODULE__{
+      uri: uri,
+      version: version,
+      document: Document.new(text),
+      path: SourceFile.Path.from_uri(uri)
+    }
+  end
+
+  @spec mark_dirty(t) :: t
+  def mark_dirty(%__MODULE__{} = source) do
+    %__MODULE__{source | dirty?: true}
+  end
+
+  @spec mark_clean(t) :: t
+  def mark_clean(%__MODULE__{} = source) do
+    %__MODULE__{source | dirty?: false}
+  end
+
+  @spec fetch_text_at(t, version()) :: {:ok, String.t()} | :error
+  def fetch_text_at(%__MODULE{} = source, line_number) do
+    with {:ok, line(text: text)} <- Document.fetch_line(source.document, line_number) do
+      {:ok, text}
+    else
+      _ ->
+        :error
+    end
+  end
+
+  @spec apply_content_changes(t, pos_integer(), [map]) ::
+          {:ok, t} | change_application_error()
+  def apply_content_changes(%__MODULE__{version: current_version}, new_version, _)
+      when new_version <= current_version do
+    {:error, :invalid_version}
+  end
+
+  def apply_content_changes(%__MODULE__{} = source, _, []) do
+    {:ok, source}
+  end
+
+  def apply_content_changes(%__MODULE__{} = source, version, changes) when is_list(changes) do
+    result =
+      Enum.reduce_while(changes, source, fn change, source ->
+        case apply_change(source, change) do
+          {:ok, new_source} ->
+            {:cont, new_source}
+
+          error ->
+            {:halt, error}
+        end
+      end)
+
+    case result do
+      %__MODULE__{} = source ->
+        source = mark_dirty(%__MODULE__{source | version: version})
+
+        {:ok, source}
+
+      error ->
+        error
+    end
+  end
+
+  def to_string(%__MODULE__{} = source) do
+    source
+    |> to_iodata()
+    |> IO.iodata_to_binary()
+  end
+
+  # private
+
+  defp line_count(%__MODULE__{} = source) do
+    Document.size(source.document)
+  end
+
+  defp apply_change(
+         %__MODULE__{} = source,
+         %Range{start: %Position{} = start_pos, end: %Position{} = end_pos},
+         new_text
+       ) do
+    start_line = start_pos.line
+
+    new_lines_iodata =
+      cond do
+        start_line > line_count(source) ->
+          append_to_end(source, new_text)
+
+        start_line <= 0 ->
+          prepend_to_beginning(source, new_text)
+
+        true ->
+          apply_valid_edits(source, new_text, start_pos, end_pos)
+      end
+
+    new_document =
+      new_lines_iodata
+      |> IO.iodata_to_binary()
+      |> Document.new()
+
+    {:ok, %__MODULE__{source | document: new_document}}
+  end
+
+  defp apply_change(
+         %__MODULE__{} = source,
+         %{
+           "range" => range(start_line, start_char, end_line, end_char) = range,
+           "text" => new_text
+         }
+       )
+       when start_line >= 0 and start_char >= 0 and end_line >= 0 and end_char >= 0 do
+    with {:ok, ex_range} <- Conversions.to_elixir(range, source) do
+      apply_change(source, ex_range, new_text)
+    else
+      _ ->
+        {:error, {:invalid_range, range}}
+    end
+  end
+
+  defp apply_change(%__MODULE__{}, %{"range" => invalid_range}) do
+    {:error, {:invalid_range, invalid_range}}
+  end
+
+  defp apply_change(
+         %__MODULE__{} = source,
+         %{"text" => new_text}
+       ) do
+    {:ok, %__MODULE__{source | document: Document.new(new_text)}}
+  end
+
+  defp append_to_end(%__MODULE__{} = source, edit_text) do
+    [to_iodata(source), edit_text]
+  end
+
+  defp prepend_to_beginning(%__MODULE__{} = source, edit_text) do
+    [edit_text, to_iodata(source)]
+  end
+
+  defp apply_valid_edits(%__MODULE{} = source, edit_text, start_pos, end_pos) do
+    Enum.reduce(source.document, [], fn line() = line, acc ->
+      case edit_action(line, edit_text, start_pos, end_pos) do
+        :drop ->
+          acc
+
+        {:append, io_data} ->
+          [acc, io_data]
+      end
+    end)
+  end
+
+  defp edit_action(line() = line, edit_text, %Position{} = start_pos, %Position{} = end_pos) do
+    %Position{line: start_line, character: start_char} = start_pos
+    %Position{line: end_line, character: end_char} = end_pos
+
+    line(line_number: line_number, text: text, ending: ending) = line
+
+    cond do
+      line_number < start_line ->
+        {:append, [text, ending]}
+
+      line_number > end_line ->
+        {:append, [text, ending]}
+
+      line_number == start_line && line_number == end_line ->
+        prefix_text = utf8_prefix(text, start_char)
+        suffix_text = utf8_suffix(text, end_char)
+
+        {:append, [prefix_text, edit_text, suffix_text, ending]}
+
+      line_number == start_line ->
+        prefix_text = utf8_prefix(text, start_char)
+        {:append, [prefix_text, edit_text]}
+
+      line_number == end_line ->
+        suffix_text = utf8_suffix(text, end_char)
+        {:append, [suffix_text, ending]}
+
+      true ->
+        :drop
+    end
+  end
+
+  defp utf8_prefix(text, start_index) do
+    length = max(0, start_index)
+    binary_part(text, 0, length)
+  end
+
+  defp utf8_suffix(text, start_index) do
+    byte_count = byte_size(text)
+    start_index = min(start_index, byte_count)
+    length = byte_count - start_index
+    binary_part(text, start_index, length)
+  end
+
+  defp to_iodata(%__MODULE__{} = source) do
+    Document.to_iodata(source.document)
+  end
+
+  defp increment_version(%__MODULE__{} = source) do
+    version =
+      case source.version do
+        v when is_integer(v) -> v + 1
+        _ -> 1
+      end
+
+    %__MODULE__{source | version: version}
+  end
+end