Fix annotations swallowing comments (#72)

This PR attempts to fix a long standing bug that was carried over from
the original gem. Comments near an annotation would get removed when
adding or updating annotations.

ctran/annotate_models#951,
https://github.com/ctran/annotate_models/#warning

**Context**
Prior to this change, the AnnotateRb and the old gem Annotate would
annotate model files in 1 of 2 ways:
1. The "update" method, existing annotations would be found using a
regex and then new annotations would be changed in place. If a human
made comment was added after the annotation block **without** a new
line, it would get removed during the update process.
2. The "overwrite" method, which would happen when annotations did not
exist OR if the `:force` option was set to true. A regex would be used
to find annotation, remove it from the file, and then add annotations
again to the file depending on options such as `:position`.

Relying on regexes to identify annotations in a file made it hard to
reason about and hard to change, so most of them were removed and
replaced with a custom Ruby file parser using Ripper [1], which makes
working with Ruby files deterministic.

This PR makes the following behavioral changes:
- When updating annotations (i.e. annotations exist && `force: false`),
it only updates the annotations and should no longer delete any human
comments that are before or after an annotation block
- When generating (no annotations exist in the file) or re-generating
annotations (`force: true`), it removes the annotation block and
surrounding whitespace, if it exists, and then depending on the
`position` option, will add annotations before the class declaration or
after the class declaration. In the case of re-generating annotations,
any surrounding whitespace will be preserved assuming annotations are
being written to the same position.

[1] A great write up on Ripper:
https://kddnewton.com/2022/02/14/formatting-ruby-part-1.html

Author: drwl

lib/annotate_rb/model_annotator.rb

@@ -7,7 +7,6 @@ module ModelAnnotator
     autoload :BadModelFileError, "annotate_rb/model_annotator/bad_model_file_error"
     autoload :FileNameResolver, "annotate_rb/model_annotator/file_name_resolver"
     autoload :SingleFileAnnotationRemover, "annotate_rb/model_annotator/single_file_annotation_remover"
-    autoload :AnnotationPatternGenerator, "annotate_rb/model_annotator/annotation_pattern_generator"
     autoload :ModelClassGetter, "annotate_rb/model_annotator/model_class_getter"
     autoload :ModelFilesGetter, "annotate_rb/model_annotator/model_files_getter"
     autoload :SingleFileAnnotator, "annotate_rb/model_annotator/single_file_annotator"
@@ -22,7 +21,6 @@ module ModelAnnotator
     autoload :SingleFileRemoveAnnotationInstruction, "annotate_rb/model_annotator/single_file_remove_annotation_instruction"
     autoload :AnnotationDiffGenerator, "annotate_rb/model_annotator/annotation_diff_generator"
     autoload :AnnotationDiff, "annotate_rb/model_annotator/annotation_diff"
-    autoload :FileComponents, "annotate_rb/model_annotator/file_components"
     autoload :ProjectAnnotator, "annotate_rb/model_annotator/project_annotator"
     autoload :ProjectAnnotationRemover, "annotate_rb/model_annotator/project_annotation_remover"
     autoload :AnnotatedFile, "annotate_rb/model_annotator/annotated_file"

lib/annotate_rb/model_annotator/annotated_file/generator.rb

@@ -5,24 +5,56 @@ module ModelAnnotator
     module AnnotatedFile
       # Generates the file with fresh annotations
       class Generator
-        def initialize(file_components, annotation_position, options)
-          @file_components = file_components
+        def initialize(file_content, new_annotations, annotation_position, options)
           @annotation_position = annotation_position
           @options = options
 
-          @new_wrapped_annotations = wrapped_content(@file_components.new_annotations)
+          @new_wrapped_annotations = wrapped_content(new_annotations)
+
+          @new_annotations = new_annotations
+          @file_content = file_content
+
+          @parsed_file = FileParser::ParsedFile.new(@file_content, @new_annotations, options).parse
         end
 
         def generate
           # Need to keep `.to_s` for now since the it can be either a String or Symbol
           annotation_write_position = @options[@annotation_position].to_s
 
+          # New method: first remove annotations
+          content_without_annotations = if @parsed_file.has_annotations?
+            @file_content.sub(@parsed_file.annotations_with_whitespace, "")
+          elsif @options[:force]
+            @file_content.sub(@parsed_file.annotations_with_whitespace, "")
+          else
+            @file_content
+          end
+
+          # We need to get class start and class end depending on the position
+          parsed = FileParser::CustomParser.new(content_without_annotations, "", 0).tap(&:parse)
+
+          same_write_position = @parsed_file.has_annotations? && @parsed_file.annotation_position.to_s == annotation_write_position
+
+          # Could error if there's no class or module declaration
+          _constant_name, line_number_before = parsed.starts.first
+          content_with_annotations_written_before = []
+          content_with_annotations_written_before << content_without_annotations.lines[0...line_number_before]
+          content_with_annotations_written_before << $/ if @parsed_file.has_leading_whitespace? && same_write_position
+          content_with_annotations_written_before << @new_wrapped_annotations.lines
+          content_with_annotations_written_before << $/ if @parsed_file.has_trailing_whitespace? && same_write_position
+          content_with_annotations_written_before << content_without_annotations.lines[line_number_before..]
+
+          _constant_name, line_number_after = parsed.ends.last
+          content_with_annotations_written_after = []
+          content_with_annotations_written_after << content_without_annotations.lines[0..line_number_after]
+          content_with_annotations_written_after << $/
+          content_with_annotations_written_after << @new_wrapped_annotations.lines
+          content_with_annotations_written_after << content_without_annotations.lines[(line_number_after + 1)..]
+
           _content = if %w[after bottom].include?(annotation_write_position)
-            @file_components.magic_comments + (@file_components.pure_file_content.rstrip + "\n\n" + @new_wrapped_annotations)
-          elsif @file_components.magic_comments.empty?
-            @file_components.magic_comments + @new_wrapped_annotations + @file_components.pure_file_content.lstrip
+            content_with_annotations_written_after.join
           else
-            @file_components.magic_comments + "\n" + @new_wrapped_annotations + @file_components.pure_file_content.lstrip
+            content_with_annotations_written_before.join
           end
         end
 

lib/annotate_rb/model_annotator/annotated_file/updater.rb

@@ -5,22 +5,21 @@ module ModelAnnotator
     module AnnotatedFile
       # Updates existing annotations
       class Updater
-        def initialize(file_components, annotation_position, options)
-          @file_components = file_components
-          @annotation_position = annotation_position
+        def initialize(file_content, new_annotations, _annotation_position, options)
           @options = options
 
-          @new_wrapped_annotations = wrapped_content(@file_components.new_annotations)
+          @new_annotations = new_annotations
+          @file_content = file_content
+
+          @parsed_file = FileParser::ParsedFile.new(@file_content, @new_annotations, options).parse
         end
 
         def update
-          return "" if !@file_components.has_annotations?
-
-          annotation_pattern = AnnotationPatternGenerator.call(@options)
+          return "" if !@parsed_file.has_annotations?
 
-          new_annotation = @file_components.space_before_annotation + @new_wrapped_annotations + @file_components.space_after_annotation
+          new_annotation = wrapped_content(@new_annotations)
 
-          _content = @file_components.current_file_content.sub(annotation_pattern, new_annotation)
+          _content = @file_content.sub(@parsed_file.annotations, new_annotation)
         end
 
         private

lib/annotate_rb/model_annotator/annotation_pattern_generator.rb

@@ -3,7 +3,10 @@
 module AnnotateRb
   module ModelAnnotator
     module FileParser
-      autoload :MagicCommentParser, "annotate_rb/model_annotator/file_parser/magic_comment_parser"
+      autoload :AnnotationFinder, "annotate_rb/model_annotator/file_parser/annotation_finder"
+      autoload :CustomParser, "annotate_rb/model_annotator/file_parser/custom_parser"
+      autoload :ParsedFile, "annotate_rb/model_annotator/file_parser/parsed_file"
+      autoload :ParsedFileResult, "annotate_rb/model_annotator/file_parser/parsed_file_result"
     end
   end
 end

lib/annotate_rb/model_annotator/file_components.rb

@@ -0,0 +1,103 @@
+# frozen_string_literal: true
+
+module AnnotateRb
+  module ModelAnnotator
+    module FileParser
+      class AnnotationFinder
+        COMPAT_PREFIX = "== Schema Info"
+        COMPAT_PREFIX_MD = "## Schema Info"
+        DEFAULT_ANNOTATION_ENDING = "#"
+
+        class MalformedAnnotation < StandardError; end
+
+        class NoAnnotationFound < StandardError; end
+
+        # Returns the line index (not the line number) that the annotation starts.
+        attr_reader :annotation_start
+        # Returns the line index (not the line number) that the annotation ends, inclusive.
+        attr_reader :annotation_end
+
+        attr_reader :parser
+
+        def initialize(content, wrapper_open, wrapper_close)
+          @content = content
+          @wrapper_open = wrapper_open
+          @wrapper_close = wrapper_close
+          @annotation_start = nil
+          @annotation_end = nil
+          @parser = nil
+        end
+
+        # Find the annotation's line start and line end
+        def run
+          # CustomParser returns line numbers as 0-indexed
+          @parser = FileParser::CustomParser.new(@content, "", 0).tap(&:parse)
+          comments = @parser.comments
+
+          start = comments.find_index { |comment, _| comment.include?(COMPAT_PREFIX) || comment.include?(COMPAT_PREFIX_MD) }
+          raise NoAnnotationFound if start.nil? # Stop execution because we did not find
+
+          if @wrapper_open
+            prev_comment, _prev_line_number = comments[start - 1]
+
+            # Change start to the line before if wrapper_open is defined and we find the wrapper open comment
+            if prev_comment&.include?(@wrapper_open)
+              start -= 1
+            end
+          end
+
+          # Find a contiguous block of comments from the starting point
+          ending = start
+          while ending < comments.size - 1
+            _comment, line_number = comments[ending]
+            _next_comment, next_line_number = comments[ending + 1]
+
+            if next_line_number - line_number == 1
+              ending += 1
+            else
+              break
+            end
+          end
+
+          raise MalformedAnnotation if start == ending
+
+          if @wrapper_close
+            if comments[ending].first.include?(@wrapper_close)
+              # We can end here because it's the end of the annotation block
+            else
+              # Walk back until we find the end of the annotation comment block or the wrapper close to be flexible
+              #  We check if @wrapper_close is a substring because `comments` contains strings with the comment character
+              while ending > start && comments[ending].first != DEFAULT_ANNOTATION_ENDING && !comments[ending].first.include?(@wrapper_close)
+                ending -= 1
+              end
+            end
+          else
+            # Walk back until we find the end of the annotation comment block
+            while ending > start && comments[ending].first != DEFAULT_ANNOTATION_ENDING
+              ending -= 1
+            end
+          end
+
+          # We want .last because we want the line indexes
+          @annotation_start = comments[start].last
+          @annotation_end = comments[ending].last
+
+          [@annotation_start, @annotation_end]
+        end
+
+        def annotation
+          @annotation ||=
+            begin
+              lines = @content.lines
+              lines[@annotation_start..@annotation_end].join
+            end
+        end
+
+        # Returns true if annotations are detected in the file content
+        def annotated?
+          @annotation_start && @annotation_end
+        end
+      end
+    end
+  end
+end