🔀 ✅📈 Move parser tests to yml; add tests, benchmark (#103)

The two big motivations for this:
1. to simplify generation of test data from RFCs or other sources.
2. to simplify re-use of test data as benchmark data.

Currently defined test types:
* `:parser_assert_equal`: compares parse result to the `expected` data.
* `:parser_pending`: print out yaml which can be inserted into the
  appropriate yaml test fixture file.
* `:assert_parse_failure`: asserts `ResponseParseError` is raised

📈 `benchmarks/generate_parser_benchmarks` generates a
  `benchmark-driver` benchmark for every `:parser_assert_equal` test

Additional notes:
* ✅ Test most server response examples from RFC3501
  Although there is also a lot of overlap in here, this greatly improves
  branch coverage of ResponseParser tests.  Some of our existing tests
  were moved or renamed to match.
* ✅ Add THREAD tests, using RFC5256 examples
* 📖 Internally document lex states
* BODY/BODYSTRUCTURE is relatively complicated, so it's kept in its
  own file, separate from other FETCH msg-att tests.
* 🚧 The BODYSTRUCTURE tests match _current_ behavior, so the tests
include some buggy behavior.
* 🚧 Tests for modseq on SearchResult were commented out for now.
  That PR is coming soon.  :)
* 🔥 drop test_flag_list_many_same_flags... not really an issue any more
* 📖 Add new parser test doc
* 🚚 Move  uid_mapping test to new `test_imap_response_data.rb`

Author: nevans

benchmarks/generate_parser_benchmarks

@@ -0,0 +1,51 @@
+#!/usr/bin/env ruby
+
+require "yaml"
+require "pathname"
+require "net/imap"
+
+path = Pathname.new(__dir__) / "../test/net/imap/fixtures/response_parser"
+files = path.glob("*.yml")
+tests = files.flat_map {|file|
+  file.to_s
+    .then { YAML.unsafe_load_file _1 }
+    .fetch(:tests)
+    .select {|test_name, test|
+      :parser_assert_equal == test.fetch(:test_type) {
+        test.key?(:expected) ? :parser_assert_equal : :parser_pending
+      }
+    }
+    .map {|test_name, _|
+      [
+        file.relative_path_from(__dir__).to_s,
+        test_name.to_s,
+      ]
+    }
+}
+
+init = <<RUBY
+  require "yaml"
+  require "net/imap"
+
+  def load_response(file, name)
+    YAML.unsafe_load_file(file).dig(:tests, name, :response) \\
+      or abort "ERRORO: missing %p fixture data in %p" % [name, file]
+  end
+
+  parser   = Net::IMAP::ResponseParser.new
+RUBY
+
+prelude = <<RUBY
+  response = load_response(%p,
+                           %p)
+RUBY
+script = "parser.parse(response)"
+
+benchmarks = tests.map {|file, fixture_name|
+  name = fixture_name.delete_prefix("test_")
+  {name:, prelude: prelude % [file, fixture_name], script:}
+    .transform_keys(&:to_s)
+}
+  .sort_by { _1["name"] }
+
+puts YAML.dump({"prelude" => init, "benchmark" => benchmarks})

benchmarks/parser.yml

@@ -36,30 +36,31 @@ def parse(str)
 
       # :stopdoc:
 
-      EXPR_BEG          = :EXPR_BEG
-      EXPR_DATA         = :EXPR_DATA
-      EXPR_TEXT         = :EXPR_TEXT
-      EXPR_RTEXT        = :EXPR_RTEXT
-      EXPR_CTEXT        = :EXPR_CTEXT
-
-      T_SPACE   = :SPACE
-      T_NIL     = :NIL
-      T_NUMBER  = :NUMBER
-      T_ATOM    = :ATOM
-      T_QUOTED  = :QUOTED
-      T_LPAR    = :LPAR
-      T_RPAR    = :RPAR
-      T_BSLASH  = :BSLASH
-      T_STAR    = :STAR
-      T_LBRA    = :LBRA
-      T_RBRA    = :RBRA
-      T_LITERAL = :LITERAL
-      T_PLUS    = :PLUS
-      T_PERCENT = :PERCENT
-      T_CRLF    = :CRLF
-      T_EOF     = :EOF
-      T_TEXT    = :TEXT
-
+      EXPR_BEG   = :EXPR_BEG     # the default, used in most places
+      EXPR_DATA  = :EXPR_DATA    # envelope, body(structure), namespaces
+      EXPR_TEXT  = :EXPR_TEXT    # text, after 'resp-text-code "]"'
+      EXPR_RTEXT = :EXPR_RTEXT   # resp-text, before "["
+      EXPR_CTEXT = :EXPR_CTEXT   # resp-text-code, after 'atom SP'
+
+      T_SPACE    = :SPACE        # atom special
+      T_ATOM     = :ATOM         # atom (subset of astring chars)
+      T_NIL      = :NIL          # subset of atom and label
+      T_NUMBER   = :NUMBER       # subset of atom
+      T_LBRA     = :LBRA         # subset of atom
+      T_PLUS     = :PLUS         # subset of atom; tag special
+      T_RBRA     = :RBRA         # atom special; resp_special; valid astring char
+      T_QUOTED   = :QUOTED       # starts/end with atom special
+      T_BSLASH   = :BSLASH       # atom special; quoted special
+      T_LPAR     = :LPAR         # atom special; paren list delimiter
+      T_RPAR     = :RPAR         # atom special; paren list delimiter
+      T_STAR     = :STAR         # atom special; list wildcard
+      T_PERCENT  = :PERCENT      # atom special; list wildcard
+      T_LITERAL  = :LITERAL      # starts with atom special
+      T_CRLF     = :CRLF         # atom special; text special; quoted special
+      T_TEXT     = :TEXT         # any char except CRLF
+      T_EOF      = :EOF          # end of response string
+
+      # the default, used in most places
       BEG_REGEXP = /\G(?:\
 (?# 1:  SPACE   )( +)|\
 (?# 2:  NIL     )(NIL)(?=[\x80-\xff(){ \x00-\x1f\x7f%*"\\\[\]+])|\
@@ -78,6 +79,7 @@ def parse(str)
 (?# 15: CRLF    )(\r\n)|\
 (?# 16: EOF     )(\z))/ni
 
+      # envelope, body(structure), namespaces
       DATA_REGEXP = /\G(?:\
 (?# 1:  SPACE   )( )|\
 (?# 2:  NIL     )(NIL)|\
@@ -87,13 +89,16 @@ def parse(str)
 (?# 6:  LPAR    )(\()|\
 (?# 7:  RPAR    )(\)))/ni
 
+      # text, after 'resp-text-code "]"'
       TEXT_REGEXP = /\G(?:\
 (?# 1:  TEXT    )([^\x00\r\n]*))/ni
 
+      # resp-text, before "["
       RTEXT_REGEXP = /\G(?:\
 (?# 1:  LBRA    )(\[)|\
 (?# 2:  TEXT    )([^\x00\r\n]*))/ni
 
+      # resp-text-code, after 'atom SP'
       CTEXT_REGEXP = /\G(?:\
 (?# 1:  TEXT    )([^\x00\r\n\]]*))/ni
 

lib/net/imap/response_parser.rb

@@ -0,0 +1,16 @@
+---
+:tests:
+  test_acl_response:
+    :comment: |
+      [Bug #8281]
+    :response: "* ACL \"INBOX/share\" \"[email protected]\"
+      lrswickxteda\r\n"
+    :expected: !ruby/struct:Net::IMAP::UntaggedResponse
+      name: ACL
+      data:
+      - !ruby/struct:Net::IMAP::MailboxACLItem
+        user: [email protected]
+        rights: lrswickxteda
+        mailbox: INBOX/share
+      raw_data: "* ACL \"INBOX/share\" \"[email protected]\"
+        lrswickxteda\r\n"