[APM-823] Multi Delimiter (iresearch-toolkit#580)

* First draft of multi delimiter analyzer.

* Added vpack deserializer code.

* Build real automaton.

* Intermediate state.

* Single start and end state.

* Intermediate state.

* Fix

* Still not working.

* Build deterministic automaton by hand.

* Working version for multi character words.

* Clean up.

* Use hashmap instead of array.

* Clang format.

* Fixing code style.

* Small changes.

* Unify next functions.

* Add offset calculation.

* Added error handling.

* Added test for offset computation.

* Fix formatting.

* Fix formatting.

* Fixing usage of std::distance.

* Add special case for mac.

* adress review comments

* styling

* fix build

* fix after master merge

* fix mac build

* fix UB

---------

Co-authored-by: Valery Mironov <[email protected]>
Co-authored-by: Andrei Lobov <[email protected]>

Author: 3 people

core/CMakeLists.txt

@@ -343,6 +343,7 @@ add_dependencies(iresearch-static
   iresearch-build_identifier
   iresearch-build_version
   iresearch-analyzer-delimiter-static
+  iresearch-analyzer-multi-delimiter-static
   iresearch-analyzer-ngram-static
   iresearch-analyzer-text-static
   iresearch-analyzer-norm-static
@@ -392,6 +393,7 @@ endif ()
 
 target_link_libraries(iresearch-static
   PUBLIC iresearch-analyzer-delimiter-static
+  PUBLIC iresearch-analyzer-multi-delimiter-static
   PUBLIC iresearch-analyzer-ngram-static
   PUBLIC iresearch-analyzer-text-static
   PUBLIC iresearch-analyzer-norm-static
@@ -433,7 +435,7 @@ endif ()
 add_library(iresearch-analyzer-delimiter-static
   STATIC
   ./analysis/delimited_token_stream.cpp
-  )
+)
 
 set_ipo(iresearch-analyzer-delimiter-static)
 
@@ -442,11 +444,34 @@ set_target_properties(iresearch-analyzer-delimiter-static
   PREFIX lib
   IMPORT_PREFIX lib
   OUTPUT_NAME analyzer-delimiter-s
-  )
+)
 
 target_link_libraries(iresearch-analyzer-delimiter-static
   iresearch-static
-  )
+)
+
+################################################################################
+### analysis plugin : multi-delimiter
+################################################################################
+
+add_library(iresearch-analyzer-multi-delimiter-static
+  STATIC
+  ./analysis/multi_delimited_token_stream.cpp
+)
+
+set_ipo(iresearch-analyzer-multi-delimiter-static)
+
+set_target_properties(iresearch-analyzer-multi-delimiter-static
+  PROPERTIES
+  PREFIX lib
+  IMPORT_PREFIX lib
+  OUTPUT_NAME analyzer-multi-delimiter-s
+)
+
+target_link_libraries(iresearch-analyzer-multi-delimiter-static
+  iresearch-static
+  iresearch-ofst
+)
 
 ################################################################################
 ### analysis plugin : text