Generate Charsmaps During Build (#597)

* Generate Charsmaps During Build

Use ICU package during build to generate CharsMaps with Sentencepiece builder. Don't include the ICU package in the Tokenizers distribution.

* Update Tests

Delete tests for word level tokenizer - model is no longer on hub.

* Keep ICU dll during build

* Update Linux Build

* Add C++17 Requirement

* Add generated charsmaps to repo

* Remove ICU Debug Build

* Link sentencepiece-train only for charsmaps regeneration

* Remove redundant charsmaps

* Reuse one charsmap function for all node types

* Update Charsmaps

* Remove reinterpret_cast

* Update Benchmark to Report Warmup Metrics

* Apply suggestion from @Copilot

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Apply suggestion from @Copilot

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Author: apaniukov

README.md

@@ -148,6 +148,21 @@ cmake -DCMAKE_BUILD_TYPE=Release ..
 make
 ```
 
+#### CMake Build Options
+
+| Option | Default | Description |
+|--------|---------|-------------|
+| `REGENERATE_PRECOMPILED_CHARSMAP` | `OFF` | Regenerate Unicode normalization tables. Requires ICU. |
+| `ENABLE_SYSTEM_ICU` | `OFF` | Use system ICU instead of building from source (only when regenerating). |
+
+The `precompiled_charsmap.hpp` header containing Unicode normalization tables is pre-generated and committed to the repository. Most users don't need to regenerate it. To update these tables (e.g., after updating SentencePiece):
+
+```bash
+cmake -DCMAKE_BUILD_TYPE=Release -DREGENERATE_PRECOMPILED_CHARSMAP=ON ..
+make update_precompiled_charsmap
+# Commit the updated src/precompiled_charsmap.hpp
+```
+
 After that, you can transfer all binaries from `build/src` to `<openvino_dir>` as described in the C++ installation instruction above.
 
 ## Usage

benchmark/benchmark.py

@@ -11,6 +11,7 @@
 import matplotlib.pyplot as plt
 import openvino as ov
 import pandas as pd
+import psutil
 import seaborn as sns
 from openvino import AsyncInferQueue, CompiledModel, InferRequest, ProfilingInfo, properties
 from openvino_tokenizers import convert_tokenizer
@@ -93,8 +94,17 @@ def benchmark_tokenizers(
     results = []
 
     # warmup
+    process = psutil.Process()
     for repeat in range(1, 2):
+        # print time of the first tokenization pass
+        print(f"Warmup iteration {repeat}")
+        
+        mem_before_ov = process.memory_info().rss / 1024 / 1024  # MB
+        ov_start = perf_counter()
         ov_tokenizer(["test " * repeat])
+        ov_time = perf_counter() - ov_start
+        mem_after_ov = process.memory_info().rss / 1024 / 1024  # MB
+        print(f"OV warmup time: {ov_time:.6f} seconds, memory delta: {mem_after_ov - mem_before_ov:.2f} MB")
         hf_tokenizer(["test " * repeat])
 
     ov_input_ids = []