[Bug] Dremel assembly/shredding algorithm

[norberttech]

Change Log

Added

Fixed

Changed

• Reimplemented from scratch Dremel Shredding/Assemble algorithms to cover all edge cases and fix bugs

Removed

Deprecated

Security

---

Description

[github-actions]

Flow PHP - Benchmarks

^{_{Results of the benchmarks from this PR are compared with the results from 1.x branch.}}

Extractors

+-----------------------+-------------------+------+-----+-------------------+--------------------+-----------------+
| benchmark             | subject           | revs | its | mem_peak          | mode               | rstdev          |
+-----------------------+-------------------+------+-----+-------------------+--------------------+-----------------+
| CSVExtractorBench     | bench_extract_10k | 1    | 3   | 4.620mb +0.34%    | 515.107ms +0.03%   | ±2.81% +487.76% |
| JsonExtractorBench    | bench_extract_10k | 1    | 3   | 4.715mb +0.33%    | 1.082s -0.75%      | ±1.23% +200.55% |
| ParquetExtractorBench | bench_extract_10k | 1    | 3   | 29.315mb +249.19% | 951.698ms +115.05% | ±0.29% -88.28%  |
| TextExtractorBench    | bench_extract_10k | 1    | 3   | 4.348mb +0.36%    | 33.675ms +0.42%    | ±1.07% +101.19% |
| XmlExtractorBench     | bench_extract_10k | 1    | 3   | 4.329mb +0.36%    | 664.847ms -0.47%   | ±0.64% -16.83%  |
+-----------------------+-------------------+------+-----+-------------------+--------------------+-----------------+

Transformers

+-----------------------------+--------------------------+------+-----+------------------+-----------------+----------------+
| benchmark                   | subject                  | revs | its | mem_peak         | mode            | rstdev         |
+-----------------------------+--------------------------+------+-----+------------------+-----------------+----------------+
| RenameEntryTransformerBench | bench_transform_10k_rows | 1    | 3   | 116.624mb +0.01% | 61.243ms -2.90% | ±0.59% -66.02% |
+-----------------------------+--------------------------+------+-----+------------------+-----------------+----------------+

Loaders

+--------------------+----------------+------+-----+--------------------+-------------------+-----------------+
| benchmark          | subject        | revs | its | mem_peak           | mode              | rstdev          |
+--------------------+----------------+------+-----+--------------------+-------------------+-----------------+
| CSVLoaderBench     | bench_load_10k | 1    | 3   | 54.817mb +0.03%    | 141.791ms +1.07%  | ±0.18% -75.74%  |
| JsonLoaderBench    | bench_load_10k | 1    | 3   | 90.408mb +0.02%    | 117.382ms +0.55%  | ±0.77% -1.26%   |
| ParquetLoaderBench | bench_load_10k | 1    | 3   | 124.671mb +104.73% | 21.763s +1560.72% | ±0.37% -80.84%  |
| TextLoaderBench    | bench_load_10k | 1    | 3   | 17.539mb +0.09%    | 44.184ms -0.73%   | ±0.37% +620.99% |
+--------------------+----------------+------+-----+--------------------+-------------------+-----------------+

Building Blocks

+-------------------------+----------------------------+------+-----+------------------+------------------+-----------------+
| benchmark               | subject                    | revs | its | mem_peak         | mode             | rstdev          |
+-------------------------+----------------------------+------+-----+------------------+------------------+-----------------+
| RowsBench               | bench_chunk_10_on_10k      | 2    | 3   | 87.372mb +0.02%  | 3.768ms +2.72%   | ±0.43% -79.71%  |
| RowsBench               | bench_diff_left_1k_on_10k  | 2    | 3   | 102.975mb +0.02% | 186.721ms +0.02% | ±0.82% -26.95%  |
| RowsBench               | bench_diff_right_1k_on_10k | 2    | 3   | 85.695mb +0.02%  | 19.166ms +2.70%  | ±1.59% +112.39% |
| RowsBench               | bench_drop_1k_on_10k       | 2    | 3   | 88.612mb +0.02%  | 2.071ms +3.08%   | ±1.91% +129.60% |
| RowsBench               | bench_drop_right_1k_on_10k | 2    | 3   | 88.612mb +0.02%  | 1.952ms -0.91%   | ±3.64% +25.50%  |
| RowsBench               | bench_entries_on_10k       | 2    | 3   | 85.724mb +0.02%  | 3.068ms -3.43%   | ±1.20% -57.74%  |
| RowsBench               | bench_filter_on_10k        | 2    | 3   | 86.253mb +0.02%  | 16.992ms -1.89%  | ±0.32% -65.98%  |
| RowsBench               | bench_find_on_10k          | 2    | 3   | 86.253mb +0.02%  | 17.040ms -0.02%  | ±0.94% -62.77%  |
| RowsBench               | bench_find_one_on_10k      | 10   | 3   | 84.156mb +0.02%  | 2.000μs -4.49%   | ±0.00% -100.00% |
| RowsBench               | bench_first_on_10k         | 10   | 3   | 84.156mb +0.02%  | 0.400μs 0.00%    | ±0.00% 0.00%    |
| RowsBench               | bench_flat_map_on_1k       | 2    | 3   | 93.507mb +0.02%  | 13.602ms -9.10%  | ±1.91% +105.47% |
| RowsBench               | bench_map_on_10k           | 2    | 3   | 122.878mb +0.01% | 64.342ms -0.74%  | ±1.15% +50.43%  |
| RowsBench               | bench_merge_1k_on_10k      | 2    | 3   | 86.772mb +0.02%  | 1.711ms -2.63%   | ±1.91% -36.71%  |
| RowsBench               | bench_partition_by_on_10k  | 2    | 3   | 90.124mb +0.02%  | 65.557ms +0.28%  | ±1.05% +41.36%  |
| RowsBench               | bench_remove_on_10k        | 2    | 3   | 88.874mb +0.02%  | 4.353ms -6.09%   | ±1.08% -9.26%   |
| RowsBench               | bench_sort_asc_on_1k       | 2    | 3   | 84.306mb +0.02%  | 40.049ms -1.85%  | ±0.45% -67.58%  |
| RowsBench               | bench_sort_by_on_1k        | 2    | 3   | 84.307mb +0.02%  | 40.276ms -4.89%  | ±1.02% -36.00%  |
| RowsBench               | bench_sort_desc_on_1k      | 2    | 3   | 84.306mb +0.02%  | 39.640ms -8.71%  | ±1.17% -58.08%  |
| RowsBench               | bench_sort_entries_on_1k   | 2    | 3   | 86.598mb +0.02%  | 7.542ms -4.96%   | ±0.82% -67.84%  |
| RowsBench               | bench_sort_on_1k           | 2    | 3   | 84.156mb +0.02%  | 29.186ms -7.28%  | ±1.27% -50.61%  |
| RowsBench               | bench_take_1k_on_10k       | 10   | 3   | 84.156mb +0.02%  | 13.959μs -1.14%  | ±1.77% +46.41%  |
| RowsBench               | bench_take_right_1k_on_10k | 10   | 3   | 84.156mb +0.02%  | 16.404μs -3.05%  | ±2.42% +227.46% |
| RowsBench               | bench_unique_on_1k         | 2    | 3   | 102.976mb +0.02% | 191.777ms +1.13% | ±0.69% +303.16% |
| TypeDetectorBench       | bench_type_detector        | 1    | 3   | 53.198mb +0.03%  | 393.264ms -1.60% | ±0.36% -1.46%   |
| TypeDetectorBench       | bench_type_detector        | 1    | 3   | 13.464mb +0.12%  | 81.735ms -0.38%  | ±0.87% +0.14%   |
| NativeEntryFactoryBench | bench_entry_factory        | 1    | 3   | 107.460mb +0.01% | 477.111ms -3.58% | ±2.34% +598.95% |
| NativeEntryFactoryBench | bench_entry_factory        | 1    | 3   | 55.818mb +0.03%  | 240.476ms -2.48% | ±0.50% -9.44%   |
| NativeEntryFactoryBench | bench_entry_factory        | 1    | 3   | 14.656mb +0.11%  | 52.512ms -5.01%  | ±1.91% +58.31%  |
+-------------------------+----------------------------+------+-----+------------------+------------------+-----------------+