Ran benchmarks on AVX2-capable CPU

Author: Vitruves

README.md

@@ -33,7 +33,7 @@ Carquet is **not** a replacement for Apache Arrow. Arrow is the industry standar
 | Write speed (ARM) | Baseline | **1.5-5x faster** |
 | Write speed (x86) | Baseline | ~same |
 | Read speed (ARM) | Baseline | ~same to 1.3x faster |
-| Read speed (x86) | **Baseline** | 2.5-5x slower |
+| Read speed (x86) | **Baseline** | 2-3x slower |
 | ZSTD file size | Baseline | **~1.4x smaller** |
 | Nested types | **Full support** | Basic |
 | Encryption | **Yes** | No |
@@ -888,33 +888,33 @@ Carquet's performance varies by platform and use case. These benchmarks show whe
 | SNAPPY | 191 MB | 174 MB | 1.1x larger |
 | UNCOMPRESSED | 191 MB | 201 MB | 1.05x smaller |
 
-### Intel Xeon E5-2680 (x86_64, Linux)
+### Intel Xeon D-1531 (x86_64, Linux)
 
-*Dell Precision T7600, 2x Intel Xeon E5-2680 (32 cores @ 2.7GHz), 94GB RAM, Ubuntu 24.04*
+*Supermicro SYS-5038MD-H24TRF, Intel Xeon D-1531 (12 threads @ 2.7GHz), 32GB RAM, Ubuntu 24.04, PyArrow 23.0.0*
 
 #### Writing
 
 | Codec | Carquet | PyArrow | Speedup |
 |-------|---------|---------|---------|
-| UNCOMPRESSED | 4.3 M rows/sec | 4.2 M rows/sec | ~same |
-| SNAPPY | 3.4 M rows/sec | 4.1 M rows/sec | 0.84x slower |
-| ZSTD | 3.1 M rows/sec | 3.4 M rows/sec | 0.93x slower |
+| UNCOMPRESSED | 4.6 M rows/sec | 4.1 M rows/sec | **1.14x faster** |
+| SNAPPY | 3.6 M rows/sec | 3.8 M rows/sec | ~same |
+| ZSTD | 3.8 M rows/sec | 3.5 M rows/sec | **1.09x faster** |
 
 #### Reading (PyArrow Faster)
 
 | Codec | Carquet | PyArrow | Ratio |
 |-------|---------|---------|-------|
-| UNCOMPRESSED | 29 M rows/sec | 73 M rows/sec | 0.40x slower |
-| SNAPPY | 18 M rows/sec | 83 M rows/sec | 0.22x slower |
-| ZSTD | 12 M rows/sec | 70 M rows/sec | 0.17x slower |
+| UNCOMPRESSED | 27 M rows/sec | 65 M rows/sec | 0.42x slower |
+| SNAPPY | 21 M rows/sec | 51 M rows/sec | 0.41x slower |
+| ZSTD | 19 M rows/sec | 49 M rows/sec | 0.39x slower |
 
 #### Compression Ratio
 
 | Codec | Carquet | PyArrow | Ratio |
 |-------|---------|---------|-------|
-| ZSTD | 107 MB | 150 MB | **1.4x smaller** |
-| SNAPPY | 191 MB | 174 MB | 1.1x larger |
-| UNCOMPRESSED | 191 MB | 201 MB | 1.05x smaller |
+| ZSTD | 108 MB | 148 MB | **1.37x smaller** |
+| SNAPPY | 191 MB | 173 MB | 1.10x larger |
+| UNCOMPRESSED | 191 MB | 199 MB | 1.04x smaller |
 
 ### Running Benchmarks
 

benchmark/results.csv

@@ -1,19 +1,19 @@
 library,dataset,compression,rows,write_ms,read_ms,file_bytes
-carquet,small,none,100000,13.66,2.97,2000366
-carquet,small,snappy,100000,18.64,3.86,1933903
-carquet,small,zstd,100000,24.66,7.00,1120473
-carquet,medium,none,1000000,214.25,37.88,20000380
-carquet,medium,snappy,1000000,261.30,60.25,19933920
-carquet,medium,zstd,1000000,282.45,63.27,11254110
-carquet,large,none,10000000,2345.80,344.63,200000392
-carquet,large,snappy,10000000,2952.21,557.37,199933935
-carquet,large,zstd,10000000,3179.43,815.71,112583423
-pyarrow,small,none,100000,21.87,4.49,2109775
-pyarrow,small,snappy,100000,22.02,9.35,1829385
-pyarrow,small,zstd,100000,27.99,9.01,1574036
-pyarrow,medium,none,1000000,224.73,15.50,21097029
-pyarrow,medium,snappy,1000000,244.22,22.81,18291770
-pyarrow,medium,zstd,1000000,302.36,22.16,15734520
-pyarrow,large,none,10000000,2354.29,136.84,210966630
-pyarrow,large,snappy,10000000,2466.86,120.17,182919992
-pyarrow,large,zstd,10000000,2951.95,142.65,157351710
+carquet,small,none,100000,14.14,3.08,2000366
+carquet,small,snappy,100000,17.56,5.20,1933903
+carquet,small,zstd,100000,20.88,6.91,1124254
+carquet,medium,none,1000000,176.69,49.16,20000380
+carquet,medium,snappy,1000000,206.23,55.10,19933920
+carquet,medium,zstd,1000000,227.20,46.36,11328167
+carquet,large,none,10000000,2173.68,365.08,200000392
+carquet,large,snappy,10000000,2744.93,485.64,199933935
+carquet,large,zstd,10000000,2628.34,528.65,113371970
+pyarrow,small,none,100000,19.56,3.64,2090690
+pyarrow,small,snappy,100000,22.46,10.56,1809900
+pyarrow,small,zstd,100000,25.89,8.56,1549280
+pyarrow,medium,none,1000000,215.51,16.50,20906179
+pyarrow,medium,snappy,1000000,271.80,28.08,18097012
+pyarrow,medium,zstd,1000000,315.11,28.90,15486940
+pyarrow,large,none,10000000,2467.33,154.61,209058089
+pyarrow,large,snappy,10000000,2633.04,197.98,180970820
+pyarrow,large,zstd,10000000,2870.02,204.28,154876135