Skip to content

Commit b6cb5ae

Browse files
committed
Ran benchmarks on AVX2-capable CPU
1 parent dc0ea8c commit b6cb5ae

File tree

2 files changed

+30
-30
lines changed

2 files changed

+30
-30
lines changed

README.md

Lines changed: 12 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -33,7 +33,7 @@ Carquet is **not** a replacement for Apache Arrow. Arrow is the industry standar
3333
| Write speed (ARM) | Baseline | **1.5-5x faster** |
3434
| Write speed (x86) | Baseline | ~same |
3535
| Read speed (ARM) | Baseline | ~same to 1.3x faster |
36-
| Read speed (x86) | **Baseline** | 2.5-5x slower |
36+
| Read speed (x86) | **Baseline** | 2-3x slower |
3737
| ZSTD file size | Baseline | **~1.4x smaller** |
3838
| Nested types | **Full support** | Basic |
3939
| Encryption | **Yes** | No |
@@ -888,33 +888,33 @@ Carquet's performance varies by platform and use case. These benchmarks show whe
888888
| SNAPPY | 191 MB | 174 MB | 1.1x larger |
889889
| UNCOMPRESSED | 191 MB | 201 MB | 1.05x smaller |
890890

891-
### Intel Xeon E5-2680 (x86_64, Linux)
891+
### Intel Xeon D-1531 (x86_64, Linux)
892892

893-
*Dell Precision T7600, 2x Intel Xeon E5-2680 (32 cores @ 2.7GHz), 94GB RAM, Ubuntu 24.04*
893+
*Supermicro SYS-5038MD-H24TRF, Intel Xeon D-1531 (12 threads @ 2.7GHz), 32GB RAM, Ubuntu 24.04, PyArrow 23.0.0*
894894

895895
#### Writing
896896

897897
| Codec | Carquet | PyArrow | Speedup |
898898
|-------|---------|---------|---------|
899-
| UNCOMPRESSED | 4.3 M rows/sec | 4.2 M rows/sec | ~same |
900-
| SNAPPY | 3.4 M rows/sec | 4.1 M rows/sec | 0.84x slower |
901-
| ZSTD | 3.1 M rows/sec | 3.4 M rows/sec | 0.93x slower |
899+
| UNCOMPRESSED | 4.6 M rows/sec | 4.1 M rows/sec | **1.14x faster** |
900+
| SNAPPY | 3.6 M rows/sec | 3.8 M rows/sec | ~same |
901+
| ZSTD | 3.8 M rows/sec | 3.5 M rows/sec | **1.09x faster** |
902902

903903
#### Reading (PyArrow Faster)
904904

905905
| Codec | Carquet | PyArrow | Ratio |
906906
|-------|---------|---------|-------|
907-
| UNCOMPRESSED | 29 M rows/sec | 73 M rows/sec | 0.40x slower |
908-
| SNAPPY | 18 M rows/sec | 83 M rows/sec | 0.22x slower |
909-
| ZSTD | 12 M rows/sec | 70 M rows/sec | 0.17x slower |
907+
| UNCOMPRESSED | 27 M rows/sec | 65 M rows/sec | 0.42x slower |
908+
| SNAPPY | 21 M rows/sec | 51 M rows/sec | 0.41x slower |
909+
| ZSTD | 19 M rows/sec | 49 M rows/sec | 0.39x slower |
910910

911911
#### Compression Ratio
912912

913913
| Codec | Carquet | PyArrow | Ratio |
914914
|-------|---------|---------|-------|
915-
| ZSTD | 107 MB | 150 MB | **1.4x smaller** |
916-
| SNAPPY | 191 MB | 174 MB | 1.1x larger |
917-
| UNCOMPRESSED | 191 MB | 201 MB | 1.05x smaller |
915+
| ZSTD | 108 MB | 148 MB | **1.37x smaller** |
916+
| SNAPPY | 191 MB | 173 MB | 1.10x larger |
917+
| UNCOMPRESSED | 191 MB | 199 MB | 1.04x smaller |
918918

919919
### Running Benchmarks
920920

benchmark/results.csv

Lines changed: 18 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -1,19 +1,19 @@
11
library,dataset,compression,rows,write_ms,read_ms,file_bytes
2-
carquet,small,none,100000,13.66,2.97,2000366
3-
carquet,small,snappy,100000,18.64,3.86,1933903
4-
carquet,small,zstd,100000,24.66,7.00,1120473
5-
carquet,medium,none,1000000,214.25,37.88,20000380
6-
carquet,medium,snappy,1000000,261.30,60.25,19933920
7-
carquet,medium,zstd,1000000,282.45,63.27,11254110
8-
carquet,large,none,10000000,2345.80,344.63,200000392
9-
carquet,large,snappy,10000000,2952.21,557.37,199933935
10-
carquet,large,zstd,10000000,3179.43,815.71,112583423
11-
pyarrow,small,none,100000,21.87,4.49,2109775
12-
pyarrow,small,snappy,100000,22.02,9.35,1829385
13-
pyarrow,small,zstd,100000,27.99,9.01,1574036
14-
pyarrow,medium,none,1000000,224.73,15.50,21097029
15-
pyarrow,medium,snappy,1000000,244.22,22.81,18291770
16-
pyarrow,medium,zstd,1000000,302.36,22.16,15734520
17-
pyarrow,large,none,10000000,2354.29,136.84,210966630
18-
pyarrow,large,snappy,10000000,2466.86,120.17,182919992
19-
pyarrow,large,zstd,10000000,2951.95,142.65,157351710
2+
carquet,small,none,100000,14.14,3.08,2000366
3+
carquet,small,snappy,100000,17.56,5.20,1933903
4+
carquet,small,zstd,100000,20.88,6.91,1124254
5+
carquet,medium,none,1000000,176.69,49.16,20000380
6+
carquet,medium,snappy,1000000,206.23,55.10,19933920
7+
carquet,medium,zstd,1000000,227.20,46.36,11328167
8+
carquet,large,none,10000000,2173.68,365.08,200000392
9+
carquet,large,snappy,10000000,2744.93,485.64,199933935
10+
carquet,large,zstd,10000000,2628.34,528.65,113371970
11+
pyarrow,small,none,100000,19.56,3.64,2090690
12+
pyarrow,small,snappy,100000,22.46,10.56,1809900
13+
pyarrow,small,zstd,100000,25.89,8.56,1549280
14+
pyarrow,medium,none,1000000,215.51,16.50,20906179
15+
pyarrow,medium,snappy,1000000,271.80,28.08,18097012
16+
pyarrow,medium,zstd,1000000,315.11,28.90,15486940
17+
pyarrow,large,none,10000000,2467.33,154.61,209058089
18+
pyarrow,large,snappy,10000000,2633.04,197.98,180970820
19+
pyarrow,large,zstd,10000000,2870.02,204.28,154876135

0 commit comments

Comments
 (0)