WIP: Upgrade DataFusion to arrow-rs/parquet 58.0.0 #19728

alamb · 2026-01-10T12:19:14Z

Which issue does this PR close?

Follow on to Upgrade DataFusion to arrow-rs/parquet 57.2.0 #19355
related to Release arrow-rs / parquet Major version 58.0.0 (January 2026) arrow-rs#8466

Rationale for this change

Keep datafusion up to date (and test Arrow using DataFusion tests)

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

….2.0

alamb · 2026-01-10T12:30:27Z

datafusion/sqllogictest/test_files/spark/datetime/make_interval.slt

 # We expect something like 0.00 secs with query ?
 query T
 SELECT make_interval(0, 0, 0, 0, 0, 0, 0.0) || '';
 ----


This is due to @Jefffrey 's improvement in

fix: display 0 secs for empty DayTime/MonthDayNano intervals arrow-rs#9023

close #17455

Probably should remove the accompanying comments too, these

# Intervals being rendered as empty string, see issue: # https://github.com/apache/datafusion/issues/17455 # We expect something like 0.00 secs with query ?

alamb · 2026-01-10T13:42:26Z

datafusion-cli/src/main.rs

        | alltypes_plain.parquet            | 1851            | 8882                | 2    | page_index=false |
-        | alltypes_tiny_pages.parquet       | 454233          | 269266              | 2    | page_index=true  |
-        | lz4_raw_compressed_larger.parquet | 380836          | 1347                | 2    | page_index=false |
+        | alltypes_tiny_pages.parquet       | 454233          | 269074              | 2    | page_index=true  |


I think this reduction in metadata size is a direct consequence of @WaterWhisperer's PR to improve PageEncoding representation

feat: change default behavior for Parquet PageEncodingStats to bitmask arrow-rs#9051

Dandandan · 2026-01-10T14:18:49Z

Run benchmarks

alamb-ghbot · 2026-01-10T17:46:31Z

🤖 ./gh_compare_branch.sh gh_compare_branch.sh Running
Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing alamb/update_arrow_58 (35b97fa) to b9a3b9f diff using: tpch_mem clickbench_partitioned clickbench_extended
Results will be posted here when complete

alamb-ghbot · 2026-01-10T18:26:55Z

🤖: Benchmark completed

Details

Comparing HEAD and alamb_update_arrow_58
--------------------
Benchmark clickbench_extended.json
--------------------
┏━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Query    ┃        HEAD ┃ alamb_update_arrow_58 ┃    Change ┃
┡━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ QQuery 0 │  2354.54 ms │            2406.77 ms │ no change │
│ QQuery 1 │   939.82 ms │             929.76 ms │ no change │
│ QQuery 2 │  1882.93 ms │            1891.80 ms │ no change │
│ QQuery 3 │  1163.47 ms │            1165.13 ms │ no change │
│ QQuery 4 │  2306.07 ms │            2241.55 ms │ no change │
│ QQuery 5 │ 28452.06 ms │           28080.09 ms │ no change │
│ QQuery 6 │  4018.71 ms │            4055.12 ms │ no change │
│ QQuery 7 │  3790.29 ms │            3670.14 ms │ no change │
└──────────┴─────────────┴───────────────────────┴───────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary                    ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD)                    │ 44907.89ms │
│ Total Time (alamb_update_arrow_58)   │ 44440.36ms │
│ Average Time (HEAD)                  │  5613.49ms │
│ Average Time (alamb_update_arrow_58) │  5555.05ms │
│ Queries Faster                       │          0 │
│ Queries Slower                       │          0 │
│ Queries with No Change               │          8 │
│ Queries with Failure                 │          0 │
└──────────────────────────────────────┴────────────┘
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query     ┃        HEAD ┃ alamb_update_arrow_58 ┃        Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0  │     1.86 ms │               1.94 ms │     no change │
│ QQuery 1  │    50.56 ms │              50.42 ms │     no change │
│ QQuery 2  │   134.60 ms │             134.00 ms │     no change │
│ QQuery 3  │   156.94 ms │             152.15 ms │     no change │
│ QQuery 4  │  1074.92 ms │            1227.05 ms │  1.14x slower │
│ QQuery 5  │  1349.11 ms │            1512.35 ms │  1.12x slower │
│ QQuery 6  │     1.82 ms │               1.86 ms │     no change │
│ QQuery 7  │    56.21 ms │              53.80 ms │     no change │
│ QQuery 8  │  1404.44 ms │            1546.63 ms │  1.10x slower │
│ QQuery 9  │  1744.32 ms │            1884.58 ms │  1.08x slower │
│ QQuery 10 │   344.43 ms │             351.00 ms │     no change │
│ QQuery 11 │   393.88 ms │             399.97 ms │     no change │
│ QQuery 12 │  1259.31 ms │            1442.13 ms │  1.15x slower │
│ QQuery 13 │  1916.02 ms │            2053.79 ms │  1.07x slower │
│ QQuery 14 │  1228.80 ms │            1371.03 ms │  1.12x slower │
│ QQuery 15 │  1223.04 ms │            1359.64 ms │  1.11x slower │
│ QQuery 16 │  2568.94 ms │            2675.51 ms │     no change │
│ QQuery 17 │  2521.47 ms │            2647.03 ms │     no change │
│ QQuery 18 │  6120.19 ms │            5000.68 ms │ +1.22x faster │
│ QQuery 19 │   120.13 ms │             119.44 ms │     no change │
│ QQuery 20 │  1948.33 ms │            1896.06 ms │     no change │
│ QQuery 21 │  2245.14 ms │            2178.01 ms │     no change │
│ QQuery 22 │  3847.87 ms │            3802.23 ms │     no change │
│ QQuery 23 │ 21572.08 ms │           12373.92 ms │ +1.74x faster │
│ QQuery 24 │   225.72 ms │             216.64 ms │     no change │
│ QQuery 25 │   471.41 ms │             478.99 ms │     no change │
│ QQuery 26 │   218.83 ms │             235.13 ms │  1.07x slower │
│ QQuery 27 │  2823.98 ms │            2691.15 ms │     no change │
│ QQuery 28 │ 23860.88 ms │           23263.75 ms │     no change │
│ QQuery 29 │   982.83 ms │             966.29 ms │     no change │
│ QQuery 30 │  1343.19 ms │            1378.01 ms │     no change │
│ QQuery 31 │  1521.78 ms │            1379.20 ms │ +1.10x faster │
│ QQuery 32 │  5104.40 ms │            4890.24 ms │     no change │
│ QQuery 33 │  5837.64 ms │            5506.92 ms │ +1.06x faster │
│ QQuery 34 │  5933.39 ms │            5605.95 ms │ +1.06x faster │
│ QQuery 35 │  1921.56 ms │            1960.63 ms │     no change │
│ QQuery 36 │    65.96 ms │              68.18 ms │     no change │
│ QQuery 37 │    45.85 ms │              45.16 ms │     no change │
│ QQuery 38 │    66.53 ms │              65.89 ms │     no change │
│ QQuery 39 │   102.71 ms │             106.18 ms │     no change │
│ QQuery 40 │    28.23 ms │              25.52 ms │ +1.11x faster │
│ QQuery 41 │    24.10 ms │              23.51 ms │     no change │
│ QQuery 42 │    19.09 ms │              18.80 ms │     no change │
└───────────┴─────────────┴───────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┓
┃ Benchmark Summary                    ┃             ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━┩
│ Total Time (HEAD)                    │ 103882.45ms │
│ Total Time (alamb_update_arrow_58)   │  93161.37ms │
│ Average Time (HEAD)                  │   2415.87ms │
│ Average Time (alamb_update_arrow_58) │   2166.54ms │
│ Queries Faster                       │           6 │
│ Queries Slower                       │           9 │
│ Queries with No Change               │          28 │
│ Queries with Failure                 │           0 │
└──────────────────────────────────────┴─────────────┘
--------------------
Benchmark tpch_mem_sf1.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query     ┃      HEAD ┃ alamb_update_arrow_58 ┃        Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 1  │ 119.70 ms │             118.08 ms │     no change │
│ QQuery 2  │  35.82 ms │              36.60 ms │     no change │
│ QQuery 3  │  40.70 ms │              36.99 ms │ +1.10x faster │
│ QQuery 4  │  30.37 ms │              30.72 ms │     no change │
│ QQuery 5  │  91.59 ms │              90.47 ms │     no change │
│ QQuery 6  │  21.09 ms │              20.76 ms │     no change │
│ QQuery 7  │ 222.34 ms │             225.43 ms │     no change │
│ QQuery 8  │  44.85 ms │              40.29 ms │ +1.11x faster │
│ QQuery 9  │ 122.80 ms │             104.76 ms │ +1.17x faster │
│ QQuery 10 │  81.06 ms │              66.33 ms │ +1.22x faster │
│ QQuery 11 │  23.80 ms │              24.89 ms │     no change │
│ QQuery 12 │  52.22 ms │              52.78 ms │     no change │
│ QQuery 13 │  48.94 ms │              49.75 ms │     no change │
│ QQuery 14 │  14.95 ms │              14.96 ms │     no change │
│ QQuery 15 │  30.50 ms │              30.48 ms │     no change │
│ QQuery 16 │  28.47 ms │              29.06 ms │     no change │
│ QQuery 17 │ 151.61 ms │             154.03 ms │     no change │
│ QQuery 18 │ 287.03 ms │             288.27 ms │     no change │
│ QQuery 19 │  39.57 ms │              39.77 ms │     no change │
│ QQuery 20 │  57.69 ms │              57.36 ms │     no change │
│ QQuery 21 │ 322.81 ms │             318.54 ms │     no change │
│ QQuery 22 │  21.98 ms │              21.61 ms │     no change │
└───────────┴───────────┴───────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Benchmark Summary                    ┃           ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ Total Time (HEAD)                    │ 1889.89ms │
│ Total Time (alamb_update_arrow_58)   │ 1851.95ms │
│ Average Time (HEAD)                  │   85.90ms │
│ Average Time (alamb_update_arrow_58) │   84.18ms │
│ Queries Faster                       │         4 │
│ Queries Slower                       │         0 │
│ Queries with No Change               │        18 │
│ Queries with Failure                 │         0 │
└──────────────────────────────────────┴───────────┘

Dandandan · 2026-01-10T18:32:08Z

run benchmarks

Dandandan · 2026-01-10T18:32:18Z

run benchmark tpch

alamb-ghbot · 2026-01-10T18:40:16Z

🤖 ./gh_compare_branch.sh gh_compare_branch.sh Running
Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing alamb/update_arrow_58 (35b97fa) to b9a3b9f diff using: tpch_mem clickbench_partitioned clickbench_extended
Results will be posted here when complete

alamb-ghbot · 2026-01-10T19:18:22Z

🤖: Benchmark completed

Details

Comparing HEAD and alamb_update_arrow_58
--------------------
Benchmark clickbench_extended.json
--------------------
┏━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Query    ┃        HEAD ┃ alamb_update_arrow_58 ┃    Change ┃
┡━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ QQuery 0 │  2389.61 ms │            2355.66 ms │ no change │
│ QQuery 1 │   975.65 ms │             969.78 ms │ no change │
│ QQuery 2 │  1875.49 ms │            1871.78 ms │ no change │
│ QQuery 3 │  1174.32 ms │            1145.53 ms │ no change │
│ QQuery 4 │  2258.94 ms │            2223.01 ms │ no change │
│ QQuery 5 │ 27824.57 ms │           28406.72 ms │ no change │
│ QQuery 6 │  4031.59 ms │            4002.29 ms │ no change │
│ QQuery 7 │  3463.56 ms │            3533.17 ms │ no change │
└──────────┴─────────────┴───────────────────────┴───────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary                    ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD)                    │ 43993.73ms │
│ Total Time (alamb_update_arrow_58)   │ 44507.93ms │
│ Average Time (HEAD)                  │  5499.22ms │
│ Average Time (alamb_update_arrow_58) │  5563.49ms │
│ Queries Faster                       │          0 │
│ Queries Slower                       │          0 │
│ Queries with No Change               │          8 │
│ Queries with Failure                 │          0 │
└──────────────────────────────────────┴────────────┘
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query     ┃        HEAD ┃ alamb_update_arrow_58 ┃        Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0  │     1.90 ms │               1.92 ms │     no change │
│ QQuery 1  │    52.12 ms │              50.35 ms │     no change │
│ QQuery 2  │   142.92 ms │             133.95 ms │ +1.07x faster │
│ QQuery 3  │   154.00 ms │             152.77 ms │     no change │
│ QQuery 4  │  1072.51 ms │            1063.15 ms │     no change │
│ QQuery 5  │  1369.60 ms │            1320.07 ms │     no change │
│ QQuery 6  │     1.82 ms │               1.86 ms │     no change │
│ QQuery 7  │    54.39 ms │              52.98 ms │     no change │
│ QQuery 8  │  1446.73 ms │            1417.75 ms │     no change │
│ QQuery 9  │  1835.01 ms │            1706.51 ms │ +1.08x faster │
│ QQuery 10 │   344.52 ms │             354.29 ms │     no change │
│ QQuery 11 │   392.19 ms │             397.62 ms │     no change │
│ QQuery 12 │  1286.14 ms │            1275.41 ms │     no change │
│ QQuery 13 │  1947.40 ms │            1962.96 ms │     no change │
│ QQuery 14 │  1259.13 ms │            1231.11 ms │     no change │
│ QQuery 15 │  1247.90 ms │            1222.46 ms │     no change │
│ QQuery 16 │  2534.25 ms │            2515.84 ms │     no change │
│ QQuery 17 │  2514.25 ms │            2527.22 ms │     no change │
│ QQuery 18 │  5440.35 ms │            4806.73 ms │ +1.13x faster │
│ QQuery 19 │   119.26 ms │             118.76 ms │     no change │
│ QQuery 20 │  1929.28 ms │            1872.84 ms │     no change │
│ QQuery 21 │  2213.86 ms │            2178.92 ms │     no change │
│ QQuery 22 │  4264.32 ms │            3784.45 ms │ +1.13x faster │
│ QQuery 23 │ 20258.80 ms │           12236.28 ms │ +1.66x faster │
│ QQuery 24 │   215.95 ms │             219.20 ms │     no change │
│ QQuery 25 │   488.91 ms │             481.13 ms │     no change │
│ QQuery 26 │   226.35 ms │             216.76 ms │     no change │
│ QQuery 27 │  2799.99 ms │            2713.67 ms │     no change │
│ QQuery 28 │ 23840.67 ms │           23084.98 ms │     no change │
│ QQuery 29 │   974.43 ms │             971.18 ms │     no change │
│ QQuery 30 │  1351.52 ms │            1299.11 ms │     no change │
│ QQuery 31 │  1399.75 ms │            1421.76 ms │     no change │
│ QQuery 32 │  4761.26 ms │            4490.64 ms │ +1.06x faster │
│ QQuery 33 │  5521.96 ms │            5313.98 ms │     no change │
│ QQuery 34 │  5548.81 ms │            5584.56 ms │     no change │
│ QQuery 35 │  1979.74 ms │            1851.73 ms │ +1.07x faster │
│ QQuery 36 │    66.91 ms │              67.24 ms │     no change │
│ QQuery 37 │    46.16 ms │              47.24 ms │     no change │
│ QQuery 38 │    66.08 ms │              67.97 ms │     no change │
│ QQuery 39 │   102.12 ms │             100.78 ms │     no change │
│ QQuery 40 │    28.61 ms │              27.21 ms │     no change │
│ QQuery 41 │    23.45 ms │              22.02 ms │ +1.06x faster │
│ QQuery 42 │    19.24 ms │              19.38 ms │     no change │
└───────────┴─────────────┴───────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┓
┃ Benchmark Summary                    ┃             ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━┩
│ Total Time (HEAD)                    │ 101344.52ms │
│ Total Time (alamb_update_arrow_58)   │  90386.74ms │
│ Average Time (HEAD)                  │   2356.85ms │
│ Average Time (alamb_update_arrow_58) │   2102.02ms │
│ Queries Faster                       │           8 │
│ Queries Slower                       │           0 │
│ Queries with No Change               │          35 │
│ Queries with Failure                 │           0 │
└──────────────────────────────────────┴─────────────┘
--------------------
Benchmark tpch_mem_sf1.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query     ┃      HEAD ┃ alamb_update_arrow_58 ┃        Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 1  │ 118.62 ms │             117.02 ms │     no change │
│ QQuery 2  │  37.93 ms │              38.57 ms │     no change │
│ QQuery 3  │  41.97 ms │              37.52 ms │ +1.12x faster │
│ QQuery 4  │  31.39 ms │              30.22 ms │     no change │
│ QQuery 5  │  91.63 ms │              91.98 ms │     no change │
│ QQuery 6  │  21.13 ms │              20.89 ms │     no change │
│ QQuery 7  │ 229.95 ms │             228.06 ms │     no change │
│ QQuery 8  │  39.25 ms │              37.93 ms │     no change │
│ QQuery 9  │ 107.60 ms │             107.66 ms │     no change │
│ QQuery 10 │  69.45 ms │              69.75 ms │     no change │
│ QQuery 11 │  23.16 ms │              24.35 ms │  1.05x slower │
│ QQuery 12 │  53.07 ms │              52.75 ms │     no change │
│ QQuery 13 │  51.17 ms │              48.55 ms │ +1.05x faster │
│ QQuery 14 │  15.28 ms │              15.11 ms │     no change │
│ QQuery 15 │  30.14 ms │              30.73 ms │     no change │
│ QQuery 16 │  31.26 ms │              29.13 ms │ +1.07x faster │
│ QQuery 17 │ 159.46 ms │             154.29 ms │     no change │
│ QQuery 18 │ 287.59 ms │             284.81 ms │     no change │
│ QQuery 19 │  39.20 ms │              39.99 ms │     no change │
│ QQuery 20 │  56.56 ms │              57.03 ms │     no change │
│ QQuery 21 │ 300.14 ms │             325.49 ms │  1.08x slower │
│ QQuery 22 │  22.64 ms │              22.18 ms │     no change │
└───────────┴───────────┴───────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Benchmark Summary                    ┃           ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ Total Time (HEAD)                    │ 1858.59ms │
│ Total Time (alamb_update_arrow_58)   │ 1864.00ms │
│ Average Time (HEAD)                  │   84.48ms │
│ Average Time (alamb_update_arrow_58) │   84.73ms │
│ Queries Faster                       │         3 │
│ Queries Slower                       │         2 │
│ Queries with No Change               │        17 │
│ Queries with Failure                 │         0 │
└──────────────────────────────────────┴───────────┘

alamb-ghbot · 2026-01-10T19:18:26Z

🤖 ./gh_compare_branch.sh gh_compare_branch.sh Running
Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing alamb/update_arrow_58 (35b97fa) to b9a3b9f diff using: tpch
Results will be posted here when complete

alamb-ghbot · 2026-01-10T19:19:04Z

🤖: Benchmark completed

Details

Comparing HEAD and alamb_update_arrow_58
--------------------
Benchmark tpch_sf1.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query     ┃      HEAD ┃ alamb_update_arrow_58 ┃        Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 1  │ 200.13 ms │             195.58 ms │     no change │
│ QQuery 2  │ 101.86 ms │             103.65 ms │     no change │
│ QQuery 3  │ 128.67 ms │             121.90 ms │ +1.06x faster │
│ QQuery 4  │  78.40 ms │              77.50 ms │     no change │
│ QQuery 5  │ 172.69 ms │             172.92 ms │     no change │
│ QQuery 6  │  65.58 ms │              68.66 ms │     no change │
│ QQuery 7  │ 219.12 ms │             245.34 ms │  1.12x slower │
│ QQuery 8  │ 175.19 ms │             177.33 ms │     no change │
│ QQuery 9  │ 227.09 ms │             244.95 ms │  1.08x slower │
│ QQuery 10 │ 184.88 ms │             190.38 ms │     no change │
│ QQuery 11 │  80.82 ms │              78.24 ms │     no change │
│ QQuery 12 │ 117.62 ms │             116.55 ms │     no change │
│ QQuery 13 │ 223.12 ms │             215.42 ms │     no change │
│ QQuery 14 │  93.28 ms │              93.53 ms │     no change │
│ QQuery 15 │ 128.30 ms │             123.42 ms │     no change │
│ QQuery 16 │  59.98 ms │              60.86 ms │     no change │
│ QQuery 17 │ 270.35 ms │             272.73 ms │     no change │
│ QQuery 18 │ 316.62 ms │             314.87 ms │     no change │
│ QQuery 19 │ 131.64 ms │             137.17 ms │     no change │
│ QQuery 20 │ 132.77 ms │             130.38 ms │     no change │
│ QQuery 21 │ 271.59 ms │             274.52 ms │     no change │
│ QQuery 22 │  47.46 ms │              48.44 ms │     no change │
└───────────┴───────────┴───────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Benchmark Summary                    ┃           ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ Total Time (HEAD)                    │ 3427.15ms │
│ Total Time (alamb_update_arrow_58)   │ 3464.32ms │
│ Average Time (HEAD)                  │  155.78ms │
│ Average Time (alamb_update_arrow_58) │  157.47ms │
│ Queries Faster                       │         1 │
│ Queries Slower                       │         2 │
│ Queries with No Change               │        19 │
│ Queries with Failure                 │         0 │
└──────────────────────────────────────┴───────────┘

Dandandan · 2026-01-16T12:01:08Z

run benchmarks

alamb · 2026-01-16T12:09:08Z

(oh yeah -- we are all into the benchmarking now. This is going to be great)

Dandandan · 2026-01-16T12:12:13Z

(oh yeah -- we are all into the benchmarking now. This is going to be great)

Going to check apache/arrow-rs#8951 next to see the diff

Dandandan · 2026-01-16T12:12:28Z

run benchmark tpch tpcds

Dandandan · 2026-01-16T12:41:56Z

datafusion/sqllogictest/test_files/window.slt

 );
 ----
-1
+DataFusion error: Join Error


This somehow errors now 🤔

alamb-ghbot · 2026-01-16T13:03:20Z

🤖 ./gh_compare_branch.sh gh_compare_branch.sh Running
Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing alamb/update_arrow_58 (1002408) to b9a3b9f diff using: tpch_mem clickbench_partitioned clickbench_extended
Results will be posted here when complete

alamb-ghbot · 2026-01-16T13:43:27Z

🤖: Benchmark completed

Details

Comparing HEAD and alamb_update_arrow_58
--------------------
Benchmark clickbench_extended.json
--------------------
┏━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query    ┃        HEAD ┃ alamb_update_arrow_58 ┃        Change ┃
┡━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0 │  2378.55 ms │            2305.74 ms │     no change │
│ QQuery 1 │   921.75 ms │             929.11 ms │     no change │
│ QQuery 2 │  1858.00 ms │            1930.32 ms │     no change │
│ QQuery 3 │  1172.43 ms │            1090.28 ms │ +1.08x faster │
│ QQuery 4 │  2275.36 ms │            2208.56 ms │     no change │
│ QQuery 5 │ 28070.94 ms │           28172.37 ms │     no change │
│ QQuery 6 │  4054.39 ms │            3899.27 ms │     no change │
│ QQuery 7 │  3647.17 ms │            3643.91 ms │     no change │
└──────────┴─────────────┴───────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary                    ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD)                    │ 44378.58ms │
│ Total Time (alamb_update_arrow_58)   │ 44179.57ms │
│ Average Time (HEAD)                  │  5547.32ms │
│ Average Time (alamb_update_arrow_58) │  5522.45ms │
│ Queries Faster                       │          1 │
│ Queries Slower                       │          0 │
│ Queries with No Change               │          7 │
│ Queries with Failure                 │          0 │
└──────────────────────────────────────┴────────────┘
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query     ┃        HEAD ┃ alamb_update_arrow_58 ┃        Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0  │     1.90 ms │               1.91 ms │     no change │
│ QQuery 1  │    49.91 ms │              48.58 ms │     no change │
│ QQuery 2  │   132.03 ms │             153.61 ms │  1.16x slower │
│ QQuery 3  │   141.89 ms │             158.42 ms │  1.12x slower │
│ QQuery 4  │  1016.58 ms │            1068.43 ms │  1.05x slower │
│ QQuery 5  │  1325.37 ms │            1374.84 ms │     no change │
│ QQuery 6  │     1.86 ms │               1.82 ms │     no change │
│ QQuery 7  │    53.75 ms │              52.63 ms │     no change │
│ QQuery 8  │  1361.59 ms │            1404.68 ms │     no change │
│ QQuery 9  │  1684.09 ms │            1813.71 ms │  1.08x slower │
│ QQuery 10 │   334.38 ms │             335.15 ms │     no change │
│ QQuery 11 │   384.54 ms │             385.18 ms │     no change │
│ QQuery 12 │  1214.20 ms │            1267.57 ms │     no change │
│ QQuery 13 │  1914.69 ms │            1973.99 ms │     no change │
│ QQuery 14 │  1218.81 ms │            1240.57 ms │     no change │
│ QQuery 15 │  1179.55 ms │            1228.15 ms │     no change │
│ QQuery 16 │  2501.94 ms │            2529.52 ms │     no change │
│ QQuery 17 │  2442.61 ms │            2490.21 ms │     no change │
│ QQuery 18 │  5251.83 ms │            4881.45 ms │ +1.08x faster │
│ QQuery 19 │   119.28 ms │             121.29 ms │     no change │
│ QQuery 20 │  1857.91 ms │            1861.78 ms │     no change │
│ QQuery 21 │  2160.58 ms │            2169.40 ms │     no change │
│ QQuery 22 │  3783.17 ms │            3695.71 ms │     no change │
│ QQuery 23 │ 12272.12 ms │           12133.58 ms │     no change │
│ QQuery 24 │   214.04 ms │             210.00 ms │     no change │
│ QQuery 25 │   482.87 ms │             467.83 ms │     no change │
│ QQuery 26 │   221.57 ms │             211.74 ms │     no change │
│ QQuery 27 │  2792.23 ms │            2634.79 ms │ +1.06x faster │
│ QQuery 28 │ 23512.71 ms │           21597.38 ms │ +1.09x faster │
│ QQuery 29 │   968.00 ms │             980.25 ms │     no change │
│ QQuery 30 │  1314.72 ms │            1300.64 ms │     no change │
│ QQuery 31 │  1317.75 ms │            1325.68 ms │     no change │
│ QQuery 32 │  5876.92 ms │            4767.81 ms │ +1.23x faster │
│ QQuery 33 │  5882.52 ms │            5387.12 ms │ +1.09x faster │
│ QQuery 34 │  5607.56 ms │            5594.88 ms │     no change │
│ QQuery 35 │  1891.29 ms │            1912.68 ms │     no change │
│ QQuery 36 │    69.16 ms │              66.44 ms │     no change │
│ QQuery 37 │    45.36 ms │              48.23 ms │  1.06x slower │
│ QQuery 38 │    66.01 ms │              66.52 ms │     no change │
│ QQuery 39 │   103.26 ms │              99.87 ms │     no change │
│ QQuery 40 │    27.83 ms │              27.12 ms │     no change │
│ QQuery 41 │    22.62 ms │              23.68 ms │     no change │
│ QQuery 42 │    19.63 ms │              19.91 ms │     no change │
└───────────┴─────────────┴───────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary                    ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD)                    │ 92840.64ms │
│ Total Time (alamb_update_arrow_58)   │ 89134.75ms │
│ Average Time (HEAD)                  │  2159.08ms │
│ Average Time (alamb_update_arrow_58) │  2072.90ms │
│ Queries Faster                       │          5 │
│ Queries Slower                       │          5 │
│ Queries with No Change               │         33 │
│ Queries with Failure                 │          0 │
└──────────────────────────────────────┴────────────┘
--------------------
Benchmark tpch_mem_sf1.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query     ┃      HEAD ┃ alamb_update_arrow_58 ┃        Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 1  │ 140.58 ms │             114.58 ms │ +1.23x faster │
│ QQuery 2  │  38.55 ms │              36.08 ms │ +1.07x faster │
│ QQuery 3  │  36.18 ms │              41.04 ms │  1.13x slower │
│ QQuery 4  │  31.34 ms │              30.64 ms │     no change │
│ QQuery 5  │  91.39 ms │              84.21 ms │ +1.09x faster │
│ QQuery 6  │  20.91 ms │              20.72 ms │     no change │
│ QQuery 7  │ 229.89 ms │             210.69 ms │ +1.09x faster │
│ QQuery 8  │  40.58 ms │              44.74 ms │  1.10x slower │
│ QQuery 9  │ 103.62 ms │             117.21 ms │  1.13x slower │
│ QQuery 10 │  69.19 ms │              72.20 ms │     no change │
│ QQuery 11 │  23.43 ms │              26.18 ms │  1.12x slower │
│ QQuery 12 │  51.56 ms │              63.28 ms │  1.23x slower │
│ QQuery 13 │  49.09 ms │              58.61 ms │  1.19x slower │
│ QQuery 14 │  15.53 ms │              15.80 ms │     no change │
│ QQuery 15 │  30.07 ms │              33.55 ms │  1.12x slower │
│ QQuery 16 │  29.19 ms │              30.20 ms │     no change │
│ QQuery 17 │ 148.80 ms │             191.68 ms │  1.29x slower │
│ QQuery 18 │ 286.26 ms │             279.39 ms │     no change │
│ QQuery 19 │  38.92 ms │              38.99 ms │     no change │
│ QQuery 20 │  56.12 ms │              61.36 ms │  1.09x slower │
│ QQuery 21 │ 310.10 ms │             322.87 ms │     no change │
│ QQuery 22 │  21.79 ms │              22.00 ms │     no change │
└───────────┴───────────┴───────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Benchmark Summary                    ┃           ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ Total Time (HEAD)                    │ 1863.09ms │
│ Total Time (alamb_update_arrow_58)   │ 1916.04ms │
│ Average Time (HEAD)                  │   84.69ms │
│ Average Time (alamb_update_arrow_58) │   87.09ms │
│ Queries Faster                       │         4 │
│ Queries Slower                       │         9 │
│ Queries with No Change               │         9 │
│ Queries with Failure                 │         0 │
└──────────────────────────────────────┴───────────┘

alamb-ghbot · 2026-01-16T13:43:31Z

🤖 ./gh_compare_branch.sh gh_compare_branch.sh Running
Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing alamb/update_arrow_58 (1002408) to b9a3b9f diff using: tpch
Results will be posted here when complete

alamb-ghbot · 2026-01-16T13:44:09Z

🤖: Benchmark completed

Details

Comparing HEAD and alamb_update_arrow_58
--------------------
Benchmark tpch_sf1.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query     ┃      HEAD ┃ alamb_update_arrow_58 ┃        Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 1  │ 197.26 ms │             195.54 ms │     no change │
│ QQuery 2  │ 101.24 ms │             100.20 ms │     no change │
│ QQuery 3  │ 132.62 ms │             125.34 ms │ +1.06x faster │
│ QQuery 4  │  76.33 ms │              76.87 ms │     no change │
│ QQuery 5  │ 178.44 ms │             167.44 ms │ +1.07x faster │
│ QQuery 6  │  69.24 ms │              66.58 ms │     no change │
│ QQuery 7  │ 216.81 ms │             206.84 ms │     no change │
│ QQuery 8  │ 164.65 ms │             165.07 ms │     no change │
│ QQuery 9  │ 231.75 ms │             226.05 ms │     no change │
│ QQuery 10 │ 200.95 ms │             180.96 ms │ +1.11x faster │
│ QQuery 11 │  80.23 ms │              81.95 ms │     no change │
│ QQuery 12 │ 119.42 ms │             112.77 ms │ +1.06x faster │
│ QQuery 13 │ 224.21 ms │             219.75 ms │     no change │
│ QQuery 14 │  89.36 ms │              90.83 ms │     no change │
│ QQuery 15 │ 124.17 ms │             124.40 ms │     no change │
│ QQuery 16 │  59.39 ms │              59.88 ms │     no change │
│ QQuery 17 │ 270.86 ms │             304.88 ms │  1.13x slower │
│ QQuery 18 │ 319.99 ms │             309.80 ms │     no change │
│ QQuery 19 │ 135.77 ms │             132.16 ms │     no change │
│ QQuery 20 │ 131.78 ms │             130.69 ms │     no change │
│ QQuery 21 │ 270.89 ms │             265.24 ms │     no change │
│ QQuery 22 │  46.98 ms │              47.08 ms │     no change │
└───────────┴───────────┴───────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Benchmark Summary                    ┃           ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ Total Time (HEAD)                    │ 3442.33ms │
│ Total Time (alamb_update_arrow_58)   │ 3390.29ms │
│ Average Time (HEAD)                  │  156.47ms │
│ Average Time (alamb_update_arrow_58) │  154.10ms │
│ Queries Faster                       │         4 │
│ Queries Slower                       │         1 │
│ Queries with No Change               │        17 │
│ Queries with Failure                 │         0 │
└──────────────────────────────────────┴───────────┘

alamb-ghbot · 2026-01-16T13:44:12Z

🤖 ./gh_compare_branch.sh gh_compare_branch.sh Running
Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing alamb/update_arrow_58 (1002408) to b9a3b9f diff using: tpcds
Results will be posted here when complete

alamb-ghbot · 2026-01-16T13:53:08Z

🤖: Benchmark completed

Details

Comparing HEAD and alamb_update_arrow_58
--------------------
Benchmark tpcds_sf1.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┓
┃ Query     ┃        HEAD ┃ alamb_update_arrow_58 ┃       Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━┩
│ QQuery 1  │    72.26 ms │              71.31 ms │    no change │
│ QQuery 2  │   208.91 ms │             220.31 ms │ 1.05x slower │
│ QQuery 3  │   155.73 ms │             159.96 ms │    no change │
│ QQuery 4  │  1954.88 ms │            1927.14 ms │    no change │
│ QQuery 5  │   280.34 ms │             287.87 ms │    no change │
│ QQuery 6  │  1470.12 ms │            1471.41 ms │    no change │
│ QQuery 7  │   492.07 ms │             479.17 ms │    no change │
│ QQuery 8  │   176.03 ms │             177.82 ms │    no change │
│ QQuery 9  │   290.91 ms │             302.12 ms │    no change │
│ QQuery 10 │   171.12 ms │             173.35 ms │    no change │
│ QQuery 11 │  1276.13 ms │            1310.47 ms │    no change │
│ QQuery 12 │    73.82 ms │              73.98 ms │    no change │
│ QQuery 13 │   547.13 ms │             560.92 ms │    no change │
│ QQuery 14 │  1992.25 ms │            2107.12 ms │ 1.06x slower │
│ QQuery 15 │    31.53 ms │              33.10 ms │ 1.05x slower │
│ QQuery 16 │    65.97 ms │              67.90 ms │    no change │
│ QQuery 17 │   359.25 ms │             356.30 ms │    no change │
│ QQuery 18 │   197.16 ms │             187.50 ms │    no change │
│ QQuery 19 │   226.92 ms │             233.85 ms │    no change │
│ QQuery 20 │    26.79 ms │              27.98 ms │    no change │
│ QQuery 21 │    41.59 ms │              42.88 ms │    no change │
│ QQuery 22 │   783.99 ms │             873.70 ms │ 1.11x slower │
│ QQuery 23 │  1829.91 ms │            1953.86 ms │ 1.07x slower │
│ QQuery 24 │   658.69 ms │             639.09 ms │    no change │
│ QQuery 25 │   508.64 ms │             498.77 ms │    no change │
│ QQuery 26 │   127.93 ms │             123.76 ms │    no change │
│ QQuery 27 │   490.82 ms │             466.51 ms │    no change │
│ QQuery 28 │   306.09 ms │             308.24 ms │    no change │
│ QQuery 29 │   440.36 ms │             435.06 ms │    no change │
│ QQuery 30 │    74.53 ms │              71.38 ms │    no change │
│ QQuery 31 │   306.94 ms │             309.13 ms │    no change │
│ QQuery 32 │    82.91 ms │              84.48 ms │    no change │
│ QQuery 33 │   206.11 ms │             210.12 ms │    no change │
│ QQuery 34 │   157.46 ms │             166.32 ms │ 1.06x slower │
│ QQuery 35 │   173.99 ms │             171.55 ms │    no change │
│ QQuery 36 │   292.07 ms │             293.08 ms │    no change │
│ QQuery 37 │   258.01 ms │             256.35 ms │    no change │
│ QQuery 38 │   154.53 ms │             153.35 ms │    no change │
│ QQuery 39 │   229.87 ms │             230.70 ms │    no change │
│ QQuery 40 │   196.47 ms │             196.69 ms │    no change │
│ QQuery 41 │    23.49 ms │              25.27 ms │ 1.08x slower │
│ QQuery 42 │   140.24 ms │             149.55 ms │ 1.07x slower │
│ QQuery 43 │   126.90 ms │             126.49 ms │    no change │
│ QQuery 44 │    29.16 ms │              30.44 ms │    no change │
│ QQuery 45 │    87.14 ms │              85.45 ms │    no change │
│ QQuery 46 │   319.59 ms │             322.35 ms │    no change │
│ QQuery 47 │  1138.41 ms │            1160.88 ms │    no change │
│ QQuery 48 │   415.17 ms │             422.17 ms │    no change │
│ QQuery 49 │   368.96 ms │             375.35 ms │    no change │
│ QQuery 50 │   339.51 ms │             353.45 ms │    no change │
│ QQuery 51 │   301.94 ms │             301.69 ms │    no change │
│ QQuery 52 │   143.90 ms │             149.70 ms │    no change │
│ QQuery 53 │   152.38 ms │             156.61 ms │    no change │
│ QQuery 54 │   215.33 ms │             237.93 ms │ 1.10x slower │
│ QQuery 55 │   142.08 ms │             147.76 ms │    no change │
│ QQuery 56 │   205.09 ms │             214.49 ms │    no change │
│ QQuery 57 │   311.86 ms │             315.02 ms │    no change │
│ QQuery 58 │   511.16 ms │             519.75 ms │    no change │
│ QQuery 59 │   286.97 ms │             301.94 ms │ 1.05x slower │
│ QQuery 60 │   211.36 ms │             220.85 ms │    no change │
│ QQuery 61 │   245.65 ms │             245.13 ms │    no change │
│ QQuery 62 │  1278.23 ms │            1383.32 ms │ 1.08x slower │
│ QQuery 63 │   152.01 ms │             156.06 ms │    no change │
│ QQuery 64 │  1175.83 ms │            1135.56 ms │    no change │
│ QQuery 65 │   352.20 ms │             382.15 ms │ 1.09x slower │
│ QQuery 66 │   388.22 ms │             399.77 ms │    no change │
│ QQuery 67 │   551.47 ms │             568.52 ms │    no change │
│ QQuery 68 │   374.98 ms │             385.42 ms │    no change │
│ QQuery 69 │   168.37 ms │             170.78 ms │    no change │
│ QQuery 70 │   500.74 ms │             514.12 ms │    no change │
│ QQuery 71 │   186.77 ms │             188.45 ms │    no change │
│ QQuery 72 │  2458.03 ms │            2395.14 ms │    no change │
│ QQuery 73 │   153.13 ms │             161.50 ms │ 1.05x slower │
│ QQuery 74 │   808.16 ms │             851.74 ms │ 1.05x slower │
│ QQuery 75 │   398.97 ms │             415.56 ms │    no change │
│ QQuery 76 │   190.42 ms │             198.54 ms │    no change │
│ QQuery 77 │   288.56 ms │             298.49 ms │    no change │
│ QQuery 78 │   947.43 ms │             964.75 ms │    no change │
│ QQuery 79 │   326.88 ms │             358.41 ms │ 1.10x slower │
│ QQuery 80 │   509.26 ms │             507.79 ms │    no change │
│ QQuery 81 │    52.75 ms │              52.27 ms │    no change │
│ QQuery 82 │   291.05 ms │             298.75 ms │    no change │
│ QQuery 83 │    87.41 ms │              87.49 ms │    no change │
│ QQuery 84 │    69.39 ms │              73.07 ms │ 1.05x slower │
│ QQuery 85 │   222.88 ms │             227.17 ms │    no change │
│ QQuery 86 │    60.60 ms │              62.18 ms │    no change │
│ QQuery 87 │   148.32 ms │             152.13 ms │    no change │
│ QQuery 88 │   266.15 ms │             267.20 ms │    no change │
│ QQuery 89 │   171.88 ms │             170.82 ms │    no change │
│ QQuery 90 │    46.02 ms │              46.46 ms │    no change │
│ QQuery 91 │    97.95 ms │             101.38 ms │    no change │
│ QQuery 92 │    82.25 ms │              85.47 ms │    no change │
│ QQuery 93 │   265.49 ms │             270.46 ms │    no change │
│ QQuery 94 │    90.55 ms │              94.09 ms │    no change │
│ QQuery 95 │   265.12 ms │             276.74 ms │    no change │
│ QQuery 96 │   114.23 ms │             114.12 ms │    no change │
│ QQuery 97 │   180.95 ms │             186.06 ms │    no change │
│ QQuery 98 │   231.62 ms │             237.11 ms │    no change │
│ QQuery 99 │ 14086.96 ms │           15853.27 ms │ 1.13x slower │
└───────────┴─────────────┴───────────────────────┴──────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary                    ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD)                    │ 51147.74ms │
│ Total Time (alamb_update_arrow_58)   │ 53669.19ms │
│ Average Time (HEAD)                  │   516.64ms │
│ Average Time (alamb_update_arrow_58) │   542.11ms │
│ Queries Faster                       │          0 │
│ Queries Slower                       │         17 │
│ Queries with No Change               │         82 │
│ Queries with Failure                 │          0 │
└──────────────────────────────────────┴────────────┘

Dandandan · 2026-01-16T15:40:15Z

run benchmarks

alamb-ghbot · 2026-01-16T15:40:19Z

🤖 ./gh_compare_branch.sh gh_compare_branch.sh Running
Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing alamb/update_arrow_58 (c062756) to 3d90d4b diff using: tpch_mem clickbench_partitioned clickbench_extended
Results will be posted here when complete

alamb-ghbot · 2026-01-16T16:21:33Z

🤖: Benchmark completed

Details

Comparing HEAD and alamb_update_arrow_58
--------------------
Benchmark clickbench_extended.json
--------------------
┏━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Query    ┃        HEAD ┃ alamb_update_arrow_58 ┃    Change ┃
┡━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ QQuery 0 │  2483.50 ms │            2532.76 ms │ no change │
│ QQuery 1 │   939.44 ms │             953.44 ms │ no change │
│ QQuery 2 │  1929.70 ms │            1858.07 ms │ no change │
│ QQuery 3 │  1124.50 ms │            1120.99 ms │ no change │
│ QQuery 4 │  2260.53 ms │            2185.89 ms │ no change │
│ QQuery 5 │ 29088.67 ms │           28653.06 ms │ no change │
│ QQuery 6 │  4034.33 ms │            4065.03 ms │ no change │
│ QQuery 7 │  3159.40 ms │            3122.77 ms │ no change │
└──────────┴─────────────┴───────────────────────┴───────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary                    ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD)                    │ 45020.08ms │
│ Total Time (alamb_update_arrow_58)   │ 44492.01ms │
│ Average Time (HEAD)                  │  5627.51ms │
│ Average Time (alamb_update_arrow_58) │  5561.50ms │
│ Queries Faster                       │          0 │
│ Queries Slower                       │          0 │
│ Queries with No Change               │          8 │
│ Queries with Failure                 │          0 │
└──────────────────────────────────────┴────────────┘
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query     ┃        HEAD ┃ alamb_update_arrow_58 ┃        Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0  │     1.96 ms │               2.07 ms │  1.06x slower │
│ QQuery 1  │    52.23 ms │              48.92 ms │ +1.07x faster │
│ QQuery 2  │   129.82 ms │             154.75 ms │  1.19x slower │
│ QQuery 3  │   152.93 ms │             162.13 ms │  1.06x slower │
│ QQuery 4  │  1147.91 ms │            1129.45 ms │     no change │
│ QQuery 5  │  1436.22 ms │            1416.48 ms │     no change │
│ QQuery 6  │     1.85 ms │               2.17 ms │  1.17x slower │
│ QQuery 7  │    55.49 ms │              54.14 ms │     no change │
│ QQuery 8  │  1489.13 ms │            1462.28 ms │     no change │
│ QQuery 9  │  1766.19 ms │            1836.98 ms │     no change │
│ QQuery 10 │   351.88 ms │             337.02 ms │     no change │
│ QQuery 11 │   397.55 ms │             389.90 ms │     no change │
│ QQuery 12 │  1319.57 ms │            1336.45 ms │     no change │
│ QQuery 13 │  2016.60 ms │            1979.70 ms │     no change │
│ QQuery 14 │  1291.47 ms │            1260.06 ms │     no change │
│ QQuery 15 │  1315.36 ms │            1250.26 ms │     no change │
│ QQuery 16 │  2626.21 ms │            2629.03 ms │     no change │
│ QQuery 17 │  2613.24 ms │            2557.90 ms │     no change │
│ QQuery 18 │  5837.15 ms │            4916.95 ms │ +1.19x faster │
│ QQuery 19 │   122.36 ms │             120.89 ms │     no change │
│ QQuery 20 │  1955.63 ms │            1948.01 ms │     no change │
│ QQuery 21 │  2295.84 ms │            2224.75 ms │     no change │
│ QQuery 22 │  4126.66 ms │            3819.18 ms │ +1.08x faster │
│ QQuery 23 │ 12661.80 ms │           12160.36 ms │     no change │
│ QQuery 24 │   224.37 ms │             209.76 ms │ +1.07x faster │
│ QQuery 25 │   483.01 ms │             472.20 ms │     no change │
│ QQuery 26 │   220.85 ms │             218.13 ms │     no change │
│ QQuery 27 │  2802.49 ms │            2678.52 ms │     no change │
│ QQuery 28 │ 23733.50 ms │           23580.59 ms │     no change │
│ QQuery 29 │   975.95 ms │             963.51 ms │     no change │
│ QQuery 30 │  1346.03 ms │            1280.44 ms │     no change │
│ QQuery 31 │  1353.12 ms │            1310.27 ms │     no change │
│ QQuery 32 │  4588.90 ms │            4629.69 ms │     no change │
│ QQuery 33 │  5849.47 ms │            5605.52 ms │     no change │
│ QQuery 34 │  6003.17 ms │            6226.09 ms │     no change │
│ QQuery 35 │  2050.00 ms │            1944.86 ms │ +1.05x faster │
│ QQuery 36 │    70.40 ms │              69.31 ms │     no change │
│ QQuery 37 │    49.09 ms │              47.47 ms │     no change │
│ QQuery 38 │    69.75 ms │              68.12 ms │     no change │
│ QQuery 39 │   104.89 ms │             103.10 ms │     no change │
│ QQuery 40 │    28.35 ms │              27.97 ms │     no change │
│ QQuery 41 │    23.11 ms │              23.69 ms │     no change │
│ QQuery 42 │    21.92 ms │              21.39 ms │     no change │
└───────────┴─────────────┴───────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary                    ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD)                    │ 95163.43ms │
│ Total Time (alamb_update_arrow_58)   │ 92680.49ms │
│ Average Time (HEAD)                  │  2213.10ms │
│ Average Time (alamb_update_arrow_58) │  2155.36ms │
│ Queries Faster                       │          5 │
│ Queries Slower                       │          4 │
│ Queries with No Change               │         34 │
│ Queries with Failure                 │          0 │
└──────────────────────────────────────┴────────────┘
--------------------
Benchmark tpch_mem_sf1.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query     ┃      HEAD ┃ alamb_update_arrow_58 ┃        Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 1  │ 105.80 ms │             104.35 ms │     no change │
│ QQuery 2  │  32.64 ms │              31.73 ms │     no change │
│ QQuery 3  │  41.42 ms │              41.51 ms │     no change │
│ QQuery 4  │  31.46 ms │              30.57 ms │     no change │
│ QQuery 5  │  90.23 ms │              82.92 ms │ +1.09x faster │
│ QQuery 6  │  20.72 ms │              20.50 ms │     no change │
│ QQuery 7  │ 166.40 ms │             158.28 ms │     no change │
│ QQuery 8  │  42.92 ms │              39.23 ms │ +1.09x faster │
│ QQuery 9  │ 108.70 ms │              95.45 ms │ +1.14x faster │
│ QQuery 10 │  68.29 ms │              61.73 ms │ +1.11x faster │
│ QQuery 11 │  18.29 ms │              18.24 ms │     no change │
│ QQuery 12 │  52.25 ms │              50.99 ms │     no change │
│ QQuery 13 │  47.84 ms │              47.54 ms │     no change │
│ QQuery 14 │  14.97 ms │              14.82 ms │     no change │
│ QQuery 15 │  31.11 ms │              29.62 ms │     no change │
│ QQuery 16 │  29.00 ms │              27.49 ms │ +1.06x faster │
│ QQuery 17 │ 146.03 ms │             148.57 ms │     no change │
│ QQuery 18 │ 294.42 ms │             271.63 ms │ +1.08x faster │
│ QQuery 19 │  39.04 ms │              38.19 ms │     no change │
│ QQuery 20 │  56.75 ms │              57.03 ms │     no change │
│ QQuery 21 │ 188.77 ms │             194.31 ms │     no change │
│ QQuery 22 │  22.74 ms │              21.93 ms │     no change │
└───────────┴───────────┴───────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Benchmark Summary                    ┃           ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ Total Time (HEAD)                    │ 1649.80ms │
│ Total Time (alamb_update_arrow_58)   │ 1586.63ms │
│ Average Time (HEAD)                  │   74.99ms │
│ Average Time (alamb_update_arrow_58) │   72.12ms │
│ Queries Faster                       │         6 │
│ Queries Slower                       │         0 │
│ Queries with No Change               │        16 │
│ Queries with Failure                 │         0 │
└──────────────────────────────────────┴───────────┘

alamb added 10 commits January 6, 2026 17:55

Update to pre-release arrow

120d2b0

Fix clippy due to UnionFields::new is deprecated

40a6a71

Fix real bug in test

f50b9c3

Update test for fixed issue upstream

87563d8

Update tests for new feature

4afd46f

Update tests for struct casting

70e2841

Update to latest

b29b662

Merge remote-tracking branch 'apache/main' into alamb/update_arrow_57…

6a2e631

….2.0

Merge branch 'main' into alamb/update_arrow_57.2.0

ca11966

Update to pre-release main

63a85eb

github-actions bot added sqllogictest SQL Logic Tests (.slt) common Related to common crate proto Related to proto crate functions Changes to functions implementation datasource Changes to the datasource crate physical-plan Changes to the physical-plan crate labels Jan 10, 2026

Update expected value of make_interval

277aac4

alamb commented Jan 10, 2026

View reviewed changes

Update expected output

bb29b0c

alamb commented Jan 10, 2026

View reviewed changes

alamb mentioned this pull request Jan 10, 2026

feat: change default behavior for Parquet PageEncodingStats to bitmask apache/arrow-rs#9051

Merged

Update to latest arrow

35b97fa

alamb mentioned this pull request Jan 10, 2026

fix: liberal parsing of zero scale decimals apache/arrow-rs#8700

Open

Update rev

d2e8da6

update outputs

1002408

Dandandan reviewed Jan 16, 2026

View reviewed changes

datafusion/sqllogictest/test_files/window.slt

);

----

1

DataFusion error: Join Error

Copy link

Contributor

Dandandan Jan 16, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This somehow errors now 🤔

Merge

c062756

github-actions bot removed common Related to common crate proto Related to proto crate functions Changes to functions implementation datasource Changes to the datasource crate physical-plan Changes to the physical-plan crate labels Jan 16, 2026

WIP: Upgrade DataFusion to arrow-rs/parquet 58.0.0 #19728

Are you sure you want to change the base?

WIP: Upgrade DataFusion to arrow-rs/parquet 58.0.0 #19728

Conversation

alamb commented Jan 10, 2026

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

alamb Jan 10, 2026

Choose a reason for hiding this comment

Uh oh!

davidlghellin Jan 10, 2026

Choose a reason for hiding this comment

Uh oh!

Jefffrey Jan 10, 2026

Choose a reason for hiding this comment

Uh oh!

alamb Jan 10, 2026

Choose a reason for hiding this comment

Uh oh!

Dandandan commented Jan 10, 2026

Uh oh!

alamb-ghbot commented Jan 10, 2026

Uh oh!

alamb-ghbot commented Jan 10, 2026

Uh oh!

Dandandan commented Jan 10, 2026

Uh oh!

Dandandan commented Jan 10, 2026

Uh oh!

alamb-ghbot commented Jan 10, 2026

Uh oh!

alamb-ghbot commented Jan 10, 2026

Uh oh!

alamb-ghbot commented Jan 10, 2026

Uh oh!

alamb-ghbot commented Jan 10, 2026

Uh oh!

Dandandan commented Jan 16, 2026

Uh oh!

alamb commented Jan 16, 2026

Uh oh!

Dandandan commented Jan 16, 2026

Uh oh!

Dandandan commented Jan 16, 2026

Uh oh!

Dandandan Jan 16, 2026

Choose a reason for hiding this comment

Uh oh!

alamb-ghbot commented Jan 16, 2026

Uh oh!

alamb-ghbot commented Jan 16, 2026

Uh oh!

alamb-ghbot commented Jan 16, 2026

Uh oh!

alamb-ghbot commented Jan 16, 2026

Uh oh!

alamb-ghbot commented Jan 16, 2026

Uh oh!

alamb-ghbot commented Jan 16, 2026

Uh oh!

Dandandan commented Jan 16, 2026

Uh oh!

alamb-ghbot commented Jan 16, 2026

Uh oh!

alamb-ghbot commented Jan 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants