Avoid repartition in build side #19812

Dandandan · 2026-01-14T13:27:21Z

Which issue does this PR close?

Closes Reduce overhead of Partitioned hash join #19789

Rationale for this change

--------------------
Benchmark tpch_mem_sf1.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query     ┃      main ┃ avoid_repartition ┃        Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 1  │  41.74 ms │          43.67 ms │     no change │
│ QQuery 2  │  11.68 ms │           9.89 ms │ +1.18x faster │
│ QQuery 3  │  13.65 ms │          13.33 ms │     no change │
│ QQuery 4  │  11.53 ms │          11.80 ms │     no change │
│ QQuery 5  │  32.06 ms │          24.03 ms │ +1.33x faster │
│ QQuery 6  │  10.96 ms │          11.39 ms │     no change │
│ QQuery 7  │  99.81 ms │          65.82 ms │ +1.52x faster │
│ QQuery 8  │  15.80 ms │          15.75 ms │     no change │
│ QQuery 9  │  38.95 ms │          36.56 ms │ +1.07x faster │
│ QQuery 10 │  25.26 ms │          24.11 ms │     no change │
│ QQuery 11 │   6.65 ms │           5.94 ms │ +1.12x faster │
│ QQuery 12 │  21.67 ms │          22.05 ms │     no change │
│ QQuery 13 │  16.31 ms │          24.34 ms │  1.49x slower │
│ QQuery 14 │   5.13 ms │           5.90 ms │  1.15x slower │
│ QQuery 15 │  10.33 ms │          10.69 ms │     no change │
│ QQuery 16 │  11.11 ms │          11.60 ms │     no change │
│ QQuery 17 │  54.08 ms │          52.33 ms │     no change │
│ QQuery 18 │  90.86 ms │         155.74 ms │  1.71x slower │
│ QQuery 19 │  16.36 ms │          17.35 ms │  1.06x slower │
│ QQuery 20 │  17.50 ms │          19.69 ms │  1.13x slower │
│ QQuery 21 │ 197.76 ms │         140.10 ms │ +1.41x faster │
│ QQuery 22 │   7.01 ms │           7.45 ms │  1.06x slower │
└───────────┴───────────┴───────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┓
┃ Benchmark Summary                ┃          ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━┩
│ Total Time (main)                │ 756.22ms │
│ Total Time (avoid_repartition)   │ 729.53ms │
│ Average Time (main)              │  34.37ms │
│ Average Time (avoid_repartition) │  33.16ms │
│ Queries Faster                   │        6 │
│ Queries Slower                   │        6 │
│ Queries with No Change           │       10 │
│ Queries with Failure             │        0 │
└──────────────────────────────────┴──────────┘
Note: Skipping /Users/danielheres/Code/arrow-datafusion-personal/benchmarks/results/main/tpch_mem_sf10.json as /Users/danielheres/Code/arrow-datafusion-personal/benchmarks/results/avoid_repartition/tpch_mem_sf10.json does not exist
--------------------
Benchmark tpch_sf1.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query     ┃      main ┃ avoid_repartition ┃        Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 1  │  76.46 ms │          76.50 ms │     no change │
│ QQuery 2  │  47.93 ms │          43.11 ms │ +1.11x faster │
│ QQuery 3  │  51.11 ms │          50.46 ms │     no change │
│ QQuery 4  │  35.43 ms │          36.82 ms │     no change │
│ QQuery 5  │  76.12 ms │          62.88 ms │ +1.21x faster │
│ QQuery 6  │  24.54 ms │          24.53 ms │     no change │
│ QQuery 7  │  85.53 ms │          79.32 ms │ +1.08x faster │
│ QQuery 8  │  80.80 ms │          65.65 ms │ +1.23x faster │
│ QQuery 9  │  95.69 ms │          78.09 ms │ +1.23x faster │
│ QQuery 10 │  94.35 ms │          89.74 ms │     no change │
│ QQuery 11 │  35.24 ms │          29.10 ms │ +1.21x faster │
│ QQuery 12 │  57.99 ms │          56.96 ms │     no change │
│ QQuery 13 │ 118.43 ms │         144.93 ms │  1.22x slower │
│ QQuery 14 │  42.71 ms │          36.28 ms │ +1.18x faster │
│ QQuery 15 │  41.52 ms │          40.29 ms │     no change │
│ QQuery 16 │  28.43 ms │          28.62 ms │     no change │
│ QQuery 17 │ 105.80 ms │          98.02 ms │ +1.08x faster │
│ QQuery 18 │ 130.43 ms │         162.52 ms │  1.25x slower │
│ QQuery 19 │  57.52 ms │          57.42 ms │     no change │
│ QQuery 20 │  58.49 ms │          53.07 ms │ +1.10x faster │
│ QQuery 21 │  98.25 ms │          86.65 ms │ +1.13x faster │
│ QQuery 22 │  22.68 ms │          19.48 ms │ +1.16x faster │
└───────────┴───────────┴───────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Benchmark Summary                ┃           ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ Total Time (main)                │ 1465.45ms │
│ Total Time (avoid_repartition)   │ 1420.43ms │
│ Average Time (main)              │   66.61ms │
│ Average Time (avoid_repartition) │   64.56ms │
│ Queries Faster                   │        11 │
│ Queries Slower                   │         2 │
│ Queries with No Change           │         9 │
│ Queries with Failure             │         0 │
└──────────────────────────────────┴───────────┘

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Dandandan · 2026-01-14T14:16:41Z

Run benchmarks

Dandandan · 2026-01-14T16:29:45Z

Run benchmarks

Dandandan · 2026-01-14T19:12:55Z

Run benchmarks

adriangb · 2026-01-15T04:42:30Z

@Dandandan any breadcrumbs on the idea here?

Dandandan · 2026-01-15T06:47:27Z

@adriangb
The idea is written down here: #19789

Basically I would like to avoid copying the entire build side twice (as is done currently in RepartitionExec in take/coalesce) and only handle the matching indices during the hash join.

Dandandan · 2026-01-15T07:13:04Z

I updated the description with some early bench results - on average quite a bit faster with some (one bigger 1.90x slower QQuery 12) slowdowns, need to find out the source of slowdowns.

Dandandan · 2026-01-15T08:29:09Z

I updated the description with some early bench results - on average quite a bit faster with some (one bigger 1.90x slower QQuery 12) slowdowns, need to find out the source of slowdowns.

This was because of missing parallelism, updated results with parallel build.

(Need to fix null handling next)

Dandandan · 2026-01-15T11:26:16Z

Updated now with correct output - the results seem a bit more mixed now (but still positive on average).

Going to see if we can improve the slowdowns again.

adriangb · 2026-01-15T11:43:12Z

datafusion/core/tests/physical_optimizer/filter_pushdown/mod.rs

    -       RepartitionExec: partitioning=Hash([a@0, b@1], 12), input_partitions=1
    -         DataSourceExec: file_groups={1 group: [[test.parquet]]}, projection=[a, b, c], file_type=test, pushdown_supported=true
    -       RepartitionExec: partitioning=Hash([a@0, b@1], 12), input_partitions=1
-    -         DataSourceExec: file_groups={1 group: [[test.parquet]]}, projection=[a, b, e], file_type=test, pushdown_supported=true, predicate=DynamicFilter [ CASE hash_repartition % 12 WHEN 2 THEN a@0 >= ab AND a@0 <= ab AND b@1 >= bb AND b@1 <= bb AND struct(a@0, b@1) IN (SET) ([{c0:ab,c1:bb}]) WHEN 4 THEN a@0 >= aa AND a@0 <= aa AND b@1 >= ba AND b@1 <= ba AND struct(a@0, b@1) IN (SET) ([{c0:aa,c1:ba}]) ELSE false END ]


I guess it's not possible to do filtering during the scan anymore with this approach?

I think it should be possible - but somehow it got removed during the AI-based generation ;)

This reverts commit d03a6f6.

This reverts commit 661e716.

Dandandan · 2026-01-15T18:49:58Z

Run benchmark tpcds

alamb-ghbot · 2026-01-15T19:07:06Z

🤖 ./gh_compare_branch.sh gh_compare_branch.sh Running
Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing avoid_repartition (4bced79) to 79869a7 diff using: tpcds
Results will be posted here when complete

alamb-ghbot · 2026-01-15T19:35:59Z

🤖: Benchmark completed

Details

Comparing HEAD and avoid_repartition
--------------------
Benchmark tpcds_sf1.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query     ┃        HEAD ┃ avoid_repartition ┃        Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 1  │    74.02 ms │          74.73 ms │     no change │
│ QQuery 2  │   217.61 ms │         219.12 ms │     no change │
│ QQuery 3  │   163.73 ms │         168.84 ms │     no change │
│ QQuery 4  │  1962.13 ms │        1976.52 ms │     no change │
│ QQuery 5  │   295.59 ms │         295.00 ms │     no change │
│ QQuery 6  │  1443.76 ms │        1501.97 ms │     no change │
│ QQuery 7  │   508.08 ms │         418.33 ms │ +1.21x faster │
│ QQuery 8  │   180.07 ms │         182.34 ms │     no change │
│ QQuery 9  │   302.85 ms │         302.05 ms │     no change │
│ QQuery 10 │   177.85 ms │         174.64 ms │     no change │
│ QQuery 11 │  1327.24 ms │        1315.94 ms │     no change │
│ QQuery 12 │    70.10 ms │          72.26 ms │     no change │
│ QQuery 13 │   554.05 ms │         560.85 ms │     no change │
│ QQuery 14 │  1899.40 ms │        2018.35 ms │  1.06x slower │
│ QQuery 15 │    31.90 ms │          30.75 ms │     no change │
│ QQuery 16 │    64.79 ms │          68.34 ms │  1.05x slower │
│ QQuery 17 │   358.62 ms │         429.32 ms │  1.20x slower │
│ QQuery 18 │   199.83 ms │         170.53 ms │ +1.17x faster │
│ QQuery 19 │   231.42 ms │         231.98 ms │     no change │
│ QQuery 20 │    26.77 ms │          26.88 ms │     no change │
│ QQuery 21 │    40.93 ms │          41.56 ms │     no change │
│ QQuery 22 │   730.98 ms │         710.90 ms │     no change │
│ QQuery 23 │  1789.54 ms │        1879.59 ms │  1.05x slower │
│ QQuery 24 │   651.50 ms │         647.71 ms │     no change │
│ QQuery 25 │   523.76 ms │         495.37 ms │ +1.06x faster │
│ QQuery 26 │   129.29 ms │         119.26 ms │ +1.08x faster │
│ QQuery 27 │   505.50 ms │         414.28 ms │ +1.22x faster │
│ QQuery 28 │   314.82 ms │         310.01 ms │     no change │
│ QQuery 29 │   447.42 ms │         418.66 ms │ +1.07x faster │
│ QQuery 30 │    77.50 ms │          75.64 ms │     no change │
│ QQuery 31 │   326.72 ms │         237.60 ms │ +1.38x faster │
│ QQuery 32 │    84.06 ms │          87.56 ms │     no change │
│ QQuery 33 │   209.89 ms │         210.90 ms │     no change │
│ QQuery 34 │   163.00 ms │         165.65 ms │     no change │
│ QQuery 35 │   174.11 ms │         173.46 ms │     no change │
│ QQuery 36 │   295.34 ms │         299.67 ms │     no change │
│ QQuery 37 │   258.27 ms │         269.79 ms │     no change │
│ QQuery 38 │   158.12 ms │         141.27 ms │ +1.12x faster │
│ QQuery 39 │   208.14 ms │         211.56 ms │     no change │
│ QQuery 40 │   172.74 ms │         270.77 ms │  1.57x slower │
│ QQuery 41 │    23.67 ms │          24.12 ms │     no change │
│ QQuery 42 │   148.37 ms │         149.92 ms │     no change │
│ QQuery 43 │   130.65 ms │         129.81 ms │     no change │
│ QQuery 44 │    29.41 ms │          27.14 ms │ +1.08x faster │
│ QQuery 45 │    87.04 ms │         113.02 ms │  1.30x slower │
│ QQuery 46 │   329.79 ms │         332.09 ms │     no change │
│ QQuery 47 │  1057.17 ms │        1069.55 ms │     no change │
│ QQuery 48 │   427.21 ms │         414.87 ms │     no change │
│ QQuery 49 │   384.80 ms │         382.41 ms │     no change │
│ QQuery 50 │   341.04 ms │         357.50 ms │     no change │
│ QQuery 51 │   305.05 ms │         305.51 ms │     no change │
│ QQuery 52 │   148.29 ms │         150.45 ms │     no change │
│ QQuery 53 │   152.57 ms │         159.32 ms │     no change │
│ QQuery 54 │   220.93 ms │         230.91 ms │     no change │
│ QQuery 55 │   148.37 ms │         149.54 ms │     no change │
│ QQuery 56 │   211.89 ms │         210.97 ms │     no change │
│ QQuery 57 │   305.42 ms │         299.47 ms │     no change │
│ QQuery 58 │   495.02 ms │         503.58 ms │     no change │
│ QQuery 59 │   300.39 ms │         299.68 ms │     no change │
│ QQuery 60 │   218.41 ms │         216.87 ms │     no change │
│ QQuery 61 │   249.86 ms │         255.23 ms │     no change │
│ QQuery 62 │  1331.35 ms │        1301.83 ms │     no change │
│ QQuery 63 │   155.29 ms │         158.04 ms │     no change │
│ QQuery 64 │  1185.68 ms │        1067.54 ms │ +1.11x faster │
│ QQuery 65 │   356.20 ms │         366.47 ms │     no change │
│ QQuery 66 │   402.64 ms │         394.25 ms │     no change │
│ QQuery 67 │   557.57 ms │         553.49 ms │     no change │
│ QQuery 68 │   385.47 ms │         390.07 ms │     no change │
│ QQuery 69 │   175.74 ms │         173.74 ms │     no change │
│ QQuery 70 │   525.09 ms │         524.49 ms │     no change │
│ QQuery 71 │   189.15 ms │         196.57 ms │     no change │
│ QQuery 72 │  2094.57 ms │       62511.41 ms │ 29.84x slower │
│ QQuery 73 │   160.79 ms │         162.04 ms │     no change │
│ QQuery 74 │   861.64 ms │         824.53 ms │     no change │
│ QQuery 75 │   409.00 ms │         415.05 ms │     no change │
│ QQuery 76 │   192.34 ms │         192.83 ms │     no change │
│ QQuery 77 │   298.73 ms │         295.77 ms │     no change │
│ QQuery 78 │   954.20 ms │        1268.19 ms │  1.33x slower │
│ QQuery 79 │   336.59 ms │         338.30 ms │     no change │
│ QQuery 80 │   523.31 ms │         581.74 ms │  1.11x slower │
│ QQuery 81 │    51.98 ms │          52.46 ms │     no change │
│ QQuery 82 │   284.56 ms │         297.18 ms │     no change │
│ QQuery 83 │    79.52 ms │          80.23 ms │     no change │
│ QQuery 84 │    69.43 ms │          69.21 ms │     no change │
│ QQuery 85 │   225.67 ms │         227.14 ms │     no change │
│ QQuery 86 │    60.20 ms │          58.89 ms │     no change │
│ QQuery 87 │   154.67 ms │         142.24 ms │ +1.09x faster │
│ QQuery 88 │   271.44 ms │         272.21 ms │     no change │
│ QQuery 89 │   172.13 ms │         176.02 ms │     no change │
│ QQuery 90 │    46.91 ms │          47.13 ms │     no change │
│ QQuery 91 │    99.55 ms │          96.82 ms │     no change │
│ QQuery 92 │    84.28 ms │          86.22 ms │     no change │
│ QQuery 93 │   270.24 ms │         349.54 ms │  1.29x slower │
│ QQuery 94 │    94.22 ms │          94.53 ms │     no change │
│ QQuery 95 │   243.03 ms │         482.03 ms │  1.98x slower │
│ QQuery 96 │   116.58 ms │         117.71 ms │     no change │
│ QQuery 97 │   191.82 ms │         189.29 ms │     no change │
│ QQuery 98 │   229.14 ms │         228.97 ms │     no change │
│ QQuery 99 │ 14378.35 ms │       14152.96 ms │     no change │
└───────────┴─────────────┴───────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┓
┃ Benchmark Summary                ┃             ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━┩
│ Total Time (HEAD)                │  51285.90ms │
│ Total Time (avoid_repartition)   │ 112139.05ms │
│ Average Time (HEAD)              │    518.04ms │
│ Average Time (avoid_repartition) │   1132.72ms │
│ Queries Faster                   │          11 │
│ Queries Slower                   │          11 │
│ Queries with No Change           │          77 │
│ Queries with Failure             │           0 │
└──────────────────────────────────┴─────────────┘

Dandandan · 2026-01-15T20:51:01Z

|QQuery 72 │ 2094.57 ms │ 62511.41 ms │ 29.84x slower │

Mmmm....

Avoid repartition

dc9b5da

github-actions bot added the physical-plan Changes to the physical-plan crate label Jan 14, 2026

Dandandan added 2 commits January 14, 2026 14:50

Avoid repartition

4339772

Avoid repartition

698ca81

Avoid repartition

504e938

github-actions bot added core Core DataFusion crate sqllogictest SQL Logic Tests (.slt) labels Jan 14, 2026

Dandandan added 2 commits January 14, 2026 22:43

Avoid repartition

22f9296

Avoid repartition

89591ac

Dandandan changed the title ~~Avoid repartition~~ Avoid repartition in build side Jan 15, 2026

github-actions bot added the physical-expr Changes to the physical-expr crates label Jan 15, 2026

Dandandan mentioned this pull request Jan 15, 2026

feat: add LazyPartitioned mode for hash join to reduce RepartitionExec overhead #19808

Open

Dandandan added 2 commits January 15, 2026 08:23

Clippy

103b557

Parallel build

036b5e4

Fix

a7f22c8

Insta update

a66f35f

adriangb reviewed Jan 15, 2026

View reviewed changes

Dandandan added 5 commits January 15, 2026 13:48

Update test outputs

59c18f3

Fix

661e716

Update test outputs

d03a6f6

Revert "Update test outputs"

8943d92

This reverts commit d03a6f6.

Revert "Fix"

ce67f0a

This reverts commit 661e716.

Dandandan added 6 commits January 15, 2026 14:07

Update test outputs

ef6e0b9

Update test outputs

a8af6fb

Update test outputs

7cb03fc

Fix

6dae9b8

Remove temp file

2620924

Remove temp file

4bced79

Avoid repartition in build side #19812

Are you sure you want to change the base?

Avoid repartition in build side #19812

Conversation

Dandandan commented Jan 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

Dandandan commented Jan 14, 2026

Uh oh!

Dandandan commented Jan 14, 2026

Uh oh!

Dandandan commented Jan 14, 2026

Uh oh!

adriangb commented Jan 15, 2026

Uh oh!

Dandandan commented Jan 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Dandandan commented Jan 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Dandandan commented Jan 15, 2026

Uh oh!

Dandandan commented Jan 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

adriangb Jan 15, 2026

Choose a reason for hiding this comment

Uh oh!

Dandandan Jan 15, 2026

Choose a reason for hiding this comment

Uh oh!

Dandandan commented Jan 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alamb-ghbot commented Jan 15, 2026

Uh oh!

alamb-ghbot commented Jan 15, 2026

Uh oh!

Dandandan commented Jan 15, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Dandandan commented Jan 14, 2026 •

edited

Loading

Dandandan commented Jan 15, 2026 •

edited

Loading

Dandandan commented Jan 15, 2026 •

edited

Loading

Dandandan commented Jan 15, 2026 •

edited

Loading

Dandandan commented Jan 15, 2026 •

edited

Loading