-To execute MERGE, 2 join operations are required. The first one is joining whole target table and source data, to find *touched* files including any matched row. The other one is for actual MERGE operation only with *touched* files of the target table. The first join is lighter as it only reads columns in matching condition. Although Delta performs the first join to reduce the amount of data for the actual merge process, huge amount of *unmodified* rows in *touched* files goes through the second join process. With Low Shuffle Merge, Delta retreives "matched" rows result from the first join and utilizes it for classifying *matched* rows. Based on that, there are 2 separate write jobs for *matched* rows and *unmodified* rows, so it could result in 2x number of output files compared to default MERGE operation. The expected performance gain outweighs the possible small files problem.
0 commit comments