Kyuubi 1.19.1 Spark engine small file not merged issue

[zapjone]

Kyuubi 1.19.1 Spark engine cannot automatically merge small files. Do you need to add any parameters? I didn't see the extensions directory either
Spark version is 3.4.3;
Spark configuration:

spark.dynamicAllocation.enabled true
spark.dynamicAllocation.minExecutors 1
spark.dynamicAllocation.maxExecutors 200
spark.dynamicAllocation.shuffleTracking.enabled true
spark.sql.adaptive.enabled true
spark.sql.adaptive.merge.output.small.files.enabled true
spark.sql.adaptive.shuffle.targetPostShuffleInputSize 268435456
spark.sql.adaptive.coalescePartitions.enabled true
spark.sql.adaptive.coalescePartitions.minPartitionNum 1
spark.sql.adaptive.coalescePartitions.initialPartitionNum 1500
spark.sql.adaptive.advisoryPartitionSizeInBytes 268435456
spark.sql.adaptive.advisoryOutputFileSizeInBytes 268435456
spark.sql.optimizer.insertRepartitionBeforeWrite.enabled true

[pan3793]

First of all, why do you think those configurations combination should work?

for example, there are no such configurations in OSS Spark

spark.sql.adaptive.merge.output.small.files.enabled
spark.sql.adaptive.advisoryOutputFileSizeInBytes

and spark.sql.adaptive.shuffle.targetPostShuffleInputSize has been deprecated since Spark 3.0, replaced by spark.sql.adaptive.advisoryPartitionSizeInByte

[zapjone]

spark.sql.adaptive.merge.output.small.files.enabled
spark.sql.adaptive.advisoryOutputFileSizeInBytes
These two configurations were replicated by other frameworks and will not work.

[zapjone]

spark.sql.adaptive.enabled true
spark.sql.adaptive.coalescePartitions.enabled true
spark.sql.adaptive.coalescePartitions.minPartitionNum 1
spark.sql.adaptive.coalescePartitions.initialPartitionNum 1500
spark.sql.adaptive.advisoryPartitionSizeInBytes 268435456

[pan3793]

The small files merge functionality is implemented in Kyuubi Spark Extension, the docs is located at https://kyuubi.readthedocs.io/en/v1.9.1/extensions/engines/spark/rules.html

[zapjone]

Added the extension jar package and enabled Spark. SQL. extensions org. apache. kyuubi. SQL KyuubiSparkSQLExtension， But it still hasn't been merged

[zapjone]

insert overwrite table xxx select * from xxxx; There will be 2 small files in the final target table

[pan3793]

The discussion is archived publicly for further users to search, please write your words seriously. e.g. correctly reference the version in the original question (there is no Kyuubi 1.19.1 but 1.9.1, no configuration key is "Spark. SQL. extensions" but "spark.sql.extensions")

There is another configuration spark.sql.optimizer.insertRepartitionBeforeWriteIfNoShuffle.enabled whose default value is false, which means no extra shuffle will be introduced for queries that don't have shuffle, turn on it if you don't care about query efficiency but only merging small files.

[zapjone]

After adding the parameter spark. SQL. optimizer. insertRepartitionBeforeWriteIfNoShuffle. enabled, small files have indeed been merged for those without shufflers. Thank you.
Also, let me correct the error above: the version is 1.9.1, and the final parameters are as follows:
spark.sql.optimizer.insertRepartitionBeforeWriteIfNoShuffle.enabled=true
spark.sql.optimizer.insertRepartitionBeforeWrite.enabled=true
spark.sql.extensions=org.apache.kyuubi.sql.KyuubiSparkSQLExtension
spark.sql.adaptive.enabled=true
spark.sql.adaptive.coalescePartitions.enabled=true
spark.sql.adaptive.coalescePartitions.minPartitionNum=1
spark.sql.adaptive.coalescePartitions.initialPartitionNum=1500
spark.sql.adaptive.advisoryPartitionSizeInBytes=512M