Skip to content

Commit b2d441f

Browse files
prashantgolashfacebook-github-bot
authored andcommitted
feat: Session property to control file size for presto writer (prestodb#27054)
Summary: Session property to control the file size for presto writers Differential Revision: D91361183
1 parent feb403b commit b2d441f

File tree

2 files changed

+18
-1
lines changed

2 files changed

+18
-1
lines changed

presto-docs/src/main/sphinx/presto_cpp/properties-session.rst

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -175,6 +175,16 @@ Use ``native_max_spill_file_size`` to manage disk space usage during operations
175175
Native Execution only. The maximum allowed spilling level for hash join build.
176176
``0`` is the initial spilling level, ``-1`` means unlimited.
177177

178+
``native_max_target_file_size``
179+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
180+
181+
* **Type:** ``bigint``
182+
* **Default value:** ``0``
183+
184+
Native Execution only. Controls the maximum target file size in bytes for writers.
185+
When a file exceeds this size during writing, the writer will close the current file
186+
and start writing to a new file. If set to ``0``, there is no limit on the target file size.
187+
178188
``native_order_by_spill_memory_threshold``
179189
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
180190

presto-hive/src/main/java/com/facebook/presto/hive/HiveSessionProperties.java

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -139,6 +139,8 @@ public final class HiveSessionProperties
139139

140140
public static final String NATIVE_STATS_BASED_FILTER_REORDER_DISABLED = "native_stats_based_filter_reorder_disabled";
141141

142+
public static final String NATIVE_MAX_TARGET_FILE_SIZE = "native_max_target_file_size";
143+
142144
private final List<PropertyMetadata<?>> sessionProperties;
143145

144146
@Inject
@@ -676,7 +678,12 @@ public HiveSessionProperties(HiveClientConfig hiveClientConfig, OrcFileWriterCon
676678
NATIVE_STATS_BASED_FILTER_REORDER_DISABLED,
677679
"Native Execution only. Disable stats based filter reordering.",
678680
false,
679-
true));
681+
true),
682+
dataSizeSessionProperty(
683+
NATIVE_MAX_TARGET_FILE_SIZE,
684+
"Native Execution only. Maximum target file size. When a file exceeds this size during writing, the writer will close the current file and start writing to a new file. Zero means no limit.",
685+
new DataSize(0, DataSize.Unit.BYTE),
686+
false));
680687
}
681688

682689
public List<PropertyMetadata<?>> getSessionProperties()

0 commit comments

Comments
 (0)