11---
2- title : " Expire Partition "
3- weight : 96
2+ title : " Manage Partitions "
3+ weight : 12
44type : docs
55aliases :
6- - /flink/expire-partition .html
6+ - /maintenance/manage-partitions .html
77---
88<!--
99Licensed to the Apache Software Foundation (ASF) under one
@@ -24,6 +24,10 @@ specific language governing permissions and limitations
2424under the License.
2525-->
2626
27+ # Manage Partitions
28+ Paimon provides multiple ways to manage partitions, including expire historical partitions by different strategies or
29+ mark a partition done to notify the downstream application that the partition has finished writing.
30+
2731## Expiring Partitions
2832
2933You can set ` partition.expiration-time ` when creating a partitioned table. Paimon streaming sink will periodically check
@@ -138,3 +142,29 @@ More options:
138142 </tr>
139143 </tbody>
140144</table >
145+
146+ ## Partition Mark Done
147+
148+ You can use the option ` 'partition.mark-done-action' ` to configure the action when a partition needs to be mark done.
149+ - ` success-file ` : add '_ success' file to directory.
150+ - ` done-partition ` : add 'xxx.done' partition to metastore.
151+ - ` mark-event ` : mark partition event to metastore.
152+ - ` http-report ` : report partition mark done to remote http server.
153+ - ` custom ` : use policy class to create a mark-partition policy.
154+ These actions can be configured at the same time: 'done-partition,success-file,mark-event,custom'.
155+
156+ Paimon partition mark done can be triggered both by streaming write and batch write.
157+
158+ ### Streaming Mark Done
159+
160+ You can use the options ` 'partition.idle-time-to-done' ` to set a partition idle time to done duration. When a partition
161+ has no new data after this time duration, the mark done action will be triggered to indicate that the data is ready.
162+
163+ By default, Flink will use process time as idle time to trigger partition mark done. You can also use watermark to
164+ trigger partition mark done. This will make the partition mark done time more accurate when data is delayed. You can
165+ enable this by setting ` 'partition.mark-done-action.mode' = 'watermark' ` .
166+
167+ ### Batch Mark Done
168+
169+ For batch mode, you can trigger partition mark done when end input by setting ` 'partition.end-input-to-done'='true' ` .
170+
0 commit comments