Skip to content

Commit 7c7bcd9

Browse files
authored
[doc] Add manage partition introduction page (#5357)
1 parent b0141c2 commit 7c7bcd9

File tree

2 files changed

+34
-4
lines changed

2 files changed

+34
-4
lines changed

docs/content/flink/sql-ddl.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -187,7 +187,7 @@ If you need cross partition upsert (primary keys not contain all partition field
187187
{{< /hint >}}
188188

189189
{{< hint info >}}
190-
By configuring [partition.expiration-time]({{< ref "flink/expire-partition" >}}), expired partitions can be automatically deleted.
190+
By configuring [partition.expiration-time]({{< ref "maintenance/manage-partitions" >}}), expired partitions can be automatically deleted.
191191
{{< /hint >}}
192192

193193
### Specify Statistics Mode

docs/content/flink/expire-partition.md renamed to docs/content/maintenance/manage-partitions.md

Lines changed: 33 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,9 @@
11
---
2-
title: "Expire Partition"
3-
weight: 96
2+
title: "Manage Partitions"
3+
weight: 12
44
type: docs
55
aliases:
6-
- /flink/expire-partition.html
6+
- /maintenance/manage-partitions.html
77
---
88
<!--
99
Licensed to the Apache Software Foundation (ASF) under one
@@ -24,6 +24,10 @@ specific language governing permissions and limitations
2424
under the License.
2525
-->
2626

27+
# Manage Partitions
28+
Paimon provides multiple ways to manage partitions, including expire historical partitions by different strategies or
29+
mark a partition done to notify the downstream application that the partition has finished writing.
30+
2731
## Expiring Partitions
2832

2933
You can set `partition.expiration-time` when creating a partitioned table. Paimon streaming sink will periodically check
@@ -138,3 +142,29 @@ More options:
138142
</tr>
139143
</tbody>
140144
</table>
145+
146+
## Partition Mark Done
147+
148+
You can use the option `'partition.mark-done-action'` to configure the action when a partition needs to be mark done.
149+
- `success-file`: add '_success' file to directory.
150+
- `done-partition`: add 'xxx.done' partition to metastore.
151+
- `mark-event`: mark partition event to metastore.
152+
- `http-report`: report partition mark done to remote http server.
153+
- `custom`: use policy class to create a mark-partition policy.
154+
These actions can be configured at the same time: 'done-partition,success-file,mark-event,custom'.
155+
156+
Paimon partition mark done can be triggered both by streaming write and batch write.
157+
158+
### Streaming Mark Done
159+
160+
You can use the options `'partition.idle-time-to-done'` to set a partition idle time to done duration. When a partition
161+
has no new data after this time duration, the mark done action will be triggered to indicate that the data is ready.
162+
163+
By default, Flink will use process time as idle time to trigger partition mark done. You can also use watermark to
164+
trigger partition mark done. This will make the partition mark done time more accurate when data is delayed. You can
165+
enable this by setting `'partition.mark-done-action.mode' = 'watermark'`.
166+
167+
### Batch Mark Done
168+
169+
For batch mode, you can trigger partition mark done when end input by setting `'partition.end-input-to-done'='true'`.
170+

0 commit comments

Comments
 (0)