|
| 1 | +--- |
| 2 | +title: "Chain Table" |
| 3 | +weight: 6 |
| 4 | +type: docs |
| 5 | +aliases: |
| 6 | +- /primary-key-table/chain-table.html |
| 7 | +--- |
| 8 | +<!-- |
| 9 | +Licensed to the Apache Software Foundation (ASF) under one |
| 10 | +or more contributor license agreements. See the NOTICE file |
| 11 | +distributed with this work for additional information |
| 12 | +regarding copyright ownership. The ASF licenses this file |
| 13 | +to you under the Apache License, Version 2.0 (the |
| 14 | +"License"); you may not use this file except in compliance |
| 15 | +with the License. You may obtain a copy of the License at |
| 16 | +
|
| 17 | + http://www.apache.org/licenses/LICENSE-2.0 |
| 18 | +
|
| 19 | +Unless required by applicable law or agreed to in writing, |
| 20 | +software distributed under the License is distributed on an |
| 21 | +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY |
| 22 | +KIND, either express or implied. See the License for the |
| 23 | +specific language governing permissions and limitations |
| 24 | +under the License. |
| 25 | +--> |
| 26 | + |
| 27 | +# Chain Table |
| 28 | + |
| 29 | +Chain table is a new capability for primary key tables that transforms how you process incremental data. |
| 30 | +Imagine a scenario where you periodically store a full snapshot of data (for example, once a day), even |
| 31 | +though only a small portion changes between snapshots. ODS binlog dump is a typical example of this pattern. |
| 32 | + |
| 33 | +Taking a daily binlog dump job as an example. A batch job merges yesterday’s full dataset with today’s |
| 34 | +incremental changes to produce a new full dataset. This approach has two clear drawbacks: |
| 35 | +* Full computation: Merge operation includes all data, and it will involve shuffle, which results in poor performance. |
| 36 | +* Full storage: Store a full set of data every day, and the changed data usually accounts for a very small proportion. |
| 37 | + |
| 38 | +Paimon addresses this problem by directly consuming only the changed data and performing merge-on-read. |
| 39 | +In this way, full computation and storage are turned into incremental mode: |
| 40 | +* Incremental computation: The offline ETL daily job only needs to consume the changed data of the current day and do not require merging all data. |
| 41 | +* Incremental Storage: Only store the changed data each day, and asynchronously compact it periodically (e.g., weekly) to build a global chain table within the lifecycle. |
| 42 | + {{< img src="/img/chain-table.png">}} |
| 43 | + |
| 44 | +Based on the regular table, chain table introduces snapshot and delta branches to represent full and incremental |
| 45 | +data respectively. When writing, you specify the branch to write full or incremental data. When reading, paimon |
| 46 | +automatically chooses the appropriate strategy based on the read mode, such as full, incremental, or hybrid. |
| 47 | + |
| 48 | +To enable chain table, you must config `chain-table.enabled` to true in the table options when creating the |
| 49 | +table, and the snapshot and delta branch need to be created as well. Consider an example via Spark SQL: |
| 50 | + |
| 51 | +```sql |
| 52 | +CREATE TABLE default.t ( |
| 53 | + `t1` string , |
| 54 | + `t2` string , |
| 55 | + `t3` string |
| 56 | +) PARTITIONED BY (`date` string) |
| 57 | +TBLPROPERTIES ( |
| 58 | + 'chain-table.enabled' = 'true', |
| 59 | + -- props about primary key table |
| 60 | + 'primary-key' = 'date,t1', |
| 61 | + 'sequence.field' = 't2', |
| 62 | + 'bucket-key' = 't1', |
| 63 | + 'bucket' = '2', |
| 64 | + -- props about partition |
| 65 | + 'partition.timestamp-pattern' = '$date', |
| 66 | + 'partition.timestamp-formatter' = 'yyyyMMdd' |
| 67 | +); |
| 68 | + |
| 69 | +CALL sys.create_branch('default.t', 'snapshot'); |
| 70 | + |
| 71 | +CALL sys.create_branch('default.t', 'delta'); |
| 72 | + |
| 73 | +ALTER TABLE default.t SET tblproperties |
| 74 | + ('scan.fallback-snapshot-branch' = 'snapshot', |
| 75 | + 'scan.fallback-delta-branch' = 'delta'); |
| 76 | + |
| 77 | +ALTER TABLE `default`.`t$branch_snapshot` SET tblproperties |
| 78 | + ('scan.fallback-snapshot-branch' = 'snapshot', |
| 79 | + 'scan.fallback-delta-branch' = 'delta'); |
| 80 | + |
| 81 | +ALTER TABLE `default`.`t$branch_delta` SET tblproperties |
| 82 | + ('scan.fallback-snapshot-branch' = 'snapshot', |
| 83 | + 'scan.fallback-delta-branch' = 'delta'); |
| 84 | +``` |
| 85 | + |
| 86 | +Notice that: |
| 87 | +- Chain table is only supported for primary key table, which means you should define `bucket` and `bucket-key` for the table. |
| 88 | +- Chain table should ensure that the schema of each branch is consistent. |
| 89 | +- Only spark support now, flink will be supported later. |
| 90 | +- Chain compact is not supported for now, and it will be supported later. |
| 91 | + |
| 92 | +After creating a chain table, you can read and write data in the following ways. |
| 93 | + |
| 94 | +- Full Write: Write data to t$branch_snapshot. |
| 95 | +```sql |
| 96 | +insert overwrite `default`.`t$branch_snapshot` partition (date = '20250810') |
| 97 | + values ('1', '1', '1'); |
| 98 | +``` |
| 99 | + |
| 100 | +- Incremental Write: Write data to t$branch_delta. |
| 101 | +```sql |
| 102 | +insert overwrite `default`.`t$branch_delta` partition (date = '20250811') |
| 103 | + values ('2', '1', '1'); |
| 104 | +``` |
| 105 | + |
| 106 | +- Full Query: If the snapshot branch has full partition, read it directly; otherwise, read on chain merge mode. |
| 107 | +```sql |
| 108 | +select t1, t2, t3 from default.t where date = '20250811' |
| 109 | +``` |
| 110 | +you will get the following result: |
| 111 | +```text |
| 112 | ++---+----+-----+ |
| 113 | +| t1| t2| t3| |
| 114 | ++---+----+-----+ |
| 115 | +| 1 | 1| 1 | |
| 116 | +| 2 | 1| 1 | |
| 117 | ++---+----+-----+ |
| 118 | +``` |
| 119 | + |
| 120 | +- Incremental Query: Read the incremental partition from t$branch_delta |
| 121 | +```sql |
| 122 | +select t1, t2, t3 from `default`.`t$branch_delta` where date = '20250811' |
| 123 | +``` |
| 124 | +you will get the following result: |
| 125 | +```text |
| 126 | ++---+----+-----+ |
| 127 | +| t1| t2| t3| |
| 128 | ++---+----+-----+ |
| 129 | +| 2 | 1| 1 | |
| 130 | ++---+----+-----+ |
| 131 | +``` |
| 132 | + |
| 133 | +- Hybrid Query: Read both full and incremental data simultaneously. |
| 134 | +```sql |
| 135 | +select t1, t2, t3 from default.t where date = '20250811' |
| 136 | +union all |
| 137 | +select t1, t2, t3 from `default`.`t$branch_delta` where date = '20250811' |
| 138 | +``` |
| 139 | +you will get the following result: |
| 140 | +```text |
| 141 | ++---+----+-----+ |
| 142 | +| t1| t2| t3| |
| 143 | ++---+----+-----+ |
| 144 | +| 1 | 1| 1 | |
| 145 | +| 2 | 1| 1 | |
| 146 | +| 2 | 1| 1 | |
| 147 | ++---+----+-----+ |
| 148 | +``` |
0 commit comments