|
| 1 | +--- |
| 2 | +title: Disaster Recovery |
| 3 | +description: Enable Databend to recover from disasters involving the loss of either metadata or data. |
| 4 | +--- |
| 5 | + |
| 6 | +- RFC PR: [databendlabs/databend-docs#1546](https://github.com/databendlabs/databend-docs/pull/1546) |
| 7 | +- Tracking Issue: [datafuselabs/databend#17234](https://github.com/databendlabs/databend/issues/17234) |
| 8 | + |
| 9 | +## Summary |
| 10 | + |
| 11 | +Enable databend to recover from disasters involving the loss of either metadata or data. |
| 12 | + |
| 13 | +## Motivation |
| 14 | + |
| 15 | +Databend is designed to be highly available and fault-tolerant. Its metadata is served by Databend MetaSrv, which is powered by [OpenRaft](https://github.com/databendlabs/openraft). The data is stored in object storage systems such as S3, GCS, and others, which guarantee 99.99% availability and 99.999999999% durability. |
| 16 | + |
| 17 | +However, it is insufficient for our enterprise users who require a robust disaster recovery plan. These users either have significant needs for cross-continent disaster recovery or must comply with stringent regulatory requirements. |
| 18 | + |
| 19 | +For example, [The Health Insurance Portability and Accountability Act (HIPAA)](https://www.hhs.gov/hipaa/index.html) mandates that healthcare organizations develop and implement contingency plans. Such planning ensures that, in the event of a natural or man-made disaster disrupting operations, the business can continue functioning until regular services are restored. |
| 20 | + |
| 21 | +This RFC proposes a solution to enable Databend to recover from disasters involving the loss of metadata or data. |
| 22 | + |
| 23 | +## Guide-Level Explanation |
| 24 | + |
| 25 | +This RFC introduces the first step toward enabling Databend to recover from disasters, such as metadata or data loss, by providing a robust backup and restore solution. Our proposed product, tentatively named `bendsave`, will allow users to back up and restore both metadata and data efficiently. |
| 26 | + |
| 27 | +*The name of this product is not decided yet, let's call it `bendsave`* |
| 28 | + |
| 29 | +### 1. Backup |
| 30 | + |
| 31 | +Create backups of cluster data and metadata using the `bendsave backup` command. Incremental backups are supported, ensuring that only changes since the last backup are saved. This simplifies daily backups. |
| 32 | + |
| 33 | +Example: |
| 34 | + |
| 35 | +```shell |
| 36 | +bendsave backup --from /path/to/query-node-1.toml --to s3://backup/ |
| 37 | +``` |
| 38 | + |
| 39 | +Key Points: |
| 40 | +- Metadata and data are stored in the backup location. |
| 41 | +- Enables complete cluster recovery, even in cases of total failure. |
| 42 | + |
| 43 | +### 2. List Backups |
| 44 | + |
| 45 | +To view all backups stored in a specified location, use the `bendsave list` command. |
| 46 | + |
| 47 | +Example: |
| 48 | + |
| 49 | +```shell |
| 50 | +bendsave list s3://backup/ |
| 51 | +``` |
| 52 | + |
| 53 | +### 3. Restore |
| 54 | + |
| 55 | +Restore a Databend cluster from a backup using the `bendsave restore` command. By default, this operates in dry-run mode to prevent accidental restoration. For automatic restoration, use the `--confirm` flag. |
| 56 | + |
| 57 | +Example: |
| 58 | + |
| 59 | +```shell |
| 60 | +# Dry-run mode (default) |
| 61 | +bendsave restore --from s3://backup/path/to/backup/manifest --to /path/to/query-node-1.toml |
| 62 | + |
| 63 | +# Perform the restoration immediately |
| 64 | +bendsave restore --from s3://backup/path/to/backup/manifest --to /path/to/query-node-1.toml --confirm |
| 65 | +``` |
| 66 | + |
| 67 | +### 4. Vacuum |
| 68 | + |
| 69 | +Manage backup retention using the `bendsave vacuum` command. This ensures backups adhere to your retention policies by removing old or unnecessary backups. |
| 70 | + |
| 71 | +Example: |
| 72 | + |
| 73 | +```shell |
| 74 | +bendsave vacuum s3://backup \ |
| 75 | + --retention-days 30 \ |
| 76 | + --min-retention-days 7 \ |
| 77 | + --max-backups 5 \ |
| 78 | + --min-backups 2 |
| 79 | +``` |
| 80 | + |
| 81 | +The `bendsave` tool will provide a simple yet powerful way to secure Databend clusters through backup and restore operations. With features like incremental backups, dry-run restore mode, and vacuum-based retention management, it offers users control and reliability in disaster recovery scenarios. |
| 82 | + |
| 83 | +## Reference-level explanation |
| 84 | + |
| 85 | +`bendsave` will introduce an `BackupManifest` in which stores the following things: |
| 86 | + |
| 87 | +- metadata of given backup: like backup time, backup location, backup type (full or incremental), etc. |
| 88 | +- the locations of metadata backup: the locations which points to the metadata backup. |
| 89 | +- the locations of data backup: the locations which contains all table data. |
| 90 | + |
| 91 | +```rust |
| 92 | +struct BackupManifest { |
| 93 | + backup_meta: BackupMeta, |
| 94 | + |
| 95 | + metasrv: BackupFile, |
| 96 | + storage: Vec<BackupFile>, |
| 97 | + ... |
| 98 | +} |
| 99 | + |
| 100 | +struct BackupMeta { |
| 101 | + backup_time: DateTime<Utc>, |
| 102 | + ... |
| 103 | +} |
| 104 | + |
| 105 | +struct BackupFile { |
| 106 | + blocks: Vec<Block>, |
| 107 | + etag: String, |
| 108 | +} |
| 109 | + |
| 110 | +struct BackupBlock { |
| 111 | + block_id: String, |
| 112 | + block_size: u64, |
| 113 | + ... |
| 114 | +} |
| 115 | +``` |
| 116 | + |
| 117 | +The `BackupManifest` will be encoded by protobuf and stored inside backup storage along with the backup metadata and data. |
| 118 | + |
| 119 | +The protobuf definition of `BackupManifest` will be versioned to ensure both backward and forward compatibility. This will enable Databend Query to restore backups created using different versions of Databend. |
| 120 | + |
| 121 | +### Backup Storage Layout |
| 122 | + |
| 123 | +The backup storage layout will be as follows: |
| 124 | + |
| 125 | +``` |
| 126 | +s3://backup/bendsave.md |
| 127 | +s3://backup/manifests/20250114_201500.manifest |
| 128 | +s3://backup/manifests/20250115_201500.manifest |
| 129 | +s3://backup/manifests/20250116_201500.manifest |
| 130 | +s3://backup/data/<block_id_0> |
| 131 | +s3://backup/data/<block_id_1> |
| 132 | +s3://backup/data/<block_id_....> |
| 133 | +s3://backup/data/<block_id_N> |
| 134 | +``` |
| 135 | + |
| 136 | +- `bendsave.md` serves as a quick reference guide to help users understand backup storage and recover the cluster. |
| 137 | +- Each manifest in the `manifests/` directory includes everything needed to restore the cluster. |
| 138 | +- The `data/` directory stores all the data blocks. Bendsave splits the source data into fixed-size blocks (e.g., 8 MiB) and uses their SHA-256 checksum as the block ID. |
| 139 | + |
| 140 | +### Backup Process |
| 141 | + |
| 142 | +- Export all metasrv data and save it to the backup storage. |
| 143 | +- Enumerate the source backend storage services to create a `BackupManifest` file. |
| 144 | +- Copy all data files to the backup storage. |
| 145 | + |
| 146 | +For incremental backups, Databend examines the existing `BackupManifest` file and transfers only the modified data files to the backup storage, along with a new `BackupManifest` file. |
| 147 | + |
| 148 | +For example: |
| 149 | + |
| 150 | +The first time users perform a backup like: |
| 151 | + |
| 152 | +```shell |
| 153 | +bendsave backup --from /path/to/query-node-1.toml --to s3://backup/ |
| 154 | +``` |
| 155 | + |
| 156 | +they will see the following files created: |
| 157 | + |
| 158 | +```shell |
| 159 | +s3://backup/bendsave.md |
| 160 | +s3://backup/manifests/20250114_201500.manifest |
| 161 | +s3://backup/data/<sha256_of_block_0> |
| 162 | +s3://backup/data/<sha256_of_block_1> |
| 163 | +s3://backup/data/<sha256_of_block_....> |
| 164 | +s3://backup/data/<sha256_of_block_N> |
| 165 | +``` |
| 166 | + |
| 167 | +The second time users perform a backup, bendsave will generate the following files and omit existing blocks: |
| 168 | + |
| 169 | +```shell |
| 170 | +s3://backup/bendsave.md |
| 171 | +s3://backup/manifests/20250114_201500.manifest |
| 172 | +s3://backup/manifests/20250115_201500.manifest |
| 173 | +s3://backup/data/<sha256_of_block_0> |
| 174 | +s3://backup/data/<sha256_of_block_1> |
| 175 | +s3://backup/data/<sha256_of_block_....> |
| 176 | +s3://backup/data/<sha256_of_block_N> |
| 177 | +s3://backup/data/<sha256_of_block_....> |
| 178 | +s3://backup/data/<sha256_of_block_M> |
| 179 | +``` |
| 180 | + |
| 181 | +The block id is generated by the SHA-256 checksum of the block content. So we can reuse the same block if it has been backed up before. |
| 182 | + |
| 183 | +### Restore Process |
| 184 | + |
| 185 | +- Read the `BackupManifest` file from the backup storage. |
| 186 | +- Copies all related data files to their original location. |
| 187 | +- Read the backed up metasrv data and import into new metasrv cluster. |
| 188 | + |
| 189 | +Please note that the restore process will overwrite the entire MetaSrv cluster. All existing metadata in the backup target MetaSrv cluster will be permanently lost. |
| 190 | + |
| 191 | +Users can restore from a backup using the following command: |
| 192 | + |
| 193 | +```shell |
| 194 | +bendsave restore --from s3://backup/manifests/20250114_201500.manifest --to /path/to/query-node-1.toml |
| 195 | +``` |
| 196 | + |
| 197 | +Users can also restore incrementally by specifying the latest manifest file: |
| 198 | + |
| 199 | +```shell |
| 200 | +bendsave restore --from s3://backup/manifests/20250115_201500.manifest --to /path/to/query-node-1.toml |
| 201 | +``` |
| 202 | + |
| 203 | +## Drawbacks |
| 204 | + |
| 205 | +None. |
| 206 | + |
| 207 | +## Rationale and alternatives |
| 208 | + |
| 209 | +None. |
| 210 | + |
| 211 | +## Prior art |
| 212 | + |
| 213 | +### Databricks Clone |
| 214 | + |
| 215 | +Databricks allows users to perform shadow and deep cloning of a table. |
| 216 | + |
| 217 | +For example: |
| 218 | + |
| 219 | +Use clone for data archiving |
| 220 | + |
| 221 | +```sql |
| 222 | +CREATE OR REPLACE TABLE archive_table CLONE my_prod_table; |
| 223 | +``` |
| 224 | + |
| 225 | +Or use clone for short-term experiments on a production table |
| 226 | + |
| 227 | +```sql |
| 228 | +-- Perform shallow clone |
| 229 | +CREATE OR REPLACE TABLE my_test SHALLOW CLONE my_prod_table; |
| 230 | + |
| 231 | +UPDATE my_test WHERE user_id is null SET invalid=true; |
| 232 | +-- Run a bunch of validations. Once happy: |
| 233 | + |
| 234 | +-- This should leverage the update information in the clone to prune to only |
| 235 | +-- changed files in the clone if possible |
| 236 | +MERGE INTO my_prod_table |
| 237 | +USING my_test |
| 238 | +ON my_test.user_id <=> my_prod_table.user_id |
| 239 | +WHEN MATCHED AND my_test.user_id is null THEN UPDATE *; |
| 240 | + |
| 241 | +DROP TABLE my_test; |
| 242 | +``` |
| 243 | + |
| 244 | +## Unresolved questions |
| 245 | + |
| 246 | +None. |
| 247 | + |
| 248 | +## Future possibilities |
| 249 | + |
| 250 | +### Replication |
| 251 | + |
| 252 | +In the future, we could extend the backup and restore functionality to support replication. This would allow users to replicate databases or tables across different databend clusters for disaster recovery or data distribution purposes. |
| 253 | + |
| 254 | +Databend can also implement a warm standby to ensure high availability and fault tolerance. |
0 commit comments