Skip to content

Commit 9663ebd

Browse files
authored
adding data migration optimization strategy for slurper (#25967)
* adding data migration optimization strategy for slurper * remove empty space
1 parent f0e00b1 commit 9663ebd

File tree

1 file changed

+34
-2
lines changed

1 file changed

+34
-2
lines changed

src/content/docs/r2/data-migration/migration-strategies.mdx

Lines changed: 34 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ sidebar:
77
order: 5
88
---
99

10-
import { Render } from "~/components";
10+
import { Render, FileTree } from "~/components";
1111

1212
You can use a combination of Super Slurper and Sippy to effectively migrate all objects with minimal downtime.
1313

@@ -25,4 +25,36 @@ You can use a combination of Super Slurper and Sippy to effectively migrate all
2525
- Note that Super Slurper may skip some objects if they are uploaded after it lists the objects to be copied.
2626
2. Enable Sippy on your R2 bucket, then start using the R2 bucket in your application.
2727
- New uploads will go to the R2 bucket.
28-
- Objects which were uploaded while Super Slurper was copying the objects will be copied on-demand (by Sippy) when they are requested by the application.
28+
- Objects which were uploaded while Super Slurper was copying the objects will be copied on-demand (by Sippy) when they are requested by the application.
29+
30+
### Optimizing your Slurper data migration performance
31+
32+
For an account, you can run three concurrent Slurper migration jobs at any given time, and each Slurper migration job can process a set amount of requests per second.
33+
34+
To increase overall throughput and reliability, we recommend splitting your migration into smaller, concurrent jobs using the prefix (or bucket subpath) option.
35+
36+
When creating a migration job:
37+
38+
1. Go to the **Source bucket** step.
39+
2. Under **Define rules**, in **Bucket subpath**, specify subpaths to divide your data by prefix.
40+
3. Complete the data migration set up.
41+
42+
For example, suppose your source bucket contains:
43+
44+
<FileTree>
45+
- photos
46+
- 2024
47+
- file1.jpg
48+
- file2.jpg
49+
- 2023
50+
- file3.jpg
51+
- 2019
52+
- file4.jpg
53+
</FileTree>
54+
55+
You can create separate jobs with prefixes such as:
56+
57+
- `/photos/2024` to migrate all 2024 files
58+
- `/photos/202` to migrate all files from 2023 and 2024
59+
60+
Each prefix runs as an independent migration job, allowing Slurper to transfer data in parallel. This improves total transfer speed and ensures that a failure in one job does not interrupt the others.

0 commit comments

Comments
 (0)