Skip to content

adding data migration optimization strategy for slurper #25967

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Oct 21, 2025
+34 −2
Merged

adding data migration optimization strategy for slurper #25967

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
b8425c2
adding data migration optimization strategy for slurper
aninibread Oct 21, 2025
9a8bbbd
remove empty space
aninibread Oct 21, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 34 additions & 2 deletions src/content/docs/r2/data-migration/migration-strategies.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ sidebar:
order: 5
---

import { Render } from "~/components";
import { Render, FileTree } from "~/components";

You can use a combination of Super Slurper and Sippy to effectively migrate all objects with minimal downtime.

Expand All @@ -25,4 +25,36 @@ You can use a combination of Super Slurper and Sippy to effectively migrate all
- Note that Super Slurper may skip some objects if they are uploaded after it lists the objects to be copied.
2. Enable Sippy on your R2 bucket, then start using the R2 bucket in your application.
- New uploads will go to the R2 bucket.
- Objects which were uploaded while Super Slurper was copying the objects will be copied on-demand (by Sippy) when they are requested by the application.
- Objects which were uploaded while Super Slurper was copying the objects will be copied on-demand (by Sippy) when they are requested by the application.

### Optimizing your Slurper data migration performance

For an account, you can run three concurrent Slurper migration jobs at any given time, and each Slurper migration job can process a set amount of requests per second.

To increase overall throughput and reliability, we recommend splitting your migration into smaller, concurrent jobs using the prefix (or bucket subpath) option.

When creating a migration job:

1. Go to the **Source bucket** step.
2. Under **Define rules**, in **Bucket subpath**, specify subpaths to divide your data by prefix.
3. Complete the data migration set up.

For example, suppose your source bucket contains:

<FileTree>
- photos
- 2024
- file1.jpg
- file2.jpg
- 2023
- file3.jpg
- 2019
- file4.jpg
</FileTree>

You can create separate jobs with prefixes such as:

- `/photos/2024` to migrate all 2024 files
- `/photos/202` to migrate all files from 2023 and 2024

Each prefix runs as an independent migration job, allowing Slurper to transfer data in parallel. This improves total transfer speed and ensures that a failure in one job does not interrupt the others.
Loading