Skip to content

Commit 23377d3

Browse files
ppaluchayuval-cloudinary
authored andcommitted
docs: add section for improving performance for s3
1 parent d72e0f0 commit 23377d3

File tree

1 file changed

+32
-0
lines changed

1 file changed

+32
-0
lines changed

docs/content/s3.md

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -401,6 +401,38 @@ there for more details.
401401

402402
Setting this flag increases the chance for undetected upload failures.
403403

404+
### Increasing performance
405+
406+
#### Using server-side copy
407+
408+
If you are copying objects between S3 buckets in the same region, you should
409+
use server-side copy.
410+
This is much faster than downloading and re-uploading the objects, as no data is transferred.
411+
412+
For rclone to use server-side copy, you must use the same remote for the source and destination.
413+
414+
rclone copy s3:source-bucket s3:destination-bucket
415+
416+
When using server-side copy, the performance is limited by the rate at which rclone issues
417+
API requests to S3.
418+
See below for how to increase the number of API requests rclone makes.
419+
420+
#### Increasing the rate of API requests
421+
422+
You can increase the rate of API requests to S3 by increasing the parallelism using `--transfers` and `--checkers`
423+
options.
424+
425+
Rclone uses a very conservative defaults for these settings, as not all providers support high rates of requests.
426+
Depending on your provider, you can increase significantly the number of transfers and checkers.
427+
428+
For example, with AWS S3, if you can increase the number of checkers to values like 200.
429+
If you are doing a server-side copy, you can also increase the number of transfers to 200.
430+
431+
rclone sync --transfers 200 --checkers 200 --checksum s3:source-bucket s3:destination-bucket
432+
433+
You will need to experiment with these values to find the optimal settings for your setup.
434+
435+
404436
### Versions
405437

406438
When bucket versioning is enabled (this can be done with rclone with

0 commit comments

Comments
 (0)