Skip to content

Commit 8023fe7

Browse files
authored
fix: temporarily increase S3 chunk size for rclone uploads
When using multipart upload, we're limited to 10k chunks. So with 150MB chunks, we were limited to 1.5TB data dirs. Signed-off-by: Rafael Matias <rafael@skyle.net>
1 parent d91e07f commit 8023fe7

File tree

1 file changed

+1
-1
lines changed

1 file changed

+1
-1
lines changed

internal/config/config.go

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -96,7 +96,7 @@ tar -I zstd \\
9696
--exclude=./key \\
9797
--exclude=./discovery-secret \\
9898
-cvf - . \\
99-
| rclone rcat --s3-chunk-size 150M mys3:/{{ .BucketName }}/{{ .UploadPathPrefix }}/{{ .BlockNumber }}/snapshot.tar.zst &&
99+
| rclone rcat --s3-chunk-size 300M mys3:/{{ .BucketName }}/{{ .UploadPathPrefix }}/{{ .BlockNumber }}/snapshot.tar.zst &&
100100
rclone copy {{ .DataDir }}/_snapshot_eth_getBlockByNumber.json mys3:/{{ .BucketName }}/{{ .UploadPathPrefix }}/{{ .BlockNumber }} &&
101101
rclone copy {{ .DataDir }}/_snapshot_web3_clientVersion.json mys3:/{{ .BucketName }}/{{ .UploadPathPrefix }}/{{ .BlockNumber }} &&
102102
rclone copy {{ .DataDir }}/_snapshot_metadata.json mys3:/{{ .BucketName }}/{{ .UploadPathPrefix }}/{{ .BlockNumber }} &&

0 commit comments

Comments
 (0)