s3util: S3 command line utility documentation

Introduction

Currently this page lists all my thoughts about s3util I collected locally. As the tool matures this will look more like actual documentation.

Usage Notes

Download will not overwrite local files, but create additional copies right now – maybe that should be changed?

Upload will overwrite S3 keys if these are not unique. It is recommended that separate scripts doing uploads either are written to generate unique filenames, use a configured prefix, or upload to separate buckets.

s3util is primarily intended as an S3 backend script for scripts or using the command-line. It is supposed to be as user-friendly as your average fileutil.

If you create sum files make sure they’re stored somewhere safe or also uploaded to S3.

Sample session

List all buckets:

$ ./s3util --accessKeyID ACCESSKEY --list

Create a bucket:

$ ./s3util --accessKeyID ACCESSKEY --create --bucket testbucket

Upload a file:

$ cat >testfile
this is a testfile
$ ./s3util --accessKeyID ACCESSKEY --bucket testbucket --upload testfile

List the contents of a bucket:

$ ./s3util --accessKeyID ACCESSKEY --bucket testbucket --list
...
2007-07-01 18:51:05.431 s3util[3324]    Key: testfile
2007-07-01 18:51:05.431 s3util[3324]    Size: 19 bytes
2007-07-01 18:51:05.431 s3util[3324] {
    etag = "\"afcd2f4903521cec2e268743ace2de05\"";
    key = testfile;
    lastModified = 2007-07-01T16:49:27.000Z +0000;
    owner = <S3Owner: 0xXXXXXX>;
    size = 19;
    storageClass = STANDARD;
}
2007-07-01 18:51:05.431 s3util[3324] Objects array has 1 entries
2007-07-01 18:51:05.432 s3util[3324] Operations queue empty
2007-07-01 18:51:05.432 s3util[3324] Operation completed successfully.
2007-07-01 18:51:05.432 s3util[3324] Return: 0

Download a file:

$ ./s3util --accessKeyID ACCESSKEY --bucket testbucket --download testfile
$ cat testfile
this is a testfile

Delete a file:

$ ./s3util --accessKeyID ACCESSKEY --bucket testbucket --delete testfile

Delete a bucket:

$ ./s3util --accessKeyID ACCESSKEY --delete --bucket testbucket

non-functional UI changes

• download and upload should display progress more clearly
• debug messages should be removed and output should be reduced to a minimum, though a debug switch should be provided.

future functional enhancements

• download should check the integrity of downloaded files with the md5sums stored on amazon.
• add prefix function that will be prepended to keys before they are sent to S3 so you can separate namespaces
• Upload function should be enhanced to RETRY failed transfers instead of returning failure. – Failure should only be returned after a certain number of retries. → If just one of a few files fails, the basic operation should go through if any retry worked to save correctly