feat: refer to aws s3 implimation, use list pagination

Signed-off-by: LinPr <[email protected]>

Author: LinPr

cmd/cp/examples.go

@@ -0,0 +1,227 @@
+package cp
+
+const cp_examples = `Example 1: Copying a local file to S3
+
+       The following cp command copies a single file to a specified bucket and
+       key:
+
+          aws s3 cp test.txt s3://amzn-s3-demo-bucket/test2.txt
+
+       Output:
+
+          upload: test.txt to s3://amzn-s3-demo-bucket/test2.txt
+
+       Example 2: Copying a local file to S3 with an expiration date
+
+       The following cp command copies a single file to a specified bucket and
+       key that expires at the specified ISO 8601 timestamp:
+
+          aws s3 cp test.txt s3://amzn-s3-demo-bucket/test2.txt \
+              --expires 2014-10-01T20:30:00Z
+
+       Output:
+
+          upload: test.txt to s3://amzn-s3-demo-bucket/test2.txt
+
+       Example 3: Copying a file from S3 to S3
+
+       The following cp command copies a  single  s3  object  to  a  specified
+       bucket and key:
+
+          aws s3 cp s3://amzn-s3-demo-bucket/test.txt s3://amzn-s3-demo-bucket/test2.txt
+
+       Output:
+
+          copy: s3://amzn-s3-demo-bucket/test.txt to s3://amzn-s3-demo-bucket/test2.txt
+
+       Example 4: Copying an S3 object to a local file
+
+       The following cp command copies a single object to a specified file lo-
+       cally:
+
+          aws s3 cp s3://amzn-s3-demo-bucket/test.txt test2.txt
+
+       Output:
+
+          download: s3://amzn-s3-demo-bucket/test.txt to test2.txt
+
+       Example 5: Copying an S3 object from one bucket to another
+
+       The following cp command copies a single object to a  specified  bucket
+       while retaining its original name:
+
+          aws s3 cp s3://amzn-s3-demo-bucket/test.txt s3://amzn-s3-demo-bucket2/
+
+       Output:
+
+          copy: s3://amzn-s3-demo-bucket/test.txt to s3://amzn-s3-demo-bucket2/test.txt
+
+       Example 6: Recursively copying S3 objects to a local directory
+
+       When  passed  with  the parameter --recursive, the following cp command
+       recursively copies all objects under a specified prefix and bucket to a
+       specified  directory.   In this example, the bucket amzn-s3-demo-bucket
+       has the objects test1.txt and test2.txt:
+
+          aws s3 cp s3://amzn-s3-demo-bucket . \
+              --recursive
+
+       Output:
+
+          download: s3://amzn-s3-demo-bucket/test1.txt to test1.txt
+          download: s3://amzn-s3-demo-bucket/test2.txt to test2.txt
+
+       Example 7: Recursively copying local files to S3
+
+       When passed with the parameter --recursive, the  following  cp  command
+       recursively copies all files under a specified directory to a specified
+       bucket and prefix while excluding some files by using an --exclude  pa-
+       rameter.   In this example, the directory myDir has the files test1.txt
+       and test2.jpg:
+
+          aws s3 cp myDir s3://amzn-s3-demo-bucket/ \
+              --recursive \
+              --exclude "*.jpg"
+
+       Output:
+
+          upload: myDir/test1.txt to s3://amzn-s3-demo-bucket/test1.txt
+
+       Example 8: Recursively copying S3 objects to another bucket
+
+       When passed with the parameter --recursive, the  following  cp  command
+       recursively  copies  all  objects  under  a specified bucket to another
+       bucket while excluding some objects by using  an  --exclude  parameter.
+       In  this  example,  the  bucket  amzn-s3-demo-bucket  has  the  objects
+       test1.txt and another/test1.txt:
+
+          aws s3 cp s3://amzn-s3-demo-bucket/ s3://amzn-s3-demo-bucket2/ \
+              --recursive \
+              --exclude "another/*"
+
+       Output:
+
+          copy: s3://amzn-s3-demo-bucket/test1.txt to s3://amzn-s3-demo-bucket2/test1.txt
+
+       You can combine --exclude and --include options to  copy  only  objects
+       that match a pattern, excluding all others:
+
+          aws s3 cp s3://amzn-s3-demo-bucket/logs/ s3://amzn-s3-demo-bucket2/logs/ \
+              --recursive \
+              --exclude "*" \
+              --include "*.log"
+
+       Output:
+
+          copy: s3://amzn-s3-demo-bucket/logs/test/test.log to s3://amzn-s3-demo-bucket2/logs/test/test.log
+          copy: s3://amzn-s3-demo-bucket/logs/test3.log to s3://amzn-s3-demo-bucket2/logs/test3.log
+
+       Example  9:  Setting  the Access Control List (ACL) while copying an S3
+       object
+
+       The following cp command copies a single object to a  specified  bucket
+       and key while setting the ACL to public-read-write:
+
+          aws s3 cp s3://amzn-s3-demo-bucket/test.txt s3://amzn-s3-demo-bucket/test2.txt \
+              --acl public-read-write
+
+       Output:
+
+          copy: s3://amzn-s3-demo-bucket/test.txt to s3://amzn-s3-demo-bucket/test2.txt
+
+       Note  that if you're using the --acl option, ensure that any associated
+       IAM policies include the "s3:PutObjectAcl" action:
+
+          aws iam get-user-policy \
+              --user-name myuser \
+              --policy-name mypolicy
+
+       Output:
+
+          {
+              "UserName": "myuser",
+              "PolicyName": "mypolicy",
+              "PolicyDocument": {
+                  "Version": "2012-10-17",
+                  "Statement": [
+                      {
+                          "Action": [
+                              "s3:PutObject",
+                              "s3:PutObjectAcl"
+                          ],
+                          "Resource": [
+                              "arn:aws:s3:::amzn-s3-demo-bucket/*"
+                          ],
+                          "Effect": "Allow",
+                          "Sid": "Stmt1234567891234"
+                      }
+                  ]
+              }
+          }
+
+       Example 10: Granting permissions for an S3 object
+
+       The following cp command illustrates the use of the --grants option  to
+       grant  read access to all users identified by URI and full control to a
+       specific user identified by their Canonical ID:
+
+          aws s3 cp file.txt s3://amzn-s3-demo-bucket/ --grants read=uri=http://acs.amazonaws.com/groups/global/AllUsers full=id=79a59df900b949e55d96a1e698fbacedfd6e09d98eacf8f8d5218e7cd47ef2be
+
+       Output:
+
+          upload: file.txt to s3://amzn-s3-demo-bucket/file.txt
+
+       Example 11: Uploading a local file stream to S3
+
+       WARNING:
+          PowerShell may alter the encoding of or add a CRLF to piped input.
+
+       The following cp command uploads a local file stream from standard  in-
+       put to a specified bucket and key:
+
+          aws s3 cp - s3://amzn-s3-demo-bucket/stream.txt
+
+       Example  12:  Uploading a local file stream that is larger than 50GB to
+       S3
+
+       The following cp command uploads a 51GB local file stream from standard
+       input  to  a specified bucket and key.  The --expected-size option must
+       be provided, or the upload may fail when it reaches  the  default  part
+       limit of 10,000:
+
+          aws s3 cp - s3://amzn-s3-demo-bucket/stream.txt --expected-size 54760833024
+
+       Example 13: Downloading an S3 object as a local file stream
+
+       WARNING:
+          PowerShell may alter the encoding of or add a CRLF to piped or redi-
+          rected output.
+
+       The following cp command downloads an S3 object locally as a stream  to
+       standard  output.  Downloading  as a stream is not currently compatible
+       with the --recursive parameter:
+
+          aws s3 cp s3://amzn-s3-demo-bucket/stream.txt -
+
+       Example 14: Uploading to an S3 access point
+
+       The following cp command uploads a single file (mydoc.txt) to  the  ac-
+       cess point (myaccesspoint) at the key (mykey):
+
+          aws s3 cp mydoc.txt s3://arn:aws:s3:us-west-2:123456789012:accesspoint/myaccesspoint/mykey
+
+       Output:
+
+          upload: mydoc.txt to s3://arn:aws:s3:us-west-2:123456789012:accesspoint/myaccesspoint/mykey
+
+       Example 15: Downloading from an S3 access point
+
+       The following cp command downloads a single object (mykey) from the ac-
+       cess point (myaccesspoint) to the local file (mydoc.txt):
+
+          aws s3 cp s3://arn:aws:s3:us-west-2:123456789012:accesspoint/myaccesspoint/mykey mydoc.txt
+
+       Output:
+
+          download: s3://arn:aws:s3:us-west-2:123456789012:accesspoint/myaccesspoint/mykey to mydoc.txt
+`

cmd/du/du.go

@@ -46,7 +46,8 @@ type Args struct {
 	S3Uri string `validate:"omitempty"`
 }
 type Flags struct {
-	DryRun bool `json:"DryRun" yaml:"DryRun"`
+	DryRun bool    `json:"DryRun" yaml:"DryRun"`
+	Region *string `json:"Region" yaml:"Region"`
 }
 
 type Options struct {
@@ -76,12 +77,16 @@ func (o *Options) run() error {
 	fmt.Fprintf(os.Stdout, "options: %s\n", string(j))
 	// return nil
 
-	cli, err := s3store.NewS3Client(context.TODO())
+	opt := s3store.S3Option{
+		Region: *o.Region,
+	}
+
+	cli, err := s3store.NewS3Client(context.TODO(), opt)
 	if err != nil {
 		return err
 	}
 
-	parsedUri, err := uri.ParseS3Url(o.S3Uri)
+	parsedUri, err := uri.ParseS3Uri(o.S3Uri)
 	if err != nil {
 		return err
 	}
@@ -93,12 +98,13 @@ func (o *Options) run() error {
 }
 
 func listObjects(cli *s3store.S3Store, bucket, key string) error {
-	objs, err := cli.ListObjects(context.TODO(), bucket, key)
+	objs, err := cli.ListObjectsWithPagination(context.TODO(), bucket, key)
 	if err != nil {
 		return err
 	}
 	var size int64
-	for _, obj := range objs.Contents {
+
+	for _, obj := range objs {
 		size += *obj.Size
 	}
 	unit := "bytes"

cmd/get/get.go

@@ -78,12 +78,12 @@ func (o *Options) run() error {
 	fmt.Fprintf(os.Stdout, "options: %s\n", string(j))
 	// return nil
 
-	parsedUri, err := uri.ParseS3Url(o.S3Uri)
+	parsedUri, err := uri.ParseS3Uri(o.S3Uri)
 	if err != nil {
 		return err
 	}
-
-	store, err := storage.NewStorage(context.TODO())
+	opt := storage.StorageOption{}
+	store, err := storage.NewStorage(context.TODO(), opt)
 	if err != nil {
 		return err
 	}

cmd/ls/examples.go

@@ -0,0 +1,110 @@
+package ls
+
+const ls_examples = `Example 1: Listing all user owned buckets
+
+       The following ls command lists all of the bucket owned by the user.  In
+       this  example,  the  user  owns  the  buckets  amzn-s3-demo-bucket  and
+       amzn-s3-demo-bucket2.   The  timestamp  is the date the bucket was cre-
+       ated, shown in your machine's time zone.  This  date  can  change  when
+       making changes to your bucket, such as editing its bucket policy.  Note
+       if  s3:// is used for the path argument <S3Uri>, it will  list  all  of
+       the buckets as well.
+
+          aws s3 ls
+
+       Output:
+
+          2013-07-11 17:08:50 amzn-s3-demo-bucket
+          2013-07-24 14:55:44 amzn-s3-demo-bucket2
+
+       Example 2: Listing all prefixes and objects in a bucket
+
+       The  following  ls  command  lists  objects and common prefixes under a
+       specified bucket and prefix.  In this example, the user owns the bucket
+       amzn-s3-demo-bucket  with the objects test.txt and somePrefix/test.txt.
+       The LastWriteTime and Length are arbitrary. Note that since the ls com-
+       mand has no interaction with the local filesystem, the s3:// URI scheme
+       is not required to resolve ambiguity and may be omitted.
+
+          aws s3 ls s3://amzn-s3-demo-bucket
+
+       Output:
+
+                                     PRE somePrefix/
+          2013-07-25 17:06:27         88 test.txt
+
+       Example 3: Listing all prefixes and objects in a  specific  bucket  and
+       prefix
+
+       The  following  ls  command  lists  objects and common prefixes under a
+       specified bucket and prefix.  However, there are no objects nor  common
+       prefixes under the specified bucket and prefix.
+
+          aws s3 ls s3://amzn-s3-demo-bucket/noExistPrefix
+
+       Output:
+
+          None
+
+       Example 4: Recursively listing all prefixes and objects in a bucket
+
+       The  following  ls  command  will recursively list objects in a bucket.
+       Rather than showing PRE dirname/ in the output, all the  content  in  a
+       bucket will be listed in order.
+
+          aws s3 ls s3://amzn-s3-demo-bucket \
+              --recursive
+
+       Output:
+
+          2013-09-02 21:37:53         10 a.txt
+          2013-09-02 21:37:53    2863288 foo.zip
+          2013-09-02 21:32:57         23 foo/bar/.baz/a
+          2013-09-02 21:32:58         41 foo/bar/.baz/b
+          2013-09-02 21:32:57        281 foo/bar/.baz/c
+          2013-09-02 21:32:57         73 foo/bar/.baz/d
+          2013-09-02 21:32:57        452 foo/bar/.baz/e
+          2013-09-02 21:32:57        896 foo/bar/.baz/hooks/bar
+          2013-09-02 21:32:57        189 foo/bar/.baz/hooks/foo
+          2013-09-02 21:32:57        398 z.txt
+
+       Example 5: Summarizing all prefixes and objects in a bucket
+
+       The  following ls command demonstrates the same command using the --hu-
+       man-readable and --summarize options.  --human-readable  displays  file
+       size  in  Bytes/MiB/KiB/GiB/TiB/PiB/EiB. --summarize displays the total
+       number of objects and total size at the end of the result listing:
+
+          aws s3 ls s3://amzn-s3-demo-bucket \
+              --recursive \
+              --human-readable \
+              --summarize
+
+       Output:
+
+          2013-09-02 21:37:53   10 Bytes a.txt
+          2013-09-02 21:37:53  2.9 MiB foo.zip
+          2013-09-02 21:32:57   23 Bytes foo/bar/.baz/a
+          2013-09-02 21:32:58   41 Bytes foo/bar/.baz/b
+          2013-09-02 21:32:57  281 Bytes foo/bar/.baz/c
+          2013-09-02 21:32:57   73 Bytes foo/bar/.baz/d
+          2013-09-02 21:32:57  452 Bytes foo/bar/.baz/e
+          2013-09-02 21:32:57  896 Bytes foo/bar/.baz/hooks/bar
+          2013-09-02 21:32:57  189 Bytes foo/bar/.baz/hooks/foo
+          2013-09-02 21:32:57  398 Bytes z.txt
+
+          Total Objects: 10
+             Total Size: 2.9 MiB
+
+       Example 6: Listing from an S3 access point
+
+       The following ls command list  objects  from  access  point  (myaccess-
+       point):
+
+          aws s3 ls s3://arn:aws:s3:us-west-2:123456789012:accesspoint/myaccesspoint/
+
+       Output:
+
+                                     PRE somePrefix/
+          2013-07-25 17:06:27         88 test.txt
+`