aws cp tabula muris data from s3 fail

[zhangjy859]

Hi,

I am trying to download some raw data fom tabula muris s3.

aws s3 ls --no-sign-request s3://czb-tabula-muris/
                           PRE 10x_bam_files/
                           PRE facs_bam_files/
2018-10-08 18:38:40  147226550 TM_droplet_mat.csv.gz
2018-10-08 18:38:45  247481595 TM_droplet_mat.h5ad
2018-10-08 18:38:52  245224669 TM_droplet_mat.rds
2018-10-08 18:39:00    6120187 TM_droplet_metadata.csv
2018-10-08 18:39:00  318146414 TM_facs_mat.csv.gz
2018-10-08 18:39:11  487706533 TM_facs_mat.h5ad
2018-10-08 18:39:11  520953990 TM_facs_mat.rds
2018-10-08 18:39:25    5691123 TM_facs_metadata.csv

But the aws s3 cp command fail:

aws s3 cp --no-sign-request s3://czb-tabula-muris/TM_facs_mat.h5ad ./
fatal error: An error occurred (403) when calling the HeadObject operation: Forbidden

Thank you for your kindly help

[ericmalekos]

I am also having this error. It seems that many of the files have been moved to GLACIER (aka DEEP_ARCHIVE) and are not longer available to the public.

Here are the errors I encounter when trying to access BAM files:

$ aws s3 cp s3://czbiohub-tabula-muris/facs_bam_files/B12-MAA000591-3_8_M-1-1_R1.mus.Aligned.out.sorted.bam . --no-sign-request

warning: Skipping file s3://czbiohub-tabula-muris/facs_bam_files/B12-MAA000591-3_8_M-1-1_R1.mus.Aligned.out.sorted.bam. Object is of storage class GLACIER. Unable to perform download operations on GLACIER objects. You must restore the object to be able to perform the operation. See aws s3 download help for additional parameter options to ignore or force these transfers.

When I check the files I see that most of the bams, but not all, are in DEEP_ARCHIVE

$ aws s3api list-objects-v2 --bucket czbiohub-tabula-muris --prefix facs_bam_files/ --no-sign-request --query 'Contents[].[Key,StorageClass]' --output table
-----------------------------------------------------------------------------------------------
|                                        ListObjectsV2                                        |
+-----------------------------------------------------------------------------+---------------+
|  facs_bam_files/A1-B000126-3_39_F-1-1_R1.mus.Aligned.out.sorted.bam         |  DEEP_ARCHIVE |
|  facs_bam_files/A1-B000126-3_39_F-1-1_R1.mus.Aligned.out.sorted.bam.bai     |  DEEP_ARCHIVE |
|  facs_bam_files/A1-B000126-3_39_F-1-1_R2.mus.Aligned.out.sorted.bam         |  DEEP_ARCHIVE |
|  facs_bam_files/A1-B000126-3_39_F-1-1_R2.mus.Aligned.out.sorted.bam.bai     |  DEEP_ARCHIVE |
|  facs_bam_files/A1-B000127-3_38_F-1-1_R1.mus.Aligned.out.sorted.bam         |  DEEP_ARCHIVE |
|  facs_bam_files/A1-B000127-3_38_F-1-1_R1.mus.Aligned.out.sorted.bam.bai     |  DEEP_ARCHIVE |
|  facs_bam_files/A1-B000127-3_38_F-1-1_R2.mus.Aligned.out.sorted.bam         |  DEEP_ARCHIVE |
|  facs_bam_files/A1-B000127-3_38_F-1-1_R2.mus.Aligned.out.sorted.bam.bai     |  DEEP_ARCHIVE |
|  facs_bam_files/A1-B000167-3_56_F-1-1_R1.mus.Aligned.out.sorted.bam         |  DEEP_ARCHIVE |
|  facs_bam_files/A1-B000167-3_56_F-1-1_R1.mus.Aligned.out.sorted.bam.bai     |  DEEP_ARCHIVE |
|  facs_bam_files/A1-B000167-3_56_F-1-1_R2.mus.Aligned.out.sorted.bam         |  DEEP_ARCHIVE |
|  facs_bam_files/A1-B000167-3_56_F-1-1_R2.mus.Aligned.out.sorted.bam.bai     |  DEEP_ARCHIVE |
|  facs_bam_files/A1-B000168-3_57_F-1-1_R1.mus.Aligned.out.sorted.bam         |  STANDARD     |
|  facs_bam_files/A1-B000168-3_57_F-1-1_R1.mus.Aligned.out.sorted.bam.bai     |  DEEP_ARCHIVE |
|  facs_bam_files/A1-B000168-3_57_F-1-1_R2.mus.Aligned.out.sorted.bam         |  STANDARD     |

Note that when I try to cp a file with the "STANDARD" storage label, it does work.