Skip to content

Commit 11c2ddf

Browse files
bsipoczbsipocz
authored
Merge pull request #2235 from jaymedina/mast-cloudaccess-docs-update
Mast cloudaccess docs update
2 parents 40ac26f + 8ad106b commit 11c2ddf

File tree

1 file changed

+45
-60
lines changed

1 file changed

+45
-60
lines changed

‎docs/mast/mast.rst‎

Lines changed: 45 additions & 60 deletions
Original file line numberDiff line numberDiff line change
@@ -343,26 +343,18 @@ the *local_path* keyword argument.
343343
344344
Cloud Data Access
345345
------------------
346-
Public datasets from the Hubble, Kepler and TESS telescopes are also available on Amazon Web Services
346+
Public datasets from the Hubble, Kepler and TESS telescopes are also available for free on Amazon Web Services
347347
in `public S3 buckets <https://registry.opendata.aws/collab/stsci/>`__.
348348

349-
Using AWS resources to process public data requires an `AWS account <https://aws.amazon.com/>`__ and associated
350-
`credentials file <https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html>`__. The `boto3
351-
<https://boto3.amazonaws.com/v1/documentation/api/latest/index.html>`__ library is also required as it handles
352-
connections to the AWS servers. Instructions for creating AWS credentials are available `here
353-
<https://stackoverflow.com/questions/21440709/how-do-i-get-aws-access-key-id-for-amazon>`__. Data transfer charges
354-
are the responsibility of the requester (see `request pricing <https://aws.amazon.com/s3/pricing/>`__), however
355-
transfers are free within the US-East AWS region.
349+
Using AWS resources to process public data no longer requires an AWS account for all AWS regions. To enable cloud data access for the Hubble, Kepler, and TESS missions, follow the steps below:
356350

357-
Cload data access is enabled using the `~astroquery.mast.ObservationsClass.enable_cloud_dataset` function, which
358-
will cause AWS to become the prefered source for data access until it is disabled
359-
(`~astroquery.mast.ObservationsClass.disable_cloud_dataset`).
351+
You can enable cloud data access via the `~astroquery.mast.ObservationsClass.enable_cloud_dataset` function, which sets AWS to become the preferred source for data access as opposed to on-premise MAST until it is disabled with `~astroquery.mast.ObservationsClass.disable_cloud_dataset`.
360352

361353
To directly access a list of cloud URIs for a given dataset, use the `~astroquery.mast.ObservationsClass.get_cloud_uris`
362-
function, however when cloud access is enabled, the standatd download function
363-
`~astroquery.mast.ObservationsClass.download_products` will preferentially pull files from AWS when they are avilable.
364-
There is also a ``cloud_only`` flag, which when set to True will cause all data products not available in the
365-
cloud to be skipped.
354+
function (Python will prompt you to enable cloud access if you haven't already).
355+
356+
When cloud access is enabled, the standard download function
357+
`~astroquery.mast.ObservationsClass.download_products` preferentially pulls files from AWS when they are available. When set to `True`, the ``cloud_only`` parameter in `~astroquery.mast.ObservationsClass.download_products` skips all data products not available in the cloud.
366358

367359

368360
Getting a list of S3 URIs:
@@ -372,15 +364,9 @@ Getting a list of S3 URIs:
372364
>>> import os
373365
>>> from astroquery.mast import Observations
374366
375-
>>> # If credential environment are not already set, we can set them within python.
376-
>>> os.environ['AWS_ACCESS_KEY_ID'] = 'myaccesskeyid'
377-
>>> os.environ['AWS_SECRET_ACCESS_KEY'] = 'mysecretaccesskey'
378-
379-
>>> # If your profile is not called [default], update the next line:
380-
>>> Observations.enable_cloud_dataset(provider='AWS', profile='default')
367+
>>> # Simply call the `enable_cloud_dataset` method from `Observations`. The default provider is `AWS`, but we will write it in manually for this example:
368+
>>> Observations.enable_cloud_dataset(provider='AWS')
381369
INFO: Using the S3 STScI public dataset [astroquery.mast.core]
382-
INFO: See Request Pricing in https://aws.amazon.com/s3/pricing/ for details [astroquery.mast.core]
383-
INFO: If you have not configured boto3, follow the instructions here: https://boto3.readthedocs.io/en/latest/guide/configuration.html [astroquery.mast.core]
384370
385371
>>> # Getting the cloud URIs
386372
>>> obs_table = Observations.query_criteria(obs_collection='HST',
@@ -393,8 +379,8 @@ Getting a list of S3 URIs:
393379
... productSubGroupDescription='DRZ')
394380
>>> s3_uris = Observations.get_cloud_uris(filtered)
395381
>>> print(s3_uris)
396-
['s3://stpubdata/hst/public/jbev/jbeveo010/jbeveo010_drz.fits',
397-
's3://stpubdata/hst/public/jbev/jbevet010/jbevet010_drz.fits']
382+
['s3://stpubdata/hst/public/jbev/jbeveo010/jbeveo010_drz.fits', 's3://stpubdata/hst/public/jbev/jbeveo010/jbeveo010_drz.fits', 's3://stpubdata/hst/public/jbev/jbevet010/jbevet010_drz.fits', 's3://stpubdata/hst/public/jbev/jbevet010/jbevet010_drz.fits']
383+
398384
399385
>>> Observations.disable_cloud_dataset()
400386
@@ -406,45 +392,45 @@ Downloading data products from S3:
406392
>>> import os
407393
>>> from astroquery.mast import Observations
408394
409-
>>> # If credential environment are not already set, we can set them within python.
410-
>>> os.environ['AWS_ACCESS_KEY_ID'] = 'myaccesskeyid'
411-
>>> os.environ['AWS_SECRET_ACCESS_KEY'] = 'mysecretaccesskey'
412-
413-
>>> # If your profile is not called [default], update the next line:
414-
>>> Observations.enable_cloud_dataset(provider='AWS', profile='default')
395+
>>> # Simply call the `enable_cloud_dataset` method from `Observations`. The default provider is `AWS`, but we will write it in manually for this example:
396+
>>> Observations.enable_cloud_dataset(provider='AWS')
415397
INFO: Using the S3 STScI public dataset [astroquery.mast.core]
416-
INFO: See Request Pricing in https://aws.amazon.com/s3/pricing/ for details [astroquery.mast.core]
417-
INFO: If you have not configured boto3, follow the instructions here: https://boto3.readthedocs.io/en/latest/guide/configuration.html [astroquery.mast.core]
418398
419399
>>> # Downloading from the cloud
420400
>>> obs_table = Observations.query_criteria(obs_collection=['Kepler'],
421401
... objectname="Kepler 12b", radius=0)
422402
>>> products = Observations.get_product_list(obs_table[0])
423403
>>> manifest = Observations.download_products(products[:10], cloud_only=True)
424-
ERROR: Error pulling from S3 bucket: Parameter validation failed: Invalid type for parameter Key, value: None, type: <class 'NoneType'>, valid types: <class 'str'> [astroquery.mast.core]
425-
WARNING: Skipping file... [astroquery.mast.core]
426-
ERROR: Error pulling from S3 bucket: Parameter validation failed: Invalid type for parameter Key, value: None, type: <class 'NoneType'>, valid types: <class 'str'> [astroquery.mast.core]
427-
WARNING: Skipping file... [astroquery.mast.core]
428-
ERROR: Error pulling from S3 bucket: Parameter validation failed: Invalid type for parameter Key, value: None, type: <class 'NoneType'>, valid types: <class 'str'> [astroquery.mast.core]
429-
WARNING: Skipping file... [astroquery.mast.core]
430-
ERROR: Error pulling from S3 bucket: Parameter validation failed: Invalid type for parameter Key, value: None, type: <class 'NoneType'>, valid types: <class 'str'> [astroquery.mast.core]
431-
WARNING: Skipping file... [astroquery.mast.core]
432-
ERROR: Error pulling from S3 bucket: Parameter validation failed: Invalid type for parameter Key, value: None, type: <class 'NoneType'>, valid types: <class 'str'> [astroquery.mast.core]
433-
WARNING: Skipping file... [astroquery.mast.core]
434-
Downloading URL s3://stpubdata/kepler/public/lightcurves/0118/011804465/kplr011804465-2009131105131_llc.fits to ./mastDownload/Kepler/kplr011804465_lc_Q111111110111011101/kplr011804465-2009131105131_llc.fits ... [Done]
435-
Downloading URL s3://stpubdata/kepler/public/lightcurves/0118/011804465/kplr011804465-2009166043257_llc.fits to ./mastDownload/Kepler/kplr011804465_lc_Q111111110111011101/kplr011804465-2009166043257_llc.fits ... [Done]
436-
Downloading URL s3://stpubdata/kepler/public/lightcurves/0118/011804465/kplr011804465-2009259160929_llc.fits to ./mastDownload/Kepler/kplr011804465_lc_Q111111110111011101/kplr011804465-2009259160929_llc.fits ... [Done]
437-
Downloading URL s3://stpubdata/kepler/public/lightcurves/0118/011804465/kplr011804465-2009350155506_llc.fits to ./mastDownload/Kepler/kplr011804465_lc_Q111111110111011101/kplr011804465-2009350155506_llc.fits ... [Done]
438-
Downloading URL s3://stpubdata/kepler/public/lightcurves/0118/011804465/kplr011804465-2010009091648_llc.fits to ./mastDownload/Kepler/kplr011804465_lc_Q111111110111011101/kplr011804465-2010009091648_llc.fits ... [Done]
404+
405+
manifestDownloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:KEPLER/url/missions/kepler/dv_files/0118/011804465/kplr011804465-01-20160209194854_dvs.pdf to ./mastDownload/Kepler/kplr011804465_lc_Q111111110111011101/kplr011804465-01-20160209194854_dvs.pdf ...
406+
|==========================================| 1.5M/1.5M (100.00%) 0s
407+
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:KEPLER/url/missions/kepler/dv_files/0118/011804465/kplr011804465-20160128150956_dvt.fits to ./mastDownload/Kepler/kplr011804465_lc_Q111111110111011101/kplr011804465-20160128150956_dvt.fits ...
408+
|==========================================| 17M/ 17M (100.00%) 1s
409+
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:KEPLER/url/missions/kepler/dv_files/0118/011804465/kplr011804465-20160209194854_dvr.pdf to ./mastDownload/Kepler/kplr011804465_lc_Q111111110111011101/kplr011804465-20160209194854_dvr.pdf ...
410+
|==========================================| 5.8M/5.8M (100.00%) 0s
411+
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:KEPLER/url/missions/kepler/dv_files/0118/011804465/kplr011804465_q1_q17_dr25_obs_tcert.pdf to ./mastDownload/Kepler/kplr011804465_lc_Q111111110111011101/kplr011804465_q1_q17_dr25_obs_tcert.pdf ...
412+
|==========================================| 2.2M/2.2M (100.00%) 0s
413+
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:KEPLER/url/missions/kepler/previews/0118/011804465/kplr011804465-2013011073258_llc_bw_large.png to ./mastDownload/Kepler/kplr011804465_lc_Q111111110111011101/kplr011804465-2013011073258_llc_bw_large.png ...
414+
|==========================================| 24k/ 24k (100.00%) 0s
415+
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:KEPLER/url/missions/kepler/target_pixel_files/0118/011804465/kplr011804465_tpf_lc_Q111111110111011101.tar to ./mastDownload/Kepler/kplr011804465_lc_Q111111110111011101/kplr011804465_tpf_lc_Q111111110111011101.tar ...
416+
|==========================================| 43M/ 43M (100.00%) 4s
417+
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:KEPLER/url/missions/kepler/lightcurves/0118/011804465/kplr011804465_lc_Q111111110111011101.tar to ./mastDownload/Kepler/kplr011804465_lc_Q111111110111011101/kplr011804465_lc_Q111111110111011101.tar ...
418+
|==========================================| 5.9M/5.9M (100.00%) 0s
419+
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:KEPLER/url/missions/kepler/lightcurves/0118/011804465/kplr011804465-2009131105131_llc.fits to ./mastDownload/Kepler/kplr011804465_lc_Q111111110111011101/kplr011804465-2009131105131_llc.fits ...
420+
|==========================================| 77k/ 77k (100.00%) 0s
421+
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:KEPLER/url/missions/kepler/lightcurves/0118/011804465/kplr011804465-2009166043257_llc.fits to ./mastDownload/Kepler/kplr011804465_lc_Q111111110111011101/kplr011804465-2009166043257_llc.fits ...
422+
|==========================================| 192k/192k (100.00%) 0s
423+
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:KEPLER/url/missions/kepler/lightcurves/0118/011804465/kplr011804465-2009259160929_llc.fits to ./mastDownload/Kepler/kplr011804465_lc_Q111111110111011101/kplr011804465-2009259160929_llc.fits ...
424+
|==========================================| 466k/466k (100.00%) 0s
439425
440426
>>> print(manifest["Status"])
441-
Status
427+
Status
442428
--------
443-
SKIPPED
444-
SKIPPED
445-
SKIPPED
446-
SKIPPED
447-
SKIPPED
429+
COMPLETE
430+
COMPLETE
431+
COMPLETE
432+
COMPLETE
433+
COMPLETE
448434
COMPLETE
449435
COMPLETE
450436
COMPLETE
@@ -770,7 +756,7 @@ be accessed in Astroquery by using the Tesscut class.
770756
After the user has reached this limit TESScut will return a
771757
``503 Service Temporarily Unavailable Error``.
772758

773-
If you use TESSCut for your work, please cite Brasseur et al. 2019
759+
If you use TESSCut for your work, please cite Brasseur et al. 2019
774760
https://ui.adsabs.harvard.edu/abs/2019ascl.soft05007B/abstract
775761

776762

@@ -870,9 +856,9 @@ Zcut
870856
====
871857

872858

873-
Zcut for MAST allows users to request cutouts from various Hubble deep field surveys. The cutouts can
874-
be returned as either fits or image files (jpg and png are supported). This tool can be accessed in
875-
Astroquery by using the Zcut class. The list of supported deep field surveys can be found here:
859+
Zcut for MAST allows users to request cutouts from various Hubble deep field surveys. The cutouts can
860+
be returned as either fits or image files (jpg and png are supported). This tool can be accessed in
861+
Astroquery by using the Zcut class. The list of supported deep field surveys can be found here:
876862
https://mast.stsci.edu/zcut/
877863

878864

@@ -882,7 +868,7 @@ Cutouts
882868
The `~astroquery.mast.ZcutClass.get_cutouts` function takes a coordinate and cutout size (in pixels or
883869
an angular quantity) and returns the cutout FITS file(s) as a list of ~astropy.io.fits.HDUList objects.
884870

885-
If the given coordinate appears in more than one Zcut survey, a FITS file will be produced for each survey.
871+
If the given coordinate appears in more than one Zcut survey, a FITS file will be produced for each survey.
886872

887873
.. code-block:: python
888874
@@ -1084,4 +1070,3 @@ Reference/API
10841070
.. automodapi:: astroquery.mast
10851071
:no-inheritance-diagram:
10861072
:inherited-members:
1087-

0 commit comments

Comments
 (0)