You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: documentation/spherex_data_access.md
+30-25Lines changed: 30 additions & 25 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,3 +1,4 @@
1
+
(data-access)=
1
2
# SPHEREx Data Access
2
3
3
4
IRSA serves SPHEREx data from two locations: (1) on premises at IPAC; and (2) on the cloud via Amazon Web Services (AWS).
@@ -7,7 +8,7 @@ These layers include:
7
8
***Browsable Directories:** SPHEREx on-premises data products are laid out in directories that can be navigated with standard web browsers.
8
9
These data products are mirrored on AWS.
9
10
***Application Program Interfaces:** IRSA provides program-friendly Application Program Interfaces (APIs) to access SPHEREx Spectral Image data.
10
-
The on-prem and cloud-hosted Quick Release Spectral Images that have been released thus far are accessible via the [Simple Image Access V2 protocol](https://ivoa.net/documents/SIA/20151223/) defined by the International Virtual Observatory Alliance ([IVOA](https://ivoa.net)).
11
+
The on-prem and cloud-hosted Quick Release 2 Spectral Images that have been released thus far are accessible via the [Simple Image Access V2 protocol](https://ivoa.net/documents/SIA/20151223/) defined by the International Virtual Observatory Alliance ([IVOA](https://ivoa.net)).
11
12
Cutouts of the Spectral Image data held on-prem are available via IRSA's Cutout Service.
12
13
***Python Packages:** SPHEREx data at IRSA are accessible via the Python packages [pyvo](https://pyvo.readthedocs.io/en/latest/) and [astroquery](https://astroquery.readthedocs.io/en/latest/ipac/irsa/irsa.html).
13
14
***SPHEREx Data Explorer:** IRSA provides a web-based Graphical User Interface (GUI) that makes it easy to search for, visualize, and download SPHEREx data.
@@ -17,13 +18,21 @@ Each of these data access layers is described in greater detail in the subsectio
17
18
18
19
## Browsable Directories
19
20
20
-
SPHEREx on-prem data products are laid out in directories that can be navigated with standard web browsers.
21
+
SPHEREx data products are laid out in directories that can be navigated with standard web browsers.
21
22
This is convenient for users to get a quick sense of the types of data products that are available, to quickly download some examples by clicking through the directory tree, and to script bulk downloads using `wget` or `curl`.
22
23
23
-
The root of the SPHEREx data quick release data directories is:
24
-
https://irsa.ipac.caltech.edu/ibe/data/spherex/qr
24
+
The root of the SPHEREx QR2 on-premises data directories is: [https://irsa.ipac.caltech.edu/ibe/data/spherex/qr2](https://irsa.ipac.caltech.edu/ibe/data/spherex/qr2).
25
+
For QR1, replace '/qr2' with '/qr'.
26
+
All of the data products are also available on the cloud via AWS.
27
+
Please see our [instructions for accessing on-cloud SPHEREx data](https://irsa.ipac.caltech.edu/cloud_access/#spherex).
28
+
29
+
:::{note}
30
+
QR1 is superseded by QR2.
31
+
The QR1 files will be available through January 2026 for reference.
32
+
:::
25
33
26
34
The public data products are organized into subdirectories based on the following organizational scheme:
35
+
27
36
***Absolute Gain Matrix:**`abs_gain_matrix/cal-agm-v[Version]-[Processing Date]/[Detector]/`
28
37
***Exposure-Averaged Point Spread Functions (PSFs):**`average_psf/cal-psf-v[Version]-[Processing Date]/[Detector]/`
***Spectral WCS:**`spectral_wcs/cal-wcs-v[Version]-[Processing Date]/[Detector]/` (QR2) or `spectral_wcs/base-[Processing Date]/[Detector]/` (QR1)
38
47
39
-
The content of each subdirectory is described in greater detail in the Data Products section of this user guide and in the [SPHEREx Explanatory Supplement](https://irsa.ipac.caltech.edu/data/SPHEREx/docs/SPHEREx_Expsupp_QR.pdf).
40
-
41
-
All of the data products listed above are also available on the cloud via AWS.
42
-
Please see our [instructions for accesssing cloud-hosted SPHEREx data](https://irsa.ipac.caltech.edu/cloud_access/).
48
+
The content of each subdirectory and the filename formats are described in greater detail in the {ref}`Data Products <data-products>` section of this user guide and in the [SPHEREx Explanatory Supplement](https://irsa.ipac.caltech.edu/data/SPHEREx/docs/SPHEREx_Expsupp_QR.pdf).
43
49
44
50
## Application Program Interfaces (APIs)
45
51
@@ -54,8 +60,8 @@ Additional [documentation on IRSA’s SIA2 service](https://irsa.ipac.caltech.ed
54
60
55
61
:::{note}
56
62
SPHEREx data are ingested on a weekly basis.
57
-
Due to the nature of the ingestion process, new SPHEREx data will first be available in the browsable directories and in the SPHEREx Data Explorer GUI.
58
-
Availability via SIA2 and Python libraries like Astroquery and PyVO will lag on the order of a day.
63
+
Due to the nature of the ingestion process, new SPHEREx data will first be available in the browsable directories, the SPHEREx Data Explorer GUI, and certain TAP queries (see the [cutouts notebook](https://caltech-ipac.github.io/irsa-tutorials/spherex-cutouts/) for a TAP example).
64
+
Availability via SIA2 will lag on the order of a day.
59
65
:::
60
66
61
67
IRSA's generic SIA2 endpoint is:
@@ -65,34 +71,34 @@ IRSA's generic SIA2 endpoint is:
65
71
Users must add a `COLLECTION` parameter to this endpoint to specify which dataset to search.
66
72
There are three SPHEREx-related SIA2 collections:
67
73
68
-
* SPHEREx Quick Release Spectral Image MEFs that are part of the SPHEREx **Wide Survey** can be accessed with: `COLLECTION=spherex_qr`.
69
-
Use this collection if you are interested in more uniform coverage across the entire sky and want to ignore the additional coverage in the deep fields.
74
+
* SPHEREx QR2 Spectral Image MEFs that are part of the SPHEREx **Wide Survey** can be accessed with: `COLLECTION=spherex_qr2`.
75
+
Use this collection if you are interested in more uniform coverage across the entire sky and want to ignore the additional coverage in the deep fields.
70
76
71
-
* SPHEREx Quick Release Spectral Image MEFs that are part of the SPHEREx **Deep Survey** can be accessed with: `COLLECTION=spherex_qr_deep`.
77
+
* SPHEREx QR2 Spectral Image MEFs that are part of the SPHEREx **Deep Survey** can be accessed with: `COLLECTION=spherex_qr2_deep`.
72
78
73
-
* SPHEREx Quick Release **Calibration files** can be accessed with: `COLLECTION=spherex_qr_cal`.
79
+
* SPHEREx QR2 **Calibration files** can be accessed with: `COLLECTION=spherex_qr2_cal`.
74
80
75
81
You can use `wget` or `curl` to submit SIA2 queries from the command line.
See the section on Python packages to learn how to use Python wrappers around IRSA’s SIA2 service.
81
87
88
+
(access-spectral-image-cutouts)=
82
89
### Cutouts of SPHEREx Spectral Image MEFs
83
90
84
-
If you have identified the access URL for an on-premises Spectral Image MEF using SIA2 as described above, you can request a cutout of this MEF by appending a query string containing the center and size parameters. The parameters are described in more detail here:
85
-
86
-
`https://irsa.ipac.caltech.edu/ibe/cutouts.html`
91
+
If you have identified the access URL for an on-premises Spectral Image MEF using SIA2 as described above, you can request a cutout of this MEF by appending a query string containing the center and size parameters.
92
+
The parameters are described in more detail at [https://irsa.ipac.caltech.edu/ibe/cutouts.html](https://irsa.ipac.caltech.edu/ibe/cutouts.html).
This cutout service is also invoked by the SPHEREx Data Collection Explorer Spectral Image Search when users select the cutout option upon download.
93
99
94
100
This cutout service can also be invoked via Python, as illustrated in the Python Tutorial notebook titled [Download a collection of SPHEREx Spectral Image cutouts as a multi-extension FITS file](https://caltech-ipac.github.io/irsa-tutorials/spherex-cutouts/).
95
-
Information on how to work with the PSF extension in these cutouts is documented in the [Data Products section](https://caltech-ipac.github.io/spherex-archive-documentation/spherex-data-products#cutouts-of-spectral-image-mefs) of this User Guide and demonstrated in the Python Tutorial notebook titled [Understanding and Extracting the PSF Extension in a SPHEREx Cutout](https://caltech-ipac.github.io/irsa-tutorials/spherex-psf/).
101
+
Information on how to work with the PSF extension in these cutouts is documented in the {ref}`products-spectral-image-cutouts` of this User Guide and demonstrated in the Python Tutorial notebook titled [Understanding and Extracting the PSF Extension in a SPHEREx Cutout](https://caltech-ipac.github.io/irsa-tutorials/spherex-psf/).
96
102
97
103
## Python packages: PyVO & Astroquery
98
104
@@ -110,11 +116,10 @@ The notebook titled [Introduction to SPHEREx Spectral Images](https://caltech-ip
110
116
111
117
The notebook titled [Download a collection of SPHEREx Spectral Image cutouts as a multi-extension FITS file](https://caltech-ipac.github.io/irsa-tutorials/spherex-cutouts/#id-5-query-irsa-for-a-list-of-cutouts-that-satisfy-the-criteria-specified-above) demonstrates how to use the PyVO library to execute an IVOA Table Access Protocol (TAP) query for SPHEREx spectral images that cover the specified coordinates and match the specified bandpass.
112
118
113
-
114
119
:::{note}
115
120
SPHEREx data are ingested on a weekly basis.
116
-
Due to the nature of the ingestion process, new SPHEREx data will first be available in the browsable directories and in the SPHEREx Data Explorer GUI.
117
-
Availability via SIA2 and Python libraries like Astroquery and PyVO will lag on the order of a day.
121
+
Due to the nature of the ingestion process, new SPHEREx data will first be available in the browsable directories, the SPHEREx Data Explorer GUI, and certain TAP queries (see the [cutouts notebook](https://caltech-ipac.github.io/irsa-tutorials/spherex-cutouts/) for a TAP example).
122
+
Availability via SIA2 will lag on the order of a day.
Copy file name to clipboardExpand all lines: documentation/spherex_data_products.md
+31-12Lines changed: 31 additions & 12 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,10 +1,11 @@
1
+
(data-products)=
1
2
# SPHEREx Data Products
2
3
3
4
IRSA began releasing SPHEREx Spectral Image data on a weekly basis in July 2025 (Quick Release 1; QR1).
4
5
In October 2025, IRSA began distributing SPHEREx Spectral Image data processed with substantially improved calibrations.
5
6
This new processing, referred to as QR2, includes reprocessed versions of all Spectral Image data acquired since the start of the mission.
6
7
Future quick releases will also use the QR2 pipeline.
7
-
IRSA will continue to provide access to QR1 data for reference.
8
+
IRSA will continue to provide access to QR1 data through January 2026 for reference.
8
9
However, QR2 supersedes QR1 and will be the default returned in the SPHEREx Data Explorer and by all IRSA program-friendly APIs.
9
10
QR1 data will remain available only through browsable directory listings.
10
11
@@ -29,7 +30,7 @@ Some large slews will have fewer than 4 small slews.
29
30
30
31
## Main Science Data Product: Spectral Image Multi-Extension FITS Files (MEF)
31
32
32
-
The main Quick Release data product is the Level 2 Spectral Image MEF,as described in Section 2.1 of the [Explanatory Supplement](https://irsa.ipac.caltech.edu/data/SPHEREx/docs/SPHEREx_Expsupp_QR.pdf).
33
+
The main Quick Release data product is the Level 2 Spectral Image MEF,as described in Section 2.1 of the [Explanatory Supplement](https://irsa.ipac.caltech.edu/data/SPHEREx/docs/SPHEREx_Expsupp_QR.pdf).
33
34
34
35
There are 6 Spectral MEFs (one for each detector) for each sky pointing.
35
36
Because data quality assessments are evaluated per spectral image band, some observations will not include all 6 bands in the archive.
@@ -38,7 +39,7 @@ Each Spectral MEF is approximately 70 MB and contains 6 extensions:
38
39
HDU 1: IMAGE
39
40
: Calibrated surface brightness flux density in units of MJy/sr, stored as a 2040 x 2040 image.
40
41
No zodiacal light subtraction is applied.
41
-
42
+
42
43
43
44
The SPHEREx focal plane is split with a dichroic to three short-wavelength and three long-wavelength detector arrays.
44
45
Two focal plane assemblies (FPAs) simultaneously image the sky through a dichroic beam splitter.
@@ -88,32 +89,33 @@ VALUES is an array of two-element arrays: at each (X, Y) control point, the two-
IRSA's Cutout Service provides spatial subsets of the SPHEREx Spectral Image MEFs.
100
-
Information on how to use the Cutout Service is provided in the [Data Access](https://caltech-ipac.github.io/spherex-archive-documentation/spherex-data-access#cutouts-of-spherex-spectral-image-mefs) section of this User Guide.
102
+
Information on how to use the Cutout Service is provided in the {ref}`access-spectral-image-cutouts` section of this User Guide.
101
103
The cutout MEFs returned from this service contain the same HDUs as the original Spectral Images (IMAGE, FLAGS, VARIANCE, ZODI, PSF, WCS-WAVE). However, the IMAGE, FLAGS, VARIANCE, AND ZODI HDUs have been modified to include only those pixels within the specified cutout size.
102
104
The WCS-WAVE HDU has also modified to provide the correct mapping between the pixels in the cutout to wavelength.
103
105
The PSF HDU from the original spectral image is included unmodified in the cutout MEF.
104
106
105
107
The spatially-varying PSF is represented as an image cube with 121 planes.
106
108
Each plane is a 101x101 pixel image representing a PSF for a different region of the detector. Users interested in performing photometry on a cutout using the information in the cutout PSF HDU will need to understand how to find the most applicable PSF cube plane for each pixel in the cutout.
107
-
The basic steps are described below, and a [Python notebook tutorial](https://caltech-ipac.github.io/irsa-tutorials/) is provided to help users get started with a simple implementation.
109
+
The basic steps are described below, and a [Python notebook tutorial](https://caltech-ipac.github.io/irsa-tutorials/spherex-psf/) is provided to help users get started with a simple implementation.
108
110
109
111
1. Determine the 0-based pixel coordinates of the position of interest in the IMAGE HDU of the cutout.
110
112
111
113
2. Determine the 0-based pixel coordinates of the position of interest in the IMAGE HDU of the original Spectral Image.
112
114
113
-
```
114
-
xpix_orig = 1 + xpix_cutout - CRPIX1A
115
-
ypix_orig = 1 + ypix_cutout - CRPIX2A
116
-
```
115
+
```
116
+
xpix_orig = 1 + xpix_cutout - CRPIX1A
117
+
ypix_orig = 1 + ypix_cutout - CRPIX2A
118
+
```
117
119
118
120
3. Examine the header of the PSF HDU of the cutout to determine the PSF zone and cube plane corresponding to the pixel of interest in the original Spectral Image.
119
121
@@ -242,6 +244,21 @@ Each extension is an image with dimensions 2040 × 2040 and units of electrons.
242
244
243
245
-`readnoise_pars_D1_spx_base-2025-158.fits`
244
246
247
+
248
+
## Additional Product: Solid Angle Pixel Map
249
+
250
+
The Solid Angle Pixel Map products are ~16 MB FITS image files (one per detector) with dimensions 2,040 x 2,040.
251
+
Pixel values measure the solid angle in units of squared arcsec.
0 commit comments