Merge pull request #137 from Caltech-IPAC/raen/IRSA-7280/euclid-uri-to-path

Author: bsipocz

tutorials/cloud_access/euclid-cloud-access.md

@@ -277,10 +277,10 @@ spec_association_tbl
 If you picked a target other than what this notebook uses, it's possible that there is no spectrum associated for your target's object ID. In that case, `spec_association_tbl` will contain 0 rows.
 ```
 
-In above table, we can see that the `uri` column gives us location of spectra file on IBE. We can map it to S3 bucket key to retrieve spectra file from the cloud. This is a very big FITS spectra file with multiple extensions where each extension contains spectrum of one object. The `hdu` column gives us the extension number for our object. So let's extract both of these.
+In above table, we can see that the `'path'` column gives us a url that can be used to call an IRSA service to get the spectrum of our object as SpectrumDM VOTable. We can map it to an S3 bucket key to retrieve a spectra file from the cloud. This is a very big FITS spectra file with multiple extensions where each extension contains spectrum of one object. The `'hdu'` column gives us the extension number for our object. So let's extract both of these.
 
 ```{code-cell} ipython3
-spec_fpath_key = spec_association_tbl['uri'][0].replace('ibe/data/euclid/', '')
+spec_fpath_key = spec_association_tbl['path'][0].replace('api/spectrumdm/convert/euclid/', '').split('?')[0]
 spec_fpath_key
 ```
 
@@ -295,14 +295,16 @@ Again, we use astropy's lazy-loading capability of FITS to only retrieve the spe
 with fits.open(f's3://{BUCKET_NAME}/{spec_fpath_key}', fsspec_kwargs={'anon': True}) as hdul:
     spec_hdu = hdul[object_hdu_idx]
     spec_tbl = Table.read(spec_hdu)
+    spec_header = spec_hdu.header
 ```
 
 ```{code-cell} ipython3
 spec_tbl
 ```
 
 ```{code-cell} ipython3
-plt.plot(spec_tbl['WAVELENGTH'], spec_tbl['SIGNAL'])
+# The signal needs to be multiplied by the scale factor in the header.
+plt.plot(spec_tbl['WAVELENGTH'], spec_header['FSCALE'] * spec_tbl['SIGNAL'])
 plt.xlabel(spec_tbl['WAVELENGTH'].unit.to_string('latex_inline'))
 plt.ylabel(spec_tbl['SIGNAL'].unit.to_string('latex_inline'))
 
@@ -311,8 +313,8 @@ plt.title(f'Spectrum of Target: {target_name}\n(Euclid Object ID: {object_id})')
 
 ## About this Notebook
 
-**Author:** Jaladh Singhal (IRSA Developer) in conjunction with Vandana Desai, Brigitta Sipőcz, Tiffany Meshkat and the IPAC Science Platform team
+**Author:** Jaladh Singhal (IRSA Developer) in conjunction with Vandana Desai, Brigitta Sipőcz, Tiffany Meshkat, Troy Raen, and the IRSA Data Science Team
 
-**Updated:** 2025-03-17
+**Updated:** 2025-09-23
 
 **Contact:** the [IRSA Helpdesk](https://irsa.ipac.caltech.edu/docs/help_desk.html) with questions or reporting problems.

tutorials/euclid_access/3_Euclid_intro_1D_spectra.md

@@ -55,8 +55,6 @@ We rely on ``astroquery`` features that have been recently added, so please make
 ```
 
 ```{code-cell} ipython3
-import urllib
-
 import numpy as np
 import matplotlib.pyplot as plt
 
@@ -100,40 +98,26 @@ result = Irsa.query_tap(adql_object).to_table()
 Pull out the file name from the ``result`` table:
 
 ```{code-cell} ipython3
-file_uri = urllib.parse.urljoin(Irsa.tap_url, result['uri'][0])
-file_uri
+spectrum_path = f"https://irsa.ipac.caltech.edu/{result['path'][0]}"
+spectrum_path
 ```
 
 ## 3. Read in the spectrum for only our specific object
 
-Currently IRSA has the spectra stored in very large files containing multiple (14220) extensions with spectra of many targets within one tile. You can choose to read in the big file below to see what it looks like (takes a few mins to load) or skip this step and just read in the specific extension we want for the 1D spectra (recommended).
-
-```{code-cell} ipython3
-# hdul = fits.open(file_uri)
-# hdul.info()
-```
-
-Open the large FITS file without loading it entirely into memory, pulling out just the extension we want for the 1D spectra of our object
+`spectrum_path` is a url that will return a VOTable containing the spectrum of our object.
 
 ```{code-cell} ipython3
-with fits.open(file_uri) as hdul:
-    spectrum = QTable.read(hdul[result['hdu'][0]], format='fits')
-
-    spec_header = hdul[result['hdu'][0]].header
+spectrum = QTable.read(spectrum_path)
 ```
 
 ```{code-cell} ipython3
 spectrum
 ```
 
-```{code-cell} ipython3
-spec_header
-```
-
 ## 4. Plot the image of the extracted spectrum
 
 ```{tip}
-As we use astropy.visualization's ``quantity_support``, matplotlib automatically picks up the axis units from the quantitites we plot.
+As we use astropy.visualization's ``quantity_support``, matplotlib automatically picks up the axis units from the quantities we plot.
 ```
 
 ```{code-cell} ipython3
@@ -143,15 +127,11 @@ quantity_support()
 ```{note}
 The 1D combined spectra table contains 6 columns, below are a few highlights:
 
-- WAVELENGTH is in Angstroms by default
-- SIGNAL is the flux and should be multiplied by the FSCALE factor in the header
+- WAVELENGTH is in Angstroms by default.
+- SIGNAL is the flux. The values are scaled and the scaling factor is included in the column's units. This value corresponds to the `'FSCALE'` entry in the HDU header of the original FITS file.
 - MASK values can be used to determine which flux bins to discard. MASK = odd and MASK >=64 means the flux bins not be used.
 ```
 
-```{code-cell} ipython3
-signal_scaled = spectrum['SIGNAL'] * spec_header['FSCALE']
-```
-
 We investigate the MASK column to see which flux bins are recommended to keep vs "Do Not Use"
 
 ```{code-cell} ipython3
@@ -165,19 +145,19 @@ We use the MASK column to create a boolean mask for values to ignore. We use the
 ```{code-cell} ipython3
 bad_mask = (spectrum['MASK'].value % 2 == 1) | (spectrum['MASK'].value >= 64)
 
-plt.plot(spectrum['WAVELENGTH'].to(u.micron), np.ma.masked_where(bad_mask, signal_scaled), color='black', label='Spectrum')
-plt.plot(spectrum['WAVELENGTH'], np.ma.masked_where(~bad_mask, signal_scaled), color='red', label='Do not use')
-plt.plot(spectrum['WAVELENGTH'], np.sqrt(spectrum['VAR']) * spec_header['FSCALE'], color='grey', label='Error')
+plt.plot(spectrum['WAVELENGTH'].to(u.micron), np.ma.masked_where(bad_mask, spectrum['SIGNAL']), color='black', label='Spectrum')
+plt.plot(spectrum['WAVELENGTH'], np.ma.masked_where(~bad_mask, spectrum['SIGNAL']), color='red', label='Do not use')
+plt.plot(spectrum['WAVELENGTH'], np.sqrt(spectrum['VAR']), color='grey', label='Error')
 
 plt.legend(loc='upper right')
-plt.ylim(-0.15E-16, 0.25E-16)
+plt.ylim(-0.15, 0.25)
 plt.title(f'Object ID {obj_id}')
 ```
 
 ## About this Notebook
 
-**Author**: Tiffany Meshkat, Anahita Alavi, Anastasia Laity, Andreas Faisst, Brigitta Sipőcz, Dan Masters, Harry Teplitz, Jaladh Singhal, Shoubaneh Hemmati, Vandana Desai
+**Author**: Tiffany Meshkat, Anahita Alavi, Anastasia Laity, Andreas Faisst, Brigitta Sipőcz, Dan Masters, Harry Teplitz, Jaladh Singhal, Shoubaneh Hemmati, Vandana Desai, Troy Raen
 
-**Updated**: 2025-03-31
+**Updated**: 2025-09-23
 
 **Contact:** [the IRSA Helpdesk](https://irsa.ipac.caltech.edu/docs/help_desk.html) with questions or reporting problems.

tutorials/euclid_access/4_Euclid_intro_PHZ_catalog.md

@@ -307,20 +307,18 @@ result_spectra
 Pull out the file name from the ``result_spectra`` table:
 
 ```{code-cell} ipython3
-file_uri = urllib.parse.urljoin(Irsa.tap_url, result_spectra['uri'][0])
-file_uri
+spectrum_path = f"https://irsa.ipac.caltech.edu/{result_spectra['path'][0]}"
+spectrum_path
 ```
 
 ```{code-cell} ipython3
-with fits.open(file_uri) as hdul:
-    spectrum = QTable.read(hdul[result_spectra['hdu'][0]], format='fits')
-    spectrum_header = hdul[result_spectra['hdu'][0]].header
+spectrum = QTable.read(spectrum_path)
 ```
 
 ### Now the data are read in, plot the spectrum
 
 ```{tip}
-As we use astropy.visualization’s quantity_support, matplotlib automatically picks up the axis units from the quantitites we plot.
+As we use astropy.visualization’s quantity_support, matplotlib automatically picks up the axis units from the quantities we plot.
 ```
 
 ```{code-cell} ipython3
@@ -418,8 +416,8 @@ fc.show_table(uploaded_table)
 
 ## About this Notebook
 
-**Author**: Tiffany Meshkat, Anahita Alavi, Anastasia Laity, Andreas Faisst, Brigitta Sipőcz, Dan Masters, Harry Teplitz, Jaladh Singhal, Shoubaneh Hemmati, Vandana Desai
+**Author**: Tiffany Meshkat, Anahita Alavi, Anastasia Laity, Andreas Faisst, Brigitta Sipőcz, Dan Masters, Harry Teplitz, Jaladh Singhal, Shoubaneh Hemmati, Vandana Desai, Troy Raen
 
-**Updated**: 2025-04-10
+**Updated**: 2025-09-24
 
 **Contact:** [the IRSA Helpdesk](https://irsa.ipac.caltech.edu/docs/help_desk.html) with questions or reporting problems.

tutorials/euclid_access/5_Euclid_intro_SPE_catalog.md

@@ -58,9 +58,6 @@ We rely on ``astroquery`` features that have been recently added, so please make
 ```
 
 ```{code-cell} ipython3
-import re
-import urllib
-
 import matplotlib.pyplot as plt
 import numpy as np
 
@@ -159,7 +156,7 @@ We specify the following conditions on our search:
 - Signal to noise ratio column (_gf = gaussian fit) should be greater than 5
 - We want to detect H-alpha.
 - We choose in which tileID to search, usign the tileID from the first notebook.
-- Choose spectroscopic redshift (spe_z) beween 1.4 and 1.6 and spe_z_prob greater than 0.999
+- Choose spectroscopic redshift (spe_z) between 1.4 and 1.6 and spe_z_prob greater than 0.999
 - H-alpha line flux should be more than 2x10^16 erg s^-1 cm^-2
 - Join the lines and galaxy candidates tables on object_id and spe_rank
 
@@ -208,23 +205,19 @@ result_table2 = Irsa.query_tap(adql_object).to_qtable()
 
 ### The following steps to read in the spectrum follows the 3_Euclid_intro_1D_spectra notebook.
 
-This involves reading in the spectrum without readin in the full FITS file, just pulling the extension we want.
-
 ```{code-cell} ipython3
-file_uri = urllib.parse.urljoin(Irsa.tap_url, result_table2['uri'][0])
-file_uri
+spectrum_path = f"https://irsa.ipac.caltech.edu/{result_table2['path'][0]}"
+spectrum_path
 ```
 
 ```{code-cell} ipython3
-with fits.open(file_uri) as hdul:
-    spectrum = QTable.read(hdul[result_table2['hdu'][0]], format='fits')
-    spec_header = hdul[result_table2['hdu'][0]].header
+spectrum = QTable.read(spectrum_path)
 ```
 
 ### Now the data are read in, plot the spectrum with the H-alpha line labeled
 
 ```{tip}
-As we use astropy.visualization's ``quantity_support``, matplotlib automatically picks up the axis units from the quantitites we plot.
+As we use astropy.visualization's ``quantity_support``, matplotlib automatically picks up the axis units from the quantities we plot.
 ```
 
 ```{code-cell} ipython3
@@ -248,8 +241,8 @@ plt.title(f'Object ID {obj_id}')
 
 ## About this Notebook
 
-**Author**: Tiffany Meshkat, Anahita Alavi, Anastasia Laity, Andreas Faisst, Brigitta Sipőcz, Dan Masters, Harry Teplitz, Jaladh Singhal, Shoubaneh Hemmati, Vandana Desai
+**Author**: Tiffany Meshkat, Anahita Alavi, Anastasia Laity, Andreas Faisst, Brigitta Sipőcz, Dan Masters, Harry Teplitz, Jaladh Singhal, Shoubaneh Hemmati, Vandana Desai, Troy Raen
 
-**Updated**: 2025-03-31
+**Updated**: 2025-09-23
 
 **Contact:** [the IRSA Helpdesk](https://irsa.ipac.caltech.edu/docs/help_desk.html) with questions or reporting problems.