You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: book/background/1_context_motivation.md
+2-1Lines changed: 2 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -13,10 +13,11 @@ Technological developments in recent decades have engendered fundamental shifts
13
13
14
14
## *Increasingly large, cloud-optimized data means new tools and approaches for data management*
15
15
16
-
The increase in publicly available earth observation data has transformed scientific workflows across a range of fields, prompting analysts to gain new skills in order to work with larger volumes of data in new formats and locations, and to use distributed cloud-computational resources in their analysis ({cite:t}`abernathey_2021_cloud,Boulton02012018,gentemann_2021_science,mathieu_2017_esas,ramachandran_2021_open,Sudmanns_2020_big,wagemann_2021_user`).
16
+
The increasing volume of publicly available earth observation data has transformed scientific workflows across a range of fields, prompting analysts to gain new skills in order to work with larger volumes of data in new formats and locations, and to use distributed cloud-computational resources in their analysis ({cite:t}`abernathey_2021_cloud,Boulton02012018,gentemann_2021_science,mathieu_2017_esas,ramachandran_2021_open,Sudmanns_2020_big,wagemann_2021_user,wagemann_2022_FiveGuidingPrinciples`). {numref}`eo_data_trend` shows the recent trend and projected continued increases in the volume of NASA Earth Science data archives. New satellites like [NISAR](https://nisar.jpl.nasa.gov/) will add to the growth of the data archives.
17
17
18
18
```{figure} imgs/fy24-projection-chart.png
19
19
---
20
+
name: eo_data_trend
20
21
---
21
22
Volume of NASA Earth Science Data archives, including growth of existing-mission archives and new missions, projected through 2029. Source: [NASA EarthData - Open Science](https://www.earthdata.nasa.gov/about/open-science).
Copy file name to clipboardExpand all lines: book/background/2_data_cubes.md
+5-6Lines changed: 5 additions & 6 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -18,11 +18,10 @@ Fundamentally, many of these complexities can be reduced to one distinction: is
18
18
19
19
### *An example dataset as a Xarray cube*
20
20
21
-
Imagine we have a time series of [NDVI](https://www.usgs.gov/landsat-missions/landsat-normalized-difference-vegetation-index) imagery generated from a stack of Landsat scenes. Before a user accesses a satellite imagery dataset, it has likely already undergone many levels of processing, transformation and re-organization. For more background on these steps, see Montero et al. {cite:t}`montero_2024_EarthSystemData`, *Section 3: 'The Earth System Data Cube Life cycle'*.
22
-
23
-
In this example, we're accessing the dataset at a common dissemination point, an 'image collection'[^mynote1]. It looks something like this:
24
-
```{figure} imgs/image_stack.png
21
+
Imagine we have a time series of [NDVI](https://www.usgs.gov/landsat-missions/landsat-normalized-difference-vegetation-index) imagery generated from a stack of Landsat scenes. Before a user accesses a satellite imagery dataset, it has likely already undergone many levels of processing, transformation and re-organization. For more background on these steps, see Montero et al. {cite:t}`montero_2024_EarthSystemData`, *Section 3: 'The Earth System Data Cube Life cycle'*. In this example, we're accessing the dataset at a common dissemination point, an 'image collection'[^mynote1], a schematic of which is shown in {numref}`2d-stack`.
22
+
```{figure} imgs/2d_collection.png
25
23
---
24
+
name: 2d-stack
26
25
---
27
26
Illustration of earth observation time series as a stack of 2-d images and associated metadata.
28
27
@@ -34,11 +33,11 @@ Without coordinate information and metadata, the image data are abstract arrays,
34
33
To use this data for scientific analysis, we need to construct it into the form of a cube. This requires a comprehensive understanding of the different pieces of information contained in the dataset and how they relate to one another in order to map the components of the dataset onto a cube structure.
35
34
```{figure} imgs/cube.png
36
35
---
36
+
name: 3d-cube
37
37
---
38
38
Illustration of earth observation time series organized as a 3-d Xarray data cube. Source: Adapted from [Xarray Dev](https://xarray.dev/).
39
39
```
40
-
41
-
In the context of the Xarray data model, univariate data cubes can be represented by an `xr.DataArray` or a `xr.Dataset` with one `data_variable`. Multivariate data cubes should be represented by `xr.Dataset` objects. The building blocks of `xr.DataArrays` and `xr.Datasets` are dimensions, coordinates, data variables, attributes. We recommend the Xarray [terminology](https://docs.xarray.dev/en/stable/user-guide/terminology.html) for a detailed overview of Xarray objects and common operations.
40
+
In the context of the Xarray data model, univariate data cubes can be represented by an `xr.DataArray` or a `xr.Dataset` with one `data_variable`. {numref}`3d-cube` illustrates how to represent multivariate data cubes using `xr.Dataset` objects. The building blocks of `xr.DataArrays` and `xr.Datasets` are dimensions, coordinates, data variables, attributes. We recommend the Xarray [terminology](https://docs.xarray.dev/en/stable/user-guide/terminology.html) for a detailed overview of Xarray objects and common operations.
42
41
43
42
We've just discussed what a data cube is in the context of a standard earth observation dataset and how to use the Xarray data model to efficiently represent this kind of data. Another way of describing those steps is preparing the dataset so that it is fit for analysis.
Copy file name to clipboardExpand all lines: book/background/4_tutorial_data.md
+6-5Lines changed: 6 additions & 5 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -17,11 +17,12 @@ ITS_LIVE is a dataset of ice velocity observations derived from applying a featu
17
17
18
18
```{figure} imgs/lopez06-3341335.png
19
19
---
20
+
name: ITS_LIVE-time-series
20
21
---
21
-
Example of a ice velocity time series along centerline profile of Malaspina Glacier featuring velocity observations from a range of satellite sensors. Source: Reproduced with permission from {cite:t}`lopez_2023_itslive`.
22
+
Example of a ice velocity time series along a profile of Malaspina Glacier featuring velocity observations from a range of satellite sensors. Source: Reproduced with permission from {cite:t}`lopez_2023_itslive`.
22
23
```
23
24
24
-
Part of what is so exciting about ITS_LIVE is that it combines image pairs from a number of satellites, including imagery from optical (Landsat 4,5,7,8,9 & Sentinel-2) and synthetic aperture radar (Sentinel-1) sensors. For this reason, ITS_LIVE time series data can be quite large. Another exciting aspect of the ITS_LIVE dataset is that the image pair time series data is made available as Zarr data cubes stored in cloud object storage on Amazon Web Services (AWS), meaning that users don't need to download massive files to start working with the data!
25
+
{numref}`ITS_LIVE-time-series` shows an ITS_LIVE time series at various locations on the Malaspina glacier and the satellite sensors that contribute observations throughout the time series. Part of what is so exciting about ITS_LIVE is that it combines image pairs from a number of satellites, including imagery from optical (Landsat 4,5,7,8,9 & Sentinel-2) and synthetic aperture radar (Sentinel-1) sensors. For this reason, ITS_LIVE time series data can be quite large. Another exciting aspect of the ITS_LIVE dataset is that the image pair time series data is made available as Zarr data cubes stored in cloud object storage on Amazon Web Services (AWS), meaning that users don't need to download massive files to start working with the data!
25
26
26
27
27
28
:::{admonition} A note about working with image pair time series
@@ -64,13 +65,13 @@ We provide a very brief overview of RTC processing below but it is not intended
64
65
---
65
66
height: 250 px
66
67
figclass: margin-caption
67
-
name: SARdiagram
68
+
name: SAR-diagram
68
69
---
69
70
Schematic of observation geometry used to form a SAR image.
70
-
Credit: [NASA EarthData / NASA SAR Handbook](https://www.earthdata.nasa.gov/learn/earth-observation-data-basics/sar).
71
+
Source: [NASA EarthData / NASA SAR Handbook](https://www.earthdata.nasa.gov/learn/earth-observation-data-basics/sar).
71
72
```
72
73
73
-
SAR data is collected in slant range, which is the viewing geometry of the side-looking sensor and has two dimensions: range and azimuth. These are the along-track and across-track directions of the imaged swath. As data is transformed from radar coordinates (slant range) to geocoded coordinates, the spaces represented by individual pixels in the two coordinate systems do not always align, and distortions can arise due to certain viewing angle geometries and surface topography features. In addition, radiometric distortion can arise due to scattering responses from multiple scattering features within a single pixel. Radiometric terrain correction is a processing step that accounts for these distortions and the transformation from radar to geocoded coordinates that prepares SAR data for analysis.
74
+
SAR data is collected in slant range, which is the viewing geometry of the side-looking sensor and has two dimensions: range and azimuth. These are the along-track and across-track directions of the imaged swath. {numref}`SAR-diagram` illustrates the viewing geometry of a SAR image. As data is transformed from radar coordinates (slant range) to geocoded coordinates, the spaces represented by individual pixels in the two coordinate systems do not always align, and distortions can arise due to certain viewing angle geometries and surface topography features. In addition, radiometric distortion can arise due to scattering responses from multiple scattering features within a single pixel. Radiometric terrain correction is a processing step that accounts for these distortions and the transformation from radar to geocoded coordinates that prepares SAR data for analysis.
Copy file name to clipboardExpand all lines: book/book_refs.bib
+17Lines changed: 17 additions & 0 deletions
Original file line number
Diff line number
Diff line change
@@ -545,6 +545,23 @@ @article{wagemann_2021_user
545
545
year = {2021},
546
546
pages = {1758--1774},
547
547
}
548
+
@article{wagemann_2022_FiveGuidingPrinciples,
549
+
title = {Five {{Guiding Principles}} to {{Make Jupyter Notebooks Fit}} for {{Earth Observation Data Education}}},
550
+
author = {Wagemann, Julia and Fierli, Federico and Mantovani, Simone and Siemen, Stephan and Seeger, Bernhard and Bendix, J{\"o}rg},
551
+
year = {2022},
552
+
month = jul,
553
+
journal = {Remote Sensing},
554
+
volume = {14},
555
+
number = {14},
556
+
pages = {3359},
557
+
issn = {2072-4292},
558
+
doi = {10.3390/rs14143359},
559
+
urldate = {2025-04-28},
560
+
abstract = {There is a growing demand to train Earth Observation (EO) data users in how to access and use existing and upcoming data. A promising tool for data-related training is computational notebooks, which are interactive web applications that combine text, code and computational output. Here, we present the Learning Tool for Python (LTPy), which is a training course (based on Jupyter notebooks) on atmospheric composition data. LTPy consists of more than 70 notebooks and has taught over 1000 EO data users so far, whose feedback is overall positive. We adapted five guiding principles from different fields (mainly scientific computing and Jupyter notebook research) to make the Jupyter notebooks more educational and reusable. The Jupyter notebooks developed (i) follow the literate programming paradigm by a text/code ratio of 3, (ii) use instructional design elements to improve navigation and user experience, (iii) modularize functions to follow best practices for scientific computing, (iv) leverage the wider Jupyter ecosystem to make content accessible and (v) aim for being reproducible. We see two areas for future developments: first, to collect feedback and evaluate whether the instructional design elements proposed meet their objective; and second, to develop tools that automatize the implementation of best practices.},
Copy file name to clipboardExpand all lines: paper.bib
+17Lines changed: 17 additions & 0 deletions
Original file line number
Diff line number
Diff line change
@@ -263,6 +263,23 @@ @article{wagemann_2021_user
263
263
pages = {1758--1774},
264
264
}
265
265
266
+
@article{wagemann_2022_FiveGuidingPrinciples,
267
+
title = {Five {{Guiding Principles}} to {{Make Jupyter Notebooks Fit}} for {{Earth Observation Data Education}}},
268
+
author = {Wagemann, Julia and Fierli, Federico and Mantovani, Simone and Siemen, Stephan and Seeger, Bernhard and Bendix, J{\"o}rg},
269
+
year = {2022},
270
+
month = jul,
271
+
journal = {Remote Sensing},
272
+
volume = {14},
273
+
number = {14},
274
+
pages = {3359},
275
+
issn = {2072-4292},
276
+
doi = {10.3390/rs14143359},
277
+
urldate = {2025-04-28},
278
+
abstract = {There is a growing demand to train Earth Observation (EO) data users in how to access and use existing and upcoming data. A promising tool for data-related training is computational notebooks, which are interactive web applications that combine text, code and computational output. Here, we present the Learning Tool for Python (LTPy), which is a training course (based on Jupyter notebooks) on atmospheric composition data. LTPy consists of more than 70 notebooks and has taught over 1000 EO data users so far, whose feedback is overall positive. We adapted five guiding principles from different fields (mainly scientific computing and Jupyter notebook research) to make the Jupyter notebooks more educational and reusable. The Jupyter notebooks developed (i) follow the literate programming paradigm by a text/code ratio of 3, (ii) use instructional design elements to improve navigation and user experience, (iii) modularize functions to follow best practices for scientific computing, (iv) leverage the wider Jupyter ecosystem to make content accessible and (v) aim for being reproducible. We see two areas for future developments: first, to collect feedback and evaluate whether the instructional design elements proposed meet their objective; and second, to develop tools that automatize the implementation of best practices.},
Copy file name to clipboardExpand all lines: paper.md
+2-2Lines changed: 2 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -39,10 +39,10 @@ bibliography: "paper.bib"
39
39
---
40
40
41
41
# Summary
42
-
Advances in cloud computing, remote sensing, and engineering are transforming earth system science into an increasingly data-intensive field, requiring students and scientists to learn a broad range of new skills related to scientific programming, data management, and cloud infrastructure [@abernathey_2021_cloud; @gentemann_2021_science; @guo_2017_big; @mathieu_2017_esas; @ramachandran_2021_open; @wagemann_2021_user]. This work contains educational modules designed to reduce barriers to interacting with large, complex, cloud-hosted remote sensing datasets using open-source computational tools and software. The goal of these materials is to demonstrate and promote the rigorous investigation of n-dimensional multi-sensor satellite imagery datasets through scientific programming. These tutorials feature publicly available satellite imagery with global coverage and commonly used sensors such as optical and synthetic aperture radar data with different levels of processing. We include thorough discussions of specific data formats and demonstrate access patterns for two popular cloud infrastructure platforms (Amazon Web Services and Microsoft Planetary Computer) as well as public cloud computational resources for remote sensing data processing at Alaska Satellite Facility (ASF).
42
+
Advances in cloud computing, remote sensing, and engineering are transforming earth system science into an increasingly data-intensive field, requiring students and scientists to learn a broad range of new skills related to scientific programming, data management, and cloud infrastructure [@abernathey_2021_cloud; @gentemann_2021_science; @guo_2017_big; @mathieu_2017_esas; @ramachandran_2021_open; @wagemann_2021_user; wagemann_2022_FiveGuidingPrinciples]. This work contains educational modules designed to reduce barriers to interacting with large, complex, cloud-hosted remote sensing datasets using open-source computational tools and software. The goal of these materials is to demonstrate and promote the rigorous investigation of n-dimensional multi-sensor satellite imagery datasets through scientific programming. These tutorials feature publicly available satellite imagery with global coverage and commonly used sensors such as optical and synthetic aperture radar data with different levels of processing. We include thorough discussions of specific data formats and demonstrate access patterns for two popular cloud infrastructure platforms (Amazon Web Services and Microsoft Planetary Computer) as well as public cloud computational resources for remote sensing data processing at Alaska Satellite Facility (ASF).
43
43
44
44
# Statement of Need
45
-
Research on the transition to data-intensive, cloud-based science highlights the need for knowledge development to accompany technological advances in order to realize the benefit of these transformations [@abernathey_2021_cloud; @gentemann_2021_science; @guo_2017_big; @mathieu_2017_esas; @palumbo_2017_building; @radocaj_2020_global; @ramachandran_2021_open; @Sudmanns_2020_big; @wagemann_2021_user].
45
+
Research on the transition to data-intensive, cloud-based science highlights the need for knowledge development to accompany technological advances in order to realize the benefit of these transformations [@abernathey_2021_cloud; @gentemann_2021_science; @guo_2017_big; @mathieu_2017_esas; @palumbo_2017_building; @radocaj_2020_global; @ramachandran_2021_open; @Sudmanns_2020_big; @wagemann_2021_user; @wagemann_2022_FiveGuidingPrinciples].
46
46
47
47
These educational modules address this need and are guided by principles identified in Diataxis [@Procida_Diataxis_documentation_framework] in order to help analysts engage in data-driven scientific discovery using cloud-based data and open-source tools.
0 commit comments