Merge pull request #30 from nsidc/content-mapping

mikala-nsidc · web-flow · commit 357c07c5529f · 2025-10-03T08:04:46.000-06:00
added get-edl file
diff --git a/.gitignore b/.gitignore
@@ -4,3 +4,4 @@ example_data/
 *.h5
 *.nc
 _freeze
+.venv/
diff --git a/_quarto.yml b/_quarto.yml
@@ -27,60 +27,65 @@ website:
         href: index.qmd
       - section: "Getting Started"
         contents:
-          - text: "How to use this cookbook"
-            href: our-cookbook.qmd
+          - text: "Getting an Earthdata Login"
+            href: reference-guides/get-edl.qmd            
           - text: "Tools we use"
             href: tools-we-use.qmd
 #      - text: "Quick Start Guide"
 #        # href: quickstart.qmd
       - section: "NSIDC Data"
         # href: reference-guides/data-orient-overview.qmd
         contents:
-           - text: "Types of data"
+           - text: "Data types and formats"
              href: reference-guides/data-structures.qmd
-           - text: "Common file formats"
-             href: reference-guides/nsidc-file-formats.qmd
            - text: "NASA collections and granules"
            - text: "NSIDC map projections and grids"
              href: reference-guides/projections.qmd
-           - text: "Quirky Datasets"
+           - text: "Quirky datasets"
              href: reference-guides/datasets.qmd
       - section: "Working with Data"
         # href: reference-guides/working-with-data-overview.qmd
         contents:
-          - section: "Finding and Accessing Data"
+          - section: "Finding / accessing data"
             contents:
-              - text: "Getting an Earthdata Login"
-              #  href: reference-guides/get-edl.qmd
               - text: "Data search and discovery"
                 href: reference-guides/dataset-search-and-discovery.qmd
               - text: "Data access"
-          - section: "Wrangling Data"
+                href: reference-guides/data-access.qmd
+          - section: "Wrangling data"
             contents:
               - text: "Subsetting"
               - text: "Reformatting"
               - text: "Reprojecting"
               - text: "Resampling"
-          - section: "Visualizing Data"
+          - section: "Visualizing data"
             contents:
               - text: "Plotting data - time series, scatter plots"
               - text: "Plotting data on a map"
                 href: reference-guides/plotting-data.qmd
-      - text: "Best Practices"
-      - section: "How do I..."
+      - section: "How-To Guides"
         href: how-to-guides/overview.qmd
         contents:
           - section: "Search for data"
           - section: "Access data"
           - section: "Reproject data"
-          - section: "Work with NetCDF files?"
+          - section: "Work with NetCDF files"
             contents:
-            - text: "get the bounding box of a netcdf file"
+            - text: "Get the bounding box of a NetCDF file"
               href: how-to-guides/netcdf_cf.qmd
-            - text: "get the latitude and longitudes for grid cells"
+            - text: "Get the latitude and longitudes for grid cells"
               href: how-to-guides/get_latitude_and_longitude.qmd
+      - section: "Tutorials / Workshops"
+        href: tutorials/tutorials_overview.qmd
+        contents:
+          - text: "Direct cloud access of ICESat-2 data"
+            href: tutorials/ATL06-direct-access.ipynb
+          - text: "Download ICESat-2 Sea Ice data"
+            href: tutorials/working_with_icesat2_sea_ice_data.ipynb
       - section: "Appendices"
         contents:
+          - text: "Contributing"
+          - text: "Best Practices"
           - text: "What is an API?"
 
 format:
diff --git a/how-to-guides/netcdf_cf.qmd b/how-to-guides/netcdf_cf.qmd
@@ -1,5 +1,5 @@
 ---
-title: "How do I get the bounding box of a NetCDF file in latitude and longitude?"
+title: "How to get the bounding box of a NetCDF file in latitude and longitude"
 author: Andrew P. Barrett
 date: last-modified
 ---
diff --git a/how-to-guides/overview.qmd b/how-to-guides/overview.qmd
@@ -1,27 +1,23 @@
 ---
 title: "How-To Guides"
-author: Andy P. Barrett
 ---
 
 ## Introduction
 
-This section of the cookbook contains **How To Guides** to help you
+This section of the cookbook contains **How-To Guides** to help you
 solve a particular problem or task.  You can think of these How-To
-Guides as recipes.  For the most part they are short.  They are
+Guides as programmatic recipes.  For the most part they are short.  They are
 written with the assumption that you know what you want to do and have
-some understanding of programing.  You can think of each How-To as a
+some understanding of programming.  You can think of each How-To as a
 building block that can be put together with other How-Tos to construct
 a workflow.
 
-If you are just learning either programming or working in the cloud,
-we recommend looking at the other chapters in this cookbook.
-
 How-To Guides are organized into the following sections:
-_Maybe add a short description of each section here_
 
-- How do I search for data; 
-- How do I accessing data;
-- How do I reproject and resample data;
-- How do I work with CF-compliant NetCDF files
-  - [How do I get the bounding box of a NetCDF file in latitude and longitude?](netcdf_cf.qmd)
+- How to search for data (via `earthaccess`); 
+- How to access data (via `earthaccess`);
+- How to reproject and resample data;
+- How to work with CF-compliant NetCDF files:
+  - [How to get the bounding box of a NetCDF file in latitude and longitude](netcdf_cf.qmd)
+  - [How to get latitude and longitude for the grid cells of a NetCDF file](get_latitude_and_longitude.qmd)
 
diff --git a/index.qmd b/index.qmd
@@ -3,14 +3,35 @@ title: "NSIDC Data Cookbook"
 author: NSIDC
 ---
 
-## Welcome
-
-Welcome to the NSIDC Data Cookbook!
-
 <!-- prettier-ignore -->
 > [!WARNING]
 > This cookbook is under active development. Major changes to the structure of
 > the book and its content are expected. We are striving to develop content
 > that is well-tested and peer-reviewed, but nothing contained here should be
 > expected to work correctly (or at all!) in this early phase.  Many sections contain only an outline of the content.
-> These sections will have content added at the project develops.
+> These sections will have content added as the project develops.
+
+## Welcome to the NSIDC Data Cookbook! 
+
+This Cookbook for National Snow and Ice Data Center Distributed Active Archive Center ([NSIDC DAAC](https://nsidc.org/data/data-programs/nsidc-daac)) Data is more Julia Child’s “Mastering the Art of French Cooking” than just a collection of recipes (or data tutorials). The aim is to not only provide easy to follow recipes for working with data but also to provide an understanding of the data managed by NSIDC and the tools available to work with the data. It is not an in-depth guide to the many datasets housed by NSIDC but instead is an introduction to the common types of data, file formats and data structures. The hope is that the cookbook will provide a guide and foundataion to help you *master the art of working with cryospheric data*.
+
+## How to use this cookbook
+
+If you are new to NASA or NSIDC data, begin with the **Getting Started** section. Beyond that, the cookbook is divided into four major sections: **NSIDC Data**, **Working with Data**, **How-To Guides**, and **Tutorials / Workshops**. We also include some **Appendices** that include a best practices guide for Python scripting, some background details on computing, and how to contribute to the cookbook (if you are so inclined!).
+
+The **NSIDC Data** section is an introduction to the types of data managed by NSIDC, the file formats used to store that data, coordinate reference systems and grids comon to NSIDC data, and a guide to NASA terminology for Earth science data. There is also an introduction to some “quirky” datasets that do not fit more common simple data structures or are just a little more difficult to work with.
+
+The **Working with Data** section is a guide to the tools and applications for accomplishing common steps in scientific workflows: finding and accessing data, wrangling data, and visualizing data. Wrangling data covers subsetting, reformatting, reprojecting and resampling data in preparation for analysis.
+
+The **How-To Guides** section is a list of recipes to acheive common tasks. It is intended a list of solutions to common scientific programming tasks.
+
+Lastly, the **Tutorials / Workshops** section serves to highlight content that was developed for past events or specific use cases.
+
+## About the NSIDC DAAC
+
+NSIDC has managed the NASA National Snow and Ice Data Center Distributed Active Archive Center (NSIDC DAAC) since 1993, archiving and distributing cryospheric and related geophysical data from NASA Earth-observing satellite missions, airborne campaigns and field observations. The NSIDC DAAC provides hundreds of free and open-access NASA Earth science data sets, detailed data documentation, data tools, resources and tutorials, as well as  robust data user support services. These data can be used to study topics relating to snow cover, sea ice, ice sheets, ice shelves, glaciers, frozen ground, soil moisture, climate interactions, and more!
+
+
+
+
+
diff --git a/reference-guides/data-access.qmd b/reference-guides/data-access.qmd
@@ -0,0 +1,45 @@
+---
+title: "Data access"
+---
+
+#### **Prerequisite reminder** 
+
+A [NASA Earthdata Login](https://urs.earthdata.nasa.gov/) is required for all data access methods, including download and direct cloud access.
+
+#### **NASA Earthdata Cloud**
+
+The NASA Earthdata Cloud is NASA’s cloud-based archive of Earth observation data. It is hosted by Amazon Web Services (AWS) in region us-west-2. Downloading data from the Earthdata Cloud to your local computer or storage system is and will continue to be free for users. You don't need an AWS account to download data.  Alternatively, you can work directly with NSIDC DAAC holdings stored in Amazon S3 (the object storage service used by NASA Earthdata Cloud). This direct access method allows you stream data into memory and analyze it "in place", avoiding large transfers to your local computer.
+
+Options for accessing NASA Earthdata are outlined below:
+
+#### **Option 1: Browser-Based Download (No Coding Required)**
+
+If you prefer a graphical interface, use one of these web-based tools: 
+
+[NASA Earthdata Search](https://search.earthdata.nasa.gov) - Explore, filter and customize data before downloading files. Instructions for using Earthdata Search to find and access data can be found in our Earthdata Search [guide](https://nsidc.org/data/user-resources/help-center/search-order-and-customize-nsidc-daac-data-nasa-earthdata-search). 
+
+image
+
+Data Access Tool - The NSIDC Data Access Tool can be accessed from a data set's landing page. 
+Example: [ATLAS/ICESat-2 L2A Global Geolocated Photon Data. (ATL03, Version 7) Landing Page](https://nsidc.org/data/atl03/versions/7). In the right-hand menu on a data set landing page, click "Data Access & Tools".  You will be directed to the top of a list of tool and service "cards" with links to various data access methods.  Click on the card title "Data Access Tool" to be directed to the Data Access Tool interface for that particular data set.  The Data Access Tool allows users to filter files within a data set using spatial bounds, temporal ranges, and filename wildcards.
+
+Help article on using the Data Access Tool: [https://nsidc.org/data/user-resources/help-center/filter-and-order-data-set-web-page-using-data-access-tool](https://nsidc.org/data/user-resources/help-center/filter-and-order-data-set-web-page-using-data-access-tool)
+
+#### **Option 2: Python (earthaccess Library)**
+
+The earthaccess Python library provides a streamlined way to search, authenticate, and download NASA Earthdata. It works both locally and in the cloud (e.g., on EC2 or JupyterHub environments).
+
+earthaccess [documentation](https://earthaccess.readthedocs.io/en/stable/)
+
+*Have reference/link to one of our "how do I's" here?*
+
+#### **Option 3: Command Line Tools (```wget```, ```curl```, PODAAC subscriber)**
+
+If you're comfortable using the terminal, command-line tools allow flexible and efficient downloading. 
+
+```wget``` or ```curl``` – Download known files or batch download from an HTTPS URL list.
+Learn to create .txt files of download links here: [Creating Text Files of HTTPS and S3 URLs for Earthdata Cloud Data Access](https://nsidc.org/data/user-resources/help-center/creating-text-files-https-and-s3-urls-earthdata-cloud-data-access) 
+
+[PODAAC Data Subscriber](https://github.com/podaac/data-subscriber/blob/main/README.md) – A Python-based command-line tool that supports spatial and temporal filtering. Though designed for PODAAC, it can be adapted for NSIDC and other DAACs.
+
+All of these command line options are detailed in this help article: [Downloading Data from Earthdata Cloud to Your Local Computer Using the Command Line](https://nsidc.org/data/user-resources/help-center/downloading-data-earthdata-cloud-your-local-computer-using-command-line)
diff --git a/reference-guides/data-structures.qmd b/reference-guides/data-structures.qmd
@@ -1,12 +1,23 @@
 ---
-title: "Data Structures"
+title: "Data types and formats"
 ---
 
-Describes common remote sensing data structures.
+There are many common data types/ structures and terminology to go along with them. Some examples:
 
-_We should add information for tabular data and Data Frames_
+- **Tabular** – rows and columns, often stored in CSV or TSV files. Each row is an observation, and each column is a variable (e.g., time, latitude, longitude, temperature).  
+- **Data Frames** – tabular data structures used in programming languages like R or Python (pandas). Data frames allow for more complex indexing, metadata, and transformations than simple tabular files.  
+- **Swath** – along-track measurements collected as the satellite passes over an area, usually irregular in shape and resolution.  
+- **Raster / Grids** – data organized into regular grid cells, each cell representing a spatial unit (e.g., 25 km × 25 km grid of snow cover).  
+- **Resampling** – methods for transforming data between swath, raster, or other structures (e.g., nearest neighbor, bilinear interpolation).  
 
-- Swath  
-- Along-track  
-- Raster/Grids  
-- Resampling
+
+How to work with file formats commonly found at NSIDC: In most cases, it’s best to avoid low-level libraries such as `netCDF4` or `h5py`. Higher-level libraries provide more intuitive access, automatically handle metadata, and streamline analysis. Some format descriptions and reccomendations  are in the table below.
+
+| File Format           | Description                                                                                 | Recommended Tools                                        |
+| --------------------- | ------------------------------------------------------------------------------------------- | -------------------------------------------------------- |
+| **NetCDF4 / NetCDFx** | Multidimensional climate/remote sensing data (time, lat, lon, variables).                   | `xarray` (`xr.open_dataset`) in Python; `terra` or `ncdf4` in R.    |
+| **HDF5**              | Hierarchical format for storing arrays, tables, and metadata; used widely in NASA products. | `xarray`, `pandas`; avoid `h5py` unless necessary.       |
+| **HDF-EOS**           | Earth Observing System variant of HDF, often with swath, grid, or point structures.         | `xarray`, `h5netcdf`, NASA `harmony-py`.                 |
+| **Shapefile**         | Vector geospatial data (points, lines, polygons) with CRS support.                          | `geopandas` (Python); `sf` (R).                          |
+| **GeoTIFF**           | Georeferenced raster imagery and gridded data.                                              | `rasterio`, `rioxarray` (Python); `terra`, `raster` (R). |
+| **CSV/TSV**           | Tabular text-based files, rows = observations, columns = variables.                         | `pandas` (Python); `readr`/`data.table`/`tibble` (R).    |
diff --git a/reference-guides/dataset-search-and-discovery.qmd b/reference-guides/dataset-search-and-discovery.qmd
@@ -2,8 +2,45 @@
 title: "Dataset Search and Discovery"
 ---
 
-- Getting an EDL  
-- A simple earthaccess search and download  
-- EDS  
-- curl
-- wget
+### **NASA Common Metadata Repository**
+
+There are a variety of search and discovery methods allowing users to find NASA Earthdata from a graphical user interface (GUI) or using programmatic access methods, depending on preference.  Any search and discovery method leverages the [NASA Common Metadata Repository](https://www.earthdata.nasa.gov/about/esdis/eosdis/cmr) to find data of interest.
+
+*NASA's Common Metadata Repository (CMR) is a high-performance, high-quality, continuously evolving metadata system that catalogs all data and service metadata records for NASA's Earth Observing System Data and Information System (EOSDIS) and will be the authoritative management system for all EOSDIS metadata. These metadata records are registered, modified, discovered, and accessed through programmatic interfaces leveraging standard protocols and APIs.*
+
+*CMR is the keystone that makes NASA's Earth observation data discoverable. As a metadata repository, CMR contains Unified Metadata Model (UMM) schema records that describe individual Earth data files (UMM-Granules), collections of files (UMM-Collections), scientific details about the data files (UMM-Variables), related tools and services that act on the data files (UMM-Tools and -Services), and pertinent relationships between these concepts. Using the UMM allows CMR to host its metadata records in several supported native formats, with translation services available between formats.* 
+
+### **Search and Discovery Methods**
+
+
+#### **Graphical User Interfaces**
+
+
+##### **NASA Earthdata Search** 
+
+All of NASA Earthdata are available through [NASA Earthdata Search](https://search.earthdata.nasa.gov), not just data archived through the NSIDC DAAC.  [Help article](https://nsidc.org/data/user-resources/help-center/search-order-and-customize-nsidc-daac-data-nasa-earthdata-search) for using Earthdata Search.  
+
+##### **NSIDC Data Access Tool**
+
+The NSIDC Data Access Tool is accessible from landing pages on the NSIDC website, and is an easy way to filter for files of interest without leaving nsidc.org. [Help article](https://nsidc.org/data/data-access-tool) for the NSIDC Data Access Tool.
+
+#### **Programmatic**
+
+The majority of the shorter form "How-To Guides" and longer form "Tutorials / Workshops" will present programmatic data access methods using Python.  We hope to expand to other languages, such as R, in the future.  
+
+**earthaccess Python library**
+
+`earthaccess` is a Python library to **search for** and **download** or **stream** NASA Earth science data with just a few lines of code.
+
+>*"earthaccess revolutionizes NASA data access by drastically reducing the complexity and code required. Since open science is a collaborative effort involving people from different technical backgrounds, our team took the approach that data analysis can and should be made more inclusive and accessible by reducing the complexities of underlying systems."*
+>
+>Luis López, an NSIDC software developer and earthaccess creator
+
+Instructions for searching for data using earthaccess [here](https://earthaccess.readthedocs.io/en/stable/user_guide/search/).
+
+
+
+
+
+
+
diff --git a/reference-guides/get-edl.qmd b/reference-guides/get-edl.qmd
@@ -0,0 +1,9 @@
+---
+title: "NASA Earthdata Login"
+---
+
+NASA Earthdata are freely accessible to all users, but a NASA Earthdata Login is required for access.  Users can register for an Earthdata Login at [https://urs.earthdata.nasa.gov/](https://urs.earthdata.nasa.gov/).
+
+
+Some programmatic data access methods are simplified by setting up your Earthdata Login credentials in a netrc file for easy authentication.
+Instructions for creating a netrc file can be found [here](https://nsidc.org/data/user-resources/help-center/creating-netrc-file-earthdata-login).
diff --git a/tools-we-use.qmd b/tools-we-use.qmd
diff --git a/tutorials/ATL06-direct-access.ipynb b/tutorials/ATL06-direct-access.ipynb
diff --git a/tutorials/tutorials_overview.qmd b/tutorials/tutorials_overview.qmd
diff --git a/tutorials/working_with_icesat2_sea_ice_data.ipynb b/tutorials/working_with_icesat2_sea_ice_data.ipynb

-Original file line number
+Diff line change
 *.h5
 *.nc
 _freeze
 +.venv/