Skip to content

Commit bbc8cf9

Browse files
gadomskiTom Augspurger
gadomski
and
Tom Augspurger
authored
Add noaa-cdr datasets (#82)
* feat: add noaa-cdr Collections: - noaa-cdr-ocean-heat-content - noaa-cdr-ocean-heat-content-netcdf - noaa-cdr-sea-ice-concentration - noaa-cdr-sea-surface-temperature-optimum-interpolation - noaa-cdr-sea-surface-temperature-whoi - noaa-cdr-sea-surface-temperature-whoi-netcdf --------- Co-authored-by: Tom Augspurger <[email protected]>
1 parent 19049c7 commit bbc8cf9

File tree

21 files changed

+1963
-5
lines changed

21 files changed

+1963
-5
lines changed

‎datasets/noaa-cdr/Dockerfile‎

Lines changed: 74 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,74 @@
1+
FROM ubuntu:20.04
2+
3+
# Setup timezone info
4+
ENV TZ=UTC
5+
6+
ENV LC_ALL=C.UTF-8
7+
ENV LANG=C.UTF-8
8+
9+
RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone
10+
11+
RUN apt-get update && apt-get install -y software-properties-common
12+
13+
RUN add-apt-repository ppa:ubuntugis/ppa && \
14+
apt-get update && \
15+
apt-get install -y build-essential python3-dev python3-pip \
16+
jq unzip ca-certificates wget curl git && \
17+
apt-get autoremove && apt-get autoclean && apt-get clean
18+
19+
RUN update-alternatives --install /usr/bin/python python /usr/bin/python3 10
20+
21+
# See https://github.com/mapbox/rasterio/issues/1289
22+
ENV CURL_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt
23+
24+
# Install Python 3.11
25+
RUN curl -L -O "https://github.com/conda-forge/miniforge/releases/latest/download/Mambaforge-$(uname)-$(uname -m).sh" \
26+
&& bash "Mambaforge-$(uname)-$(uname -m).sh" -b -p /opt/conda \
27+
&& rm -rf "Mambaforge-$(uname)-$(uname -m).sh"
28+
29+
ENV PATH /opt/conda/bin:$PATH
30+
ENV LD_LIBRARY_PATH /opt/conda/lib/:$LD_LIBRARY_PATH
31+
32+
RUN mamba install -y -c conda-forge python=3.11 gdal pip setuptools cython numpy
33+
34+
RUN python -m pip install --upgrade pip
35+
36+
# Install common packages
37+
COPY requirements-task-base.txt /tmp/requirements.txt
38+
RUN python -m pip install --no-build-isolation -r /tmp/requirements.txt
39+
40+
#
41+
# Copy and install packages
42+
#
43+
44+
COPY pctasks/core /opt/src/pctasks/core
45+
RUN cd /opt/src/pctasks/core && \
46+
pip install .
47+
48+
COPY pctasks/cli /opt/src/pctasks/cli
49+
RUN cd /opt/src/pctasks/cli && \
50+
pip install .
51+
52+
COPY pctasks/task /opt/src/pctasks/task
53+
RUN cd /opt/src/pctasks/task && \
54+
pip install .
55+
56+
COPY pctasks/client /opt/src/pctasks/client
57+
RUN cd /opt/src/pctasks/client && \
58+
pip install .
59+
60+
COPY pctasks/ingest /opt/src/pctasks/ingest
61+
RUN cd /opt/src/pctasks/ingest && \
62+
pip install .
63+
64+
COPY pctasks/dataset /opt/src/pctasks/dataset
65+
RUN cd /opt/src/pctasks/dataset && \
66+
pip install .
67+
68+
COPY ./datasets/noaa-cdr/requirements.txt /opt/src/datasets/noaa-cdr/requirements.txt
69+
RUN python3 -m pip install -r /opt/src/datasets/noaa-cdr/requirements.txt
70+
71+
# Setup Python Path to allow import of test modules
72+
ENV PYTHONPATH=/opt/src:$PYTHONPATH
73+
74+
WORKDIR /opt/src

‎datasets/noaa-cdr/README.md‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
# NOAA Climate Data Records (CDR)
2+
3+
### Dynamic updates
4+
5+
`noaa-cdr-sea-surface-temperature-optimum-interpolation` is updated daily.
6+
7+
```console
8+
$ pctasks dataset process-items '${{ args.since }}' \
9+
-d datasets/noaa-cdr/update.yaml \
10+
-c noaa-cdr-sea-surface-temperature-optimum-interpolation \
11+
--workflow-id=noaa-cdr-sea-surface-temperature-optimum-interpolation-update \
12+
--is-update-workflow \
13+
--upsert
14+
```

‎datasets/noaa-cdr/collections/ocean-heat-content-netcdf/description.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
The Ocean Heat Content Climate Data Record (CDR) is a set of ocean heat content anomaly (OHCA) time-series for 1955-present on 3-monthly, yearly, and pentadal (five-yearly) scales. This CDR quantifies ocean heat content change over time, which is an essential metric for understanding climate change and the Earth's energy budget. It provides time-series for multiple depth ranges in the global ocean and each of the major basins (Atlantic, Pacific, and Indian) divided by hemisphere (Northern, Southern).
2+
3+
This is a NetCDF-only collection, for Cloud-Optimized GeoTIFFs use collection `noaa-cdr-ocean-heat-content`.
4+
The NetCDF files are delivered to Azure as part of the [NOAA Open Data Dissemination (NODD) Program](https://www.noaa.gov/information-technology/open-data-dissemination).

‎datasets/noaa-cdr/collections/ocean-heat-content-netcdf/template.json‎

Lines changed: 131 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,131 @@
1+
{
2+
"type": "Collection",
3+
"id": "noaa-cdr-ocean-heat-content-netcdf",
4+
"stac_version": "1.0.0",
5+
"description": "{{ collection.description }}",
6+
"links": [
7+
{
8+
"rel": "license",
9+
"href": "https://www.ncei.noaa.gov/pub/data/sds/cdr/CDRs/Ocean_Heat_Content/UseAgreement_01B-41.pdf",
10+
"type": "application/pdf",
11+
"title": "NOAA CDR Ocean Heat Content Use Agreement"
12+
},
13+
{
14+
"rel": "about",
15+
"href": "https://www.ncei.noaa.gov/products/climate-data-records/global-ocean-heat-content",
16+
"type": "text/html",
17+
"title": "Global Ocean Heat Content CDR"
18+
},
19+
{
20+
"rel": "cite-as",
21+
"href": "https://doi.org/10.7289/v53f4mvp"
22+
}
23+
],
24+
"stac_extensions": [
25+
"https://stac-extensions.github.io/scientific/v1.0.0/schema.json",
26+
"https://stac-extensions.github.io/raster/v1.1.0/schema.json",
27+
"https://stac-extensions.github.io/item-assets/v1.0.0/schema.json"
28+
],
29+
"sci:doi": "10.7289/v53f4mvp",
30+
"sci:citation": "Levitus, Sydney; Antonov, John I.; Boyer, Tim P.; Baranova, Olga K.; García, Hernán E.; Locarnini, Ricardo A.; Mishonov, Alexey V.; Reagan, James R.; [Seidov, Dan; Yarosh, Evgeney; Zweng, Melissa M. (2017). NCEI ocean heat content, temperature anomalies, salinity anomalies, thermosteric sea level anomalies, halosteric sea level anomalies, and total steric sea level anomalies from 1955 to present calculated from in situ oceanographic subsurface profile data (NCEI Accession 0164586). NOAA National Centers for Environmental Information. Dataset. https://doi.org/10.7289/v53f4mvp.",
31+
"item_assets": {
32+
"netcdf": {
33+
"type": "application/netcdf",
34+
"roles": [
35+
"data"
36+
]
37+
}
38+
},
39+
"msft:short_description": "The Ocean Heat Content Climate Data Record (CDR) is a set of ocean heat content anomaly (OHCA) time-series for 1955-present on 3-monthly, yearly, and pentadal (five-yearly) scales.",
40+
"msft:storage_account": "noaacdr",
41+
"msft:container": "ocean-heat-content",
42+
"msft:group_id": "noaa-cdr",
43+
"msft:region": "eastus",
44+
"title": "Global Ocean Heat Content CDR NetCDFs",
45+
"extent": {
46+
"spatial": {
47+
"bbox": [
48+
[
49+
-180.0,
50+
-90.0,
51+
180.0,
52+
90.0
53+
]
54+
]
55+
},
56+
"temporal": {
57+
"interval": [
58+
[
59+
"1972-03-01T00:00:00Z",
60+
"2022-03-31T23:59:59Z"
61+
]
62+
]
63+
}
64+
},
65+
"license": "proprietary",
66+
"keywords": [
67+
"Global",
68+
"Climate",
69+
"NOAA",
70+
"Temperature",
71+
"Ocean"
72+
],
73+
"providers": [
74+
{
75+
"name": "National Centers for Environmental Information",
76+
"description": "NCEI is the Nation's leading authority for environmental data, and manage one of the largest archives of atmospheric, coastal, geophysical, and oceanic research in the world. NCEI contributes to the NESDIS mission by developing new products and services that span the science disciplines and enable better data discovery.",
77+
"roles": [
78+
"producer",
79+
"processor",
80+
"licensor"
81+
],
82+
"url": "https://www.ncei.noaa.gov/"
83+
},
84+
{
85+
"name": "Microsoft",
86+
"roles": [
87+
"processor",
88+
"host"
89+
],
90+
"url": "https://planetarycomputer.microsoft.com"
91+
}
92+
],
93+
"summaries": {
94+
"noaa_cdr:interval": [
95+
"monthly",
96+
"seasonal",
97+
"yearly",
98+
"pentadal"
99+
],
100+
"noaa_cdr:max_depth": [
101+
100,
102+
700,
103+
2000
104+
]
105+
},
106+
"assets": {
107+
"thumbnail": {
108+
"href": "https://ai4edatasetspublicassets.blob.core.windows.net/assets/pc_thumbnails/noaa-cdr-ocean-heat-content-netcdf-thumb.png",
109+
"type": "image/png",
110+
"title": "Global Ocean Heat Content (NetCDF) CDR thumbnail",
111+
"roles": [
112+
"thumbnail"
113+
]
114+
},
115+
"geoparquet-items": {
116+
"href": "abfs://items/noaa-cdr-ocean-heat-content-netcdf.parquet",
117+
"type": "application/x-parquet",
118+
"title": "GeoParquet STAC items",
119+
"description": "Snapshot of the collection's STAC items exported to GeoParquet format",
120+
"msft:partition_info": {
121+
"is_partitioned": false
122+
},
123+
"table:storage_options": {
124+
"account_name": "pcstacitems"
125+
},
126+
"roles": [
127+
"stac-items"
128+
]
129+
}
130+
}
131+
}

‎datasets/noaa-cdr/collections/ocean-heat-content/description.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
The Ocean Heat Content Climate Data Record (CDR) is a set of ocean heat content anomaly (OHCA) time-series for 1955-present on 3-monthly, yearly, and pentadal (five-yearly) scales. This CDR quantifies ocean heat content change over time, which is an essential metric for understanding climate change and the Earth's energy budget. It provides time-series for multiple depth ranges in the global ocean and each of the major basins (Atlantic, Pacific, and Indian) divided by hemisphere (Northern, Southern).
2+
3+
These Cloud Optimized GeoTIFFs (COGs) were created from NetCDF files which are delivered to Azure as part of the [NOAA Open Data Dissemination (NODD) Program](https://www.noaa.gov/information-technology/open-data-dissemination).
4+
For the NetCDF files, see collection `noaa-cdr-ocean-heat-content-netcdf`.

0 commit comments

Comments
 (0)