Skip to content

Commit de780d8

Browse files
committed
adjust order of datastores and create subchapter for stac and cci
1 parent 47114b9 commit de780d8

File tree

1 file changed

+163
-146
lines changed

1 file changed

+163
-146
lines changed

docs/source/dataaccess.md

Lines changed: 163 additions & 146 deletions
Original file line numberDiff line numberDiff line change
@@ -207,6 +207,7 @@ Use `list_data_store_ids()` to list all available data stores.
207207
from xcube.core.store import list_data_store_ids
208208

209209
list_data_store_ids()
210+
```
210211

211212
### Filesystem-based data stores
212213

@@ -324,47 +325,141 @@ Common parameters for opening [xarray.Dataset] instances:
324325
been consolidated. Defaults to `False`.
325326
* `log_access: bool` - Defaults to `False`.
326327

327-
### ESA Climate Data Centre `cciodp`
328+
### Copernicus Climate Data Store `cds`
329+
330+
The data store `cds` provides datasets of the [Copernicus Climate Data Store].
331+
332+
This data store is provided by the xcube plugin [xcube-cds].
333+
You can install it using `conda install -c conda-forge xcube-cds`.
334+
335+
Data store parameters:
336+
337+
* `cds_api_key: str` - User API key for Copernicus Climate Data Store.
338+
* `endpoint_url: str` - API endpoint URL.
339+
* `num_retries: int` - Defaults to `200`.
340+
* `normalize_names: bool` - Defaults to `False`.
341+
342+
Common parameters for opening [xarray.Dataset] instances:
343+
344+
* `bbox: (float, float, float, float)` - Bounding box in geographical
345+
coordinates.
346+
* `time_range: (str, str)` - Time range.
347+
* `variable_names: list[str]` - List of names of variables to be included.
348+
Defaults to all.
349+
* `spatial_res: float` - Spatial resolution. Defaults to `0.1`.
350+
351+
### Copernicus Marine Service `cmems`
352+
353+
The data store `cmems` provides datasets of the [Copernicus Marine Service].
354+
355+
This data store is provided by the xcube plugin [xcube-cmems].
356+
You can install it using `conda install -c conda-forge xcube-cmems`.
357+
358+
Data store parameters:
328359

329-
The data store `cciodp` provides the datasets of
330-
the [ESA Climate Data Centre].
360+
* `cmems_username: str` - CMEMS API username
361+
* `cmems_password: str` - CMEMS API password
362+
* `cas_url: str` - Defaults to `'https://cmems-cas.cls.fr/cas/login'`.
363+
* `csw_url: str` - Defaults to `'https://cmems-catalog-ro.cls.fr/geonetwork/srv/eng/csw-MYOCEAN-CORE-PRODUCTS?'`.
364+
* `databases: str` - One of `['nrt', 'my']`.
365+
* `server: str` - Defaults to `'cmems-du.eu/thredds/dodsC/'`.
331366

332-
This data store is provided by the xcube plugin [xcube-cci].
367+
Common parameters for opening [xarray.Dataset] instances:
368+
369+
* `variable_names: list[str]` - List of variable names.
370+
* `time_range: [str, str]` - Time range.
371+
372+
### Copernicus Land Monitoring Service `clms`
373+
374+
The data store `clms` provides datasets of the [Copernicus Land Monitoring Service].
375+
376+
This data store is provided by the xcube plugin [xcube-clms].
377+
You can install it using `conda install -c conda-forge xcube-clms`.
378+
379+
Data store parameters:
380+
381+
* `credentials: dict`: CLMS API credentials that can be obtained following the steps outlined
382+
[here](https://eea.github.io/clms-api-docs/authentication.html).
383+
These are the credentials parameters:
384+
385+
* `client_id: str` - Required.
386+
* `issued: str`
387+
* `private_key: str` - Required.
388+
* `key_id: str`
389+
* `title: str`
390+
* `token_uri: str` - Required.
391+
* `user_id: str` - Required.
392+
393+
* `cache_store_id: str` - Store ID of cache data store. Defaults to `file`.
394+
* `cache_store_params: dict` - Store parameters of a filesystem-based data
395+
store.
396+
397+
Before opening a specific dataset from CLMS, it's required to preload the data first.
398+
Preloading lets you create data requests ahead of time, which may sit in a queue before
399+
being processed. Once processed, the data is downloaded as zip files, unzipped,
400+
extracted to a cache, and prepared for use. After this, it can be accessed through the
401+
cache data store.
402+
403+
The preload parameters are:
404+
* `blocking: bool` - Switch to make the preloading process blocking or non-blocking.
405+
If True, the preloading process blocks the script. Defaults to `True`.
406+
* `silent: bool` - Silence the output of Preload API. If True, no preload state
407+
output is given. Defaults to `False`.
408+
* `cleanup: bool` - Cleanup the download directory before and after the
409+
preload job and the cache directory when preload_handle.close() is called.
410+
Defaults to `True`.
411+
412+
Its common dataset open parameters for opening [xarray.Dataset] instances are the same
413+
as for the filesystem-based data stores described above.
414+
415+
### EOPF Sample Service
416+
417+
A data store for EOPF Sample Service is currently under development and will be released
418+
soon.
419+
420+
### ESA Climate Data Centre `cciodp`, `ccizar`, `esa-cci-kc`
421+
422+
Three data stores are provided by the xcube plugin [xcube-cci].
333423
You can install it using `conda install -c conda-forge xcube-cci`.
334424

425+
#### `cciodp`
426+
427+
The data store `cciodp` provides the datasets of
428+
the [ESA Climate Data Centre].
429+
335430
Data store parameters:
336431

337-
* `endpoint_url: str` - Defaults to
432+
* `endpoint_url: str` - Defaults to
338433
`'https://archive.opensearch.ceda.ac.uk/opensearch/request'`.
339-
* `endpoint_description_url: str` - Defaults to
434+
* `endpoint_description_url: str` - Defaults to
340435
`'https://archive.opensearch.ceda.ac.uk/opensearch/description.xml?parentIdentifier=cci'`.
341436
* `enable_warnings: bool` - Whether to output warnings. Defaults to `False`.
342-
* `num_retries: int` - Number of retries when requesting data fails.
437+
* `num_retries: int` - Number of retries when requesting data fails.
343438
Defaults to `200`.
344439
* `retry_backoff_max: int` - Defaults to `40`.
345440
* `retry_backoff_base: float` - Defaults to `1.001`.
346441

347442
Common parameters for opening [xarray.Dataset] instances:
348443

349444
* `variable_names: list[str]` - List of variable names. Defaults to all.
350-
* `bbox: (float, float, float, float)` - Bounding box in geographical
351-
coordinates.
445+
* `bbox: (float, float, float, float)` - Bounding box in geographical
446+
coordinates.
352447
* `time_range: (str, str)` - Time range.
353-
* `normalize_data: bool` - Whether to normalize and sanitize the data.
448+
* `normalize_data: bool` - Whether to normalize and sanitize the data.
354449
Defaults to `True`.
355450

356-
### ESA Climate Data Centre `ccizarr`
451+
#### `ccizarr`
357452

358-
A subset of the datasets of the `cciodp` store have been made available
359-
using the Zarr format using the data store `ccizarr`. It provides
360-
much better data access performance.
453+
A subset of the datasets of the `cciodp` store have been made available
454+
using the Zarr format using the data store `ccizarr`. It provides
455+
much better data access performance.
361456

362457
It has no dedicated data store parameters.
363458

364459
Its common dataset open parameters for opening [xarray.Dataset] instances are
365460
the same as for the filesystem-based data stores described above.
366461

367-
### ESA Climate Data Centre `esa-cci-kc`
462+
#### `esa-cci-kc`
368463

369464
The data store `esa-cci-kc` accesses datasets that are offered by the [Open Data Portal]
370465
via the references format.
@@ -374,7 +469,6 @@ Data store parameters are the same as for the filesystem-based `reference` store
374469
Its common dataset open parameters for opening [xarray.Dataset] instances are
375470
the same as for the filesystem-based data stores described above.
376471

377-
378472
### ESA SMOS `smos`
379473

380474
The data store `smos` provides L2C datasets of the [ESA Soil Moisture and Ocean Salinity]
@@ -386,65 +480,29 @@ This data store is provided by the xcube plugin [xcube-smos]. You can install it
386480
Data store parameters:
387481
* `source_path: str` - Path or URL into SMOS archive filesystem.
388482
* `source_protocol: str`: Protocol name for the SMOS archive filesystem.
389-
* `source_storage_options: dict`: Storage options for the SMOS archive filesystem. See
390-
fsspec documentation for specific filesystems. Any option can be overriden by
391-
passing it as additional data store parameter.
392-
* `cache_path: str`: Path to local cache directory. Must be given, if file caching
393-
is desired.
483+
* `source_storage_options: dict`: Storage options for the SMOS archive filesystem. See
484+
fsspec documentation for specific filesystems. Any option can be overriden by
485+
passing it as additional data store parameter.
486+
* `cache_path: str`: Path to local cache directory. Must be given, if file caching
487+
is desired.
394488
* `xarray_kwargs: dict`: Extra keyword arguments accepted by `xarray.open_dataset`.
395489

396490
Common parameters for opening [xarray.Dataset] instances:
397491

398-
* `time_range: (str, str)` - Time range given as pair of start and stop dates.
492+
* `time_range: (str, str)` - Time range given as pair of start and stop dates.
399493
Format: `YYYY-MM-DD`. Required.
400494
* `bbox: (float, float, float, float)` - Bounding box in geographical
401495
coordinates.
402-
* `res_level: int` - Spatial resolution level in the range 0 to 4. Zero refers to
496+
* `res_level: int` - Spatial resolution level in the range 0 to 4. Zero refers to
403497
the max resolution of 0.0439453125 degrees.
404498

405-
### Copernicus Climate Data Store `cds`
406-
407-
The data store `cds` provides datasets of the [Copernicus Climate Data Store].
408-
409-
This data store is provided by the xcube plugin [xcube-cds].
410-
You can install it using `conda install -c conda-forge xcube-cds`.
411-
412-
Data store parameters:
413-
414-
* `cds_api_key: str` - User API key for Copernicus Climate Data Store.
415-
* `endpoint_url: str` - API endpoint URL.
416-
* `num_retries: int` - Defaults to `200`.
417-
* `normalize_names: bool` - Defaults to `False`.
418-
419-
Common parameters for opening [xarray.Dataset] instances:
420-
421-
* `bbox: (float, float, float, float)` - Bounding box in geographical
422-
coordinates.
423-
* `time_range: (str, str)` - Time range.
424-
* `variable_names: list[str]` - List of names of variables to be included.
425-
Defaults to all.
426-
* `spatial_res: float` - Spatial resolution. Defaults to `0.1`.
427-
428-
### Copernicus Marine Service `cmems`
429-
430-
The data store `cmems` provides datasets of the [Copernicus Marine Service].
431-
432-
This data store is provided by the xcube plugin [xcube-cmems].
433-
You can install it using `conda install -c conda-forge xcube-cmems`.
434-
435-
Data store parameters:
436-
437-
* `cmems_username: str` - CMEMS API username
438-
* `cmems_password: str` - CMEMS API password
439-
* `cas_url: str` - Defaults to `'https://cmems-cas.cls.fr/cas/login'`.
440-
* `csw_url: str` - Defaults to `'https://cmems-catalog-ro.cls.fr/geonetwork/srv/eng/csw-MYOCEAN-CORE-PRODUCTS?'`.
441-
* `databases: str` - One of `['nrt', 'my']`.
442-
* `server: str` - Defaults to `'cmems-du.eu/thredds/dodsC/'`.
499+
### Global Ecosystem Dynamics Investigation `gedi`
443500

444-
Common parameters for opening [xarray.Dataset] instances:
501+
A data store for [Global Ecosystem Dynamics Investigation] (GEDI) data is currently
502+
under development and will be released soon.
445503

446-
* `variable_names: list[str]` - List of variable names.
447-
* `time_range: [str, str]` - Time range.
504+
This data store is provided by the xcube plugin [xcube-gedi]. Once available, you will
505+
be able to install it using `conda install -c conda-forge xcube-gedi`.
448506

449507
### Sentinel Hub API
450508

@@ -497,49 +555,57 @@ Common parameters for opening [xarray.Dataset] instances:
497555
catalogue query.
498556
* `max_cache_size: int` - Maximum chunk cache size in bytes.
499557

558+
### SpatioTemporal Asset Catalogs `stac`, `stac-xcube`, `stac-cdse`
500559

501-
### Copernicus Land Monitoring Service `clms`
560+
The data stores `stac`, `stac-xcube`, and `stac-cdse` provide datasets of
561+
the [SpatioTemporal Asset Catalogs].
502562

503-
The data store `clms` provides datasets of the [Copernicus Land Monitoring Service].
563+
The three data stores are provided by the xcube plugin [xcube-stac].
564+
You can install it using `conda install -c conda-forge xcube-stac.`
504565

505-
This data store is provided by the xcube plugin [xcube-clms].
506-
You can install it using `conda install -c conda-forge xcube-clms`.
507566

508-
Data store parameters:
567+
#### `stac`
568+
The data store `stac` provides datasets from a user-defined STAC API.
509569

510-
* `credentials: dict`: CLMS API credentials that can be obtained following the steps outlined
511-
[here](https://eea.github.io/clms-api-docs/authentication.html).
512-
These are the credentials parameters:
570+
The general parameters for the data store are the same as those used for `https`
571+
and `s3` stores. The STAC collection of interest determines whether data is accessed
572+
via `https` or `s3`. Specific parameters for this store are:
573+
* `url: str` - URL to STAC catalog. Required.
574+
* `stack_mode: bool` - Stacking of STAC items. Transforms data into analysis-ready
575+
format. Defaults to `False`.
513576

514-
* `client_id: str` - Required.
515-
* `issued: str`
516-
* `private_key: str` - Required.
517-
* `key_id: str`
518-
* `title: str`
519-
* `token_uri: str` - Required.
520-
* `user_id: str` - Required.
577+
#### `stac-xcube`
578+
The data store `stac-xcube` connects to STAC catalogs published on a `xcube` Server.
521579

522-
* `cache_store_id: str` - Store ID of cache data store. Defaults to `file`.
523-
* `cache_store_params: dict` - Store parameters of a filesystem-based data
524-
store.
580+
The general parameters for the data store are the same as those used for `https`
581+
and `s3` stores. The STAC collection of interest determines whether data is accessed
582+
via `https` or `s3`. Specific parameters for this store are:
583+
* `url: str` - URL to STAC catalog. Required.
584+
* `stack_mode: bool` - Stacking of STAC items. Transforms data into analysis-ready
585+
format. Defaults to `False`.
525586

526-
Before opening a specific dataset from CLMS, it's required to preload the data first.
527-
Preloading lets you create data requests ahead of time, which may sit in a queue before
528-
being processed. Once processed, the data is downloaded as zip files, unzipped,
529-
extracted to a cache, and prepared for use. After this, it can be accessed through the
530-
cache data store.
587+
#### `stac-cdse`
588+
The data store `stac-cdse` provides direct access to Sentinel data using the
589+
CSDE STAC API.
531590

532-
The preload parameters are:
533-
* `blocking: bool` - Switch to make the preloading process blocking or non-blocking.
534-
If True, the preloading process blocks the script. Defaults to `True`.
535-
* `silent: bool` - Silence the output of Preload API. If True, no preload state
536-
output is given. Defaults to `False`.
537-
* `cleanup: bool` - Cleanup the download directory before and after the
538-
preload job and the cache directory when preload_handle.close() is called.
539-
Defaults to `True`.
591+
The general parameters for the data store are the same as those used for `https`
592+
and `s3` stores. The STAC collection of interest determines whether data is accessed
593+
via `https` or `s3`. Specific parameters for this store are:
594+
* `stack_mode: bool` - Stacking of STAC items. Transforms data into analysis-ready
595+
format. Defaults to `False`.
540596

541-
Its common dataset open parameters for opening [xarray.Dataset] instances are the same
542-
as for the filesystem-based data stores described above.
597+
598+
There are no common parameters for opening datasets with the three stores.
599+
As the available datasets are varying across a wide spectrum of datatypes
600+
no specific opening parameters can be named here. The stores delegate to the
601+
general xcube Data Opener which offers a variety of parameters depending on the
602+
datatype of the dataset.
603+
604+
Use the following function to access the parameters fitting for the dataset of interest:
605+
606+
```python
607+
open_schema = store.get_open_data_params_schema("data_id")
608+
```
543609

544610
### Zenodo `zenodo`
545611

@@ -580,55 +646,6 @@ Use the following function to access the parameters fitting for the dataset of i
580646
open_schema = store.get_open_data_params_schema("data_id")
581647
```
582648

583-
### SpatioTemporal Asset Catalogs `stac`, `stac-xcube`, `stac-cdse`
584-
585-
The data stores `stac`, `stac-xcube`, and `stac-cdse` provide datasets of
586-
the [SpatioTemporal Asset Catalogs].
587-
588-
`stac` - General store that connects to a user-defined STAC API.
589-
`stac-xcube` - Connects to STAC catalogs published on a `xcube` Server.
590-
`stac-cse` - Provides direct access to Sentinel data using the CSDE STAC API.
591-
592-
This data store is provided by the xcube plugin [xcube-stac].
593-
You can install it using `conda install -c conda-forge xcube-stac.`
594-
595-
The general parameters for the data store are the same as those used for `https`
596-
and `s3` stores. The STAC collection of interest determines whether data is accessed
597-
via `https` or `s3`.
598-
599-
Specific parameters are:
600-
* `url: str` - URL to STAC catalog. Required for `stac` and `xcube-stac`
601-
* `stack_mode: bool` - Stacking of STAC items. Transforms data into analysis-ready
602-
format. Defaults to `False`.
603-
604-
605-
606-
There are no common parameters for opening datasets with the three stores.
607-
As the available datasets are varying across a wide spectrum of datatypes
608-
no specific opening parameters can be named here. The stores delegate to the
609-
general xcube Data Opener which offers a variety of parameters depending on the
610-
datatype of the dataset.
611-
612-
Use the following function to access the parameters fitting for the dataset of interest:
613-
614-
```python
615-
open_schema = store.get_open_data_params_schema("data_id")
616-
```
617-
618-
### Global Ecosystem Dynamics Investigation `gedi`
619-
620-
A data store for [Global Ecosystem Dynamics Investigation] (GEDI) data is currently
621-
under development and will be released soon.
622-
623-
This data store is provided by the xcube plugin [xcube-gedi]. Once available, you will
624-
be able to install it using `conda install -c conda-forge xcube-gedi`.
625-
626-
627-
### EOPF Sample Service
628-
629-
A data store for EOPF Sample Service is currently under development and will be released
630-
soon.
631-
632649

633650
## Developing new data stores
634651

0 commit comments

Comments
 (0)