Merge pull request #66 from TGSAI/docs/csp_connections

Add cloud service provider connection string documentation

Author: tasansal

README.md

@@ -94,13 +94,13 @@ For Python API please see the [API Reference][reference] for details.
 
 # Requirements
 
-### Minimal
+## Minimal
 
 Chunked storage and parallelization: `zarr`, `dask`, `numba`, and `psutil`.\
 SEG-Y Parsing: `segyio`\
 CLI and Progress Bars: `click`, `click-params`, and `tqdm`.
 
-### Optional
+## Optional
 
 Distributed computing `[distributed]`: `distributed` and `bokeh`.\
 Cloud Object Store I/O `[cloud]`: `s3fs`, `gcsfs`, and `adlfs`.\

docs/usage.md

@@ -1,11 +1,11 @@
 # Usage
 
-## Examples
+## Ingestion and Export
 
-The following example shows how to ingest a 3D seismic stack into
-MDIO format. Only one lossless copy will be made.
+The following example shows how to minimally ingest a 3D seismic stack into
+a local **_MDIO_** file. Only one lossless copy will be made.
 
-There are many more options, please see the [usage](#usage).
+There are many more options, please see the [CLI Reference](#cli-reference).
 
 ```shell
 $ mdio segy import \
@@ -24,6 +24,168 @@ $ mdio segy export \
     -o path_to_segy_file.segy
 ```
 
+## Cloud Connection Strings
+
+**_MDIO_** supports I/O on major cloud service providers. The cloud I/O capabilities are
+supported using the [fsspec](https://filesystem-spec.readthedocs.io/) and its specialized
+version for:
+
+- Amazon Web Services (AWS S3) - [s3fs](https://s3fs.readthedocs.io)
+- Google Cloud Provider (GCP GCS) - [gcsfs](https://gcsfs.readthedocs.io)
+- Microsoft Azure (Datalake Gen2) - [adlfs](https://github.com/fsspec/adlfs)
+
+Any other file-system supported by `fsspec` will also be supported by **_MDIO_**. However,
+we will focus on the major providers here.
+
+The protocols that help choose a backend (i.e. `s3://`, `gs://`, or `az://`) can be passed
+prepended to the **_MDIO_** path.
+
+The connection string can be passed to the command-line-interface (CLI) using the
+`-storage, --storage-options` flag as a JSON string or the Python API with the `storage_options`
+keyword argument as a Python dictionary.
+
+````{warning}
+On Windows clients, JSON strings are passed to the CLI with a special escape character.
+
+For instance a JSON string:
+```json
+{"key": "my_super_private_key", "secret": "my_super_private_secret"}
+```
+must be passed with an escape character `\` for inner quotes as:
+```shell
+"{\"key\": \"my_super_private_key\", \"secret\": \"my_super_private_secret\"}"
+```
+whereas, on Linux bash this works just fine:
+```shell
+'{"key": "my_super_private_key", "secret": "my_super_private_secret"}'
+```
+If this done incorrectly, you will get an invalid JSON string error from the CLI.
+````
+
+### Amazon Web Services
+
+Credentials can be automatically fetched from pre-authenticated AWS CLI.
+See [here](https://s3fs.readthedocs.io/en/latest/index.html#credentials) for the order `s3fs`
+checks them. If it is not pre-authenticated, you need to pass `--storage-options`.
+
+**Prefix:**
+`s3://`
+
+**Storage Options:**
+`key`: The auth key from AWS
+`secret`: The auth secret from AWS
+
+Using UNIX:
+
+```shell
+mdio segy import \
+  --input-segy-path path/to/my.segy
+  --output-mdio-file s3://bucket/prefix/my.mdio
+  --header-locations 189,193
+  --storage-options '{"key": "my_super_private_key", "secret": "my_super_private_secret"}'
+```
+
+Using Windows (note the extra escape characters `\`):
+
+```console
+mdio segy import \
+  --input-segy-path path/to/my.segy
+  --output-mdio-file s3://bucket/prefix/my.mdio
+  --header-locations 189,193
+  --storage-options "{\"key\": \"my_super_private_key\", \"secret\": \"my_super_private_secret\"}"
+```
+
+### Google Cloud Provider
+
+Credentials can be automatically fetched from pre-authenticated `gcloud` CLI.
+See [here](https://gcsfs.readthedocs.io/en/latest/#credentials) for the order `gcsfs`
+checks them. If it is not pre-authenticated, you need to pass `--storage-options`.
+
+GCP uses [service accounts](https://cloud.google.com/iam/docs/service-accounts) to pass
+authentication information to APIs.
+
+**Prefix:**
+`gs://` or `gcs://`
+
+**Storage Options:**
+`token`: The service account JSON value as string, or local path to JSON
+
+Using a service account:
+
+```shell
+mdio segy import \
+  --input-segy-path path/to/my.segy
+  --output-mdio-file gs://bucket/prefix/my.mdio
+  --header-locations 189,193
+  --storage-options '{"token": "~/.config/gcloud/application_default_credentials.json"}'
+```
+
+Using browser to populate authentication:
+
+```shell
+mdio segy import \
+  --input-segy-path path/to/my.segy
+  --output-mdio-file gs://bucket/prefix/my.mdio
+  --header-locations 189,193
+  --storage-options '{"token": "browser"}'
+```
+
+### Microsoft Azure
+
+There are various ways to authenticate with Azure Data Lake (ADL).
+See [here](https://github.com/fsspec/adlfs#details) for some details.
+If ADL is not pre-authenticated, you need to pass `--storage-options`.
+
+**Prefix:**
+`az://` or `abfs://`
+
+**Storage Options:**
+`account_name`: Azure Data Lake storage account name
+`account_key`: Azure Data Lake storage account access key
+
+```shell
+mdio segy import \
+  --input-segy-path path/to/my.segy
+  --output-mdio-file az://bucket/prefix/my.mdio
+  --header-locations 189,193
+  --storage-options '{"account_name": "myaccount", "account_key": "my_super_private_key"}'
+```
+
+### Advanced Cloud Features
+
+There are additional functions provided by `fsspec`. These are advanced features and we refer
+the user to read `fsspec` [documentation](https://filesystem-spec.readthedocs.io/en/latest/features.html).
+Some useful examples are:
+
+- Caching Files Locally
+- Remote Write Caching
+- File Buffering and random access
+- Mount anything with FUSE
+
+````{note}
+When combining advanced protocols like `simplecache` and using a remote store like `s3` the
+URL can be chained like `simplecache::s3://bucket/prefix/file.mdio`. When doing this the
+`--storage-options` argument must explicitly state parameters for the cloud backend and the
+extra protocol. For the above example it would look like this:
+
+```json
+{
+  "s3": {
+    "key": "my_super_private_key",
+    "secret": "my_super_private_secret"
+  },
+  "simplecache": {
+    "cache_storage": "/custom/temp/storage/path"
+  }
+}
+```
+
+In one line:
+```json
+{"s3": {"key": "my_super_private_key", "secret": "my_super_private_secret"}, "simplecache": {"cache_storage": "/custom/temp/storage/path"}
+```
+````
+
 ## CLI Reference
 
 MDIO provides a convenient command-line-interface (CLI) to do