Internal environment variable handling and documentation (TGSAI#730)

* Add isolated environment variable getters

* Fix pre-commit issues

* Remove unnecessary intermediate variable

* Refactor environment managmenet to expose basic functions for lighter imports

* Add documentation page on environment variables

* pre-commit

* Update to use pydantic-settings

* Remove pre-v1 special case environment variable

* Remove MDIO__SEGY__SPEC from docs

* Remove helper functions in favor of direct attribute access and pydantic validation

* Update to instantiate MDIOSettings object at beginning of functions for easy access

* Fix pre-commit

* move settings to core

* remove extra boolean parsing because pydantic handles it.

* rename env to settings

* remove validation tests since pydantic already guarantees that.

* consistent version string for pydantic-settings

* rename settings to config

* rename config and add reference to Pydantic Settings object

* refactor doc page

* reorder tables

---------

Co-authored-by: Altay Sansal <[email protected]>

Author: BrianMichell

docs/configuration.md

@@ -0,0 +1,201 @@
+```{eval-rst}
+:tocdepth: 2
+```
+
+```{currentModule} mdio.core.config
+
+```
+
+# Configuration
+
+MDIO can be configured using environment variables to customize behavior for import, export,
+and validation operations. These variables provide runtime control without requiring code changes.
+You can find a summary of the available variables and their defaults below.
+
+| **Variable**                          | **Type** | **Default**                      |
+| ------------------------------------- | -------- | -------------------------------- |
+| `MDIO__IMPORT__CPU_COUNT`             | `int`    | Number of logical CPUs available |
+| `MDIO__EXPORT__CPU_COUNT`             | `int`    | Number of logical CPUs available |
+| `MDIO__GRID__SPARSITY_RATIO_WARN`     | `float`  | `2.0`                            |
+| `MDIO__GRID__SPARSITY_RATIO_LIMIT`    | `float`  | `10.0`                           |
+| `MDIO__IMPORT__SAVE_SEGY_FILE_HEADER` | `bool`   | `False`                          |
+| `MDIO__IMPORT__CLOUD_NATIVE`          | `bool`   | `False`                          |
+| `MDIO__IMPORT__RAW_HEADERS`           | `bool`   | `False`                          |
+| `MDIO_IGNORE_CHECKS`                  | `bool`   | `False`                          |
+
+## CPU and Performance
+
+### `MDIO__EXPORT__CPU_COUNT`
+
+Controls the number of CPUs used during SEG-Y export operations. Adjust this to balance
+performance with system resource availability.
+
+```shell
+$ export MDIO__EXPORT__CPU_COUNT=8
+$ mdio segy export input.mdio output.segy
+```
+
+### `MDIO__IMPORT__CPU_COUNT`
+
+Controls the number of CPUs used during SEG-Y import operations. Higher values can
+significantly speed up ingestion of large datasets.
+
+```shell
+$ export MDIO__IMPORT__CPU_COUNT=16
+$ mdio segy import input.segy output.mdio --header-locations 189,193
+```
+
+## Grid Validation
+
+### `MDIO__GRID__SPARSITY_RATIO_WARN`
+
+Sparsity ratio threshold that triggers warnings during grid validation. The sparsity ratio
+measures how sparse the trace grid is compared to a dense grid. Values above this threshold
+will log warnings but won't prevent operations.
+
+```shell
+$ export MDIO__GRID__SPARSITY_RATIO_WARN=3.0
+```
+
+### `MDIO__GRID__SPARSITY_RATIO_LIMIT`
+
+Sparsity ratio threshold that triggers errors and prevents operations. Use this to enforce
+quality standards and prevent ingestion of excessively sparse datasets that may indicate
+data quality issues.
+
+```shell
+$ export MDIO__GRID__SPARSITY_RATIO_LIMIT=15.0
+```
+
+## SEG-Y Processing
+
+### `MDIO__IMPORT__SAVE_SEGY_FILE_HEADER`
+
+**Accepted values:** `true`, `false`, `1`, `0`, `yes`, `no`, `on`, `off`
+
+When enabled, preserves the original SEG-Y textual file header during import.
+This is useful for maintaining full SEG-Y standard compliance and preserving survey metadata.
+
+```shell
+$ export MDIO__IMPORT__SAVE_SEGY_FILE_HEADER=true
+$ mdio segy import input.segy output.mdio --header-locations 189,193
+```
+
+### `MDIO__IMPORT__CLOUD_NATIVE`
+
+**Accepted values:** `true`, `false`, `1`, `0`, `yes`, `no`, `on`, `off`
+
+Enables buffered reads during SEG-Y header scans to optimize performance when reading from or
+writing to cloud object storage (S3, GCS, Azure). This mode balances bandwidth usage with read
+latency by processing the file twice: first to determine optimal buffering, then to perform the
+actual ingestion.
+
+```{note}
+This variable is designed for cloud storage I/O, regardless of where the compute is running.
+```
+
+**When to use:**
+
+- Reading from cloud storage (e.g., `s3://bucket/input.segy`)
+- Writing to cloud storage (e.g., `gs://bucket/output.mdio`)
+
+**When to skip:**
+
+- Local file paths on fast storage
+- Very slow network connections where bandwidth is the primary bottleneck
+
+```shell
+$ export MDIO__IMPORT__CLOUD_NATIVE=true
+$ mdio segy import s3://bucket/input.segy output.mdio --header-locations 189,193
+```
+
+## Development and Testing
+
+### `MDIO_IGNORE_CHECKS`
+
+**Accepted values:** `true`, `false`, `1`, `0`, `yes`, `no`, `on`, `off`
+
+Bypasses validation checks during MDIO operations. This is primarily intended for development,
+testing, or debugging scenarios where you need to work with non-standard data.
+
+```{warning}
+Disabling checks can lead to corrupted output or unexpected behavior. Only use this
+when you understand the implications and are working in a controlled environment.
+```
+
+```shell
+$ export MDIO_IGNORE_CHECKS=true
+$ mdio segy import input.segy output.mdio --header-locations 189,193
+```
+
+## Deprecated Features
+
+### `MDIO__IMPORT__RAW_HEADERS`
+
+**Accepted values:** `true`, `false`, `1`, `0`, `yes`, `no`, `on`, `off`
+
+```{warning}
+This is a deprecated feature and is expected to be removed without warning in a future release.
+```
+
+## Configuration Best Practices
+
+### Setting Multiple Variables
+
+You can configure multiple environment variables at once:
+
+```shell
+# Set for current session
+export MDIO__IMPORT__CPU_COUNT=16
+export MDIO__GRID__SPARSITY_RATIO_LIMIT=15.0
+export MDIO__IMPORT__CLOUD_NATIVE=true
+
+# Run MDIO commands
+mdio segy import input.segy output.mdio --header-locations 189,193
+```
+
+### Persistent Configuration
+
+To make environment variables permanent, add them to your shell profile:
+
+**Bash/Zsh:**
+
+```shell
+# Add to ~/.bashrc or ~/.zshrc
+export MDIO__IMPORT__CPU_COUNT=16
+export MDIO__IMPORT__CLOUD_NATIVE=true
+```
+
+**Windows:**
+
+```console
+# Set permanently in PowerShell (run as Administrator)
+[System.Environment]::SetEnvironmentVariable('MDIO__IMPORT__CPU_COUNT', '16', 'User')
+```
+
+### Project-Specific Configuration
+
+For project-specific settings, use a `.env` file with tools like `python-dotenv`:
+
+```python
+# example_import.py
+from dotenv import load_dotenv
+import mdio
+
+load_dotenv()  # Load environment variables from .env file
+# Your MDIO operations here
+```
+
+```shell
+# .env file
+MDIO__IMPORT__CPU_COUNT=16
+MDIO__GRID__SPARSITY_RATIO_LIMIT=15.0
+MDIO__IMPORT__CLOUD_NATIVE=true
+```
+
+## Reference
+
+```{eval-rst}
+.. autopydantic_model:: MDIOSettings
+    :inherited-members: BaseSettings
+```

docs/index.md

@@ -16,6 +16,7 @@ end-before: <!-- github-only -->
 
 installation
 cli_usage
+configuration
 ```
 
 ```{toctree}

pyproject.toml

@@ -25,6 +25,7 @@ dependencies = [
     "pint>=0.25.0",
     "psutil>=7.1.0",
     "pydantic>=2.12.0",
+    "pydantic-settings>=2.6.1",
     "rich>=14.1.0",
     "segy>=0.5.3",
     "tqdm>=4.67.1",

src/mdio/converters/mdio.py

@@ -2,16 +2,15 @@
 
 from __future__ import annotations
 
-import os
 from tempfile import TemporaryDirectory
 from typing import TYPE_CHECKING
 
 import numpy as np
-from psutil import cpu_count
 from tqdm.dask import TqdmCallback
 
 from mdio.api.io import _normalize_path
 from mdio.api.io import open_mdio
+from mdio.core.config import MDIOSettings
 from mdio.segy.blocked_io import to_segy
 from mdio.segy.creation import concat_files
 from mdio.segy.creation import mdio_spec_to_segy
@@ -29,10 +28,6 @@
     from upath import UPath
 
 
-default_cpus = cpu_count(logical=True)
-NUM_CPUS = int(os.getenv("MDIO__EXPORT__CPU_COUNT", default_cpus))
-
-
 def mdio_to_segy(  # noqa: PLR0912, PLR0913, PLR0915
     segy_spec: SegySpec,
     input_path: UPath | Path | str,
@@ -73,6 +68,8 @@ def mdio_to_segy(  # noqa: PLR0912, PLR0913, PLR0915
         >>> output_path = UPath("prefix/file.segy")
         >>> mdio_to_segy(input_path, output_path)
     """
+    settings = MDIOSettings()
+
     input_path = _normalize_path(input_path)
     output_path = _normalize_path(output_path)
 
@@ -148,7 +145,7 @@ def mdio_to_segy(  # noqa: PLR0912, PLR0913, PLR0915
             if client is not None:
                 block_records = block_records.compute()
             else:
-                block_records = block_records.compute(num_workers=NUM_CPUS)
+                block_records = block_records.compute(num_workers=settings.export_cpus)
 
         ordered_files = [rec.path for rec in block_records.ravel() if rec != 0]
         ordered_files = [output_path] + ordered_files

src/mdio/converters/segy.py

@@ -4,7 +4,6 @@
 
 import base64
 import logging
-import os
 from typing import TYPE_CHECKING
 
 import numpy as np
@@ -28,10 +27,10 @@
 from mdio.builder.schemas.v1.variable import VariableMetadata
 from mdio.builder.xarray_builder import to_xarray_dataset
 from mdio.constants import ZarrFormat
-from mdio.converters.exceptions import EnvironmentFormatError
 from mdio.converters.exceptions import GridTraceCountError
 from mdio.converters.exceptions import GridTraceSparsityError
 from mdio.converters.type_converter import to_structured_type
+from mdio.core.config import MDIOSettings
 from mdio.core.grid import Grid
 from mdio.core.utils_write import MAX_COORDINATES_BYTES
 from mdio.core.utils_write import MAX_SIZE_LIVE_MASK
@@ -92,30 +91,18 @@ def grid_density_qc(grid: Grid, num_traces: int) -> None:
     Raises:
         GridTraceSparsityError: If the sparsity ratio exceeds `MDIO__GRID__SPARSITY_RATIO_LIMIT`
             and `MDIO_IGNORE_CHECKS` is not set to a truthy value (e.g., "1", "true").
-        EnvironmentFormatError: If `MDIO__GRID__SPARSITY_RATIO_WARN` or
-            `MDIO__GRID__SPARSITY_RATIO_LIMIT` cannot be converted to a float.
     """
+    settings = MDIOSettings()
     # Calculate total possible traces in the grid (excluding sample dimension)
     grid_traces = np.prod(grid.shape[:-1], dtype=np.uint64)
 
     # Handle division by zero if num_traces is 0
     sparsity_ratio = float("inf") if num_traces == 0 else grid_traces / num_traces
 
     # Fetch and validate environment variables
-    warning_ratio_env = os.getenv("MDIO__GRID__SPARSITY_RATIO_WARN", "2")
-    error_ratio_env = os.getenv("MDIO__GRID__SPARSITY_RATIO_LIMIT", "10")
-    ignore_checks_env = os.getenv("MDIO_IGNORE_CHECKS", "false").lower()
-    ignore_checks = ignore_checks_env in ("1", "true", "yes", "on")
-
-    try:
-        warning_ratio = float(warning_ratio_env)
-    except ValueError as e:
-        raise EnvironmentFormatError("MDIO__GRID__SPARSITY_RATIO_WARN", "float") from e  # noqa: EM101
-
-    try:
-        error_ratio = float(error_ratio_env)
-    except ValueError as e:
-        raise EnvironmentFormatError("MDIO__GRID__SPARSITY_RATIO_LIMIT", "float") from e  # noqa: EM101
+    warning_ratio = settings.grid_sparsity_ratio_warn
+    error_ratio = settings.grid_sparsity_ratio_limit
+    ignore_checks = settings.ignore_checks
 
     # Check sparsity
     should_warn = sparsity_ratio > warning_ratio
@@ -373,8 +360,9 @@ def _populate_coordinates(
 
 
 def _add_segy_file_headers(xr_dataset: xr_Dataset, segy_file_info: SegyFileInfo) -> xr_Dataset:
-    save_file_header = os.getenv("MDIO__IMPORT__SAVE_SEGY_FILE_HEADER", "") in ("1", "true", "yes", "on")
-    if not save_file_header:
+    settings = MDIOSettings()
+
+    if not settings.save_segy_file_header:
         return xr_dataset
 
     expected_rows = 40
@@ -398,7 +386,7 @@ def _add_segy_file_headers(xr_dataset: xr_Dataset, segy_file_info: SegyFileInfo)
             "binaryHeader": segy_file_info.binary_header_dict,
         }
     )
-    if os.getenv("MDIO__IMPORT__RAW_HEADERS") in ("1", "true", "yes", "on"):
+    if settings.raw_headers:
         raw_binary_base64 = base64.b64encode(segy_file_info.raw_binary_headers).decode("ascii")
         xr_dataset["segy_file_header"].attrs.update({"rawBinaryHeader": raw_binary_base64})
 
@@ -536,6 +524,8 @@ def segy_to_mdio(  # noqa PLR0913
     Raises:
         FileExistsError: If the output location already exists and overwrite is False.
     """
+    settings = MDIOSettings()
+
     _validate_spec_in_template(segy_spec, mdio_template)
 
     input_path = _normalize_path(input_path)
@@ -565,7 +555,7 @@ def segy_to_mdio(  # noqa PLR0913
     _, non_dim_coords = _get_coordinates(grid, segy_headers, mdio_template)
     header_dtype = to_structured_type(segy_spec.trace.header.dtype)
 
-    if os.getenv("MDIO__IMPORT__RAW_HEADERS") in ("1", "true", "yes", "on"):
+    if settings.raw_headers:
         if zarr.config.get("default_zarr_format") == ZarrFormat.V2:
             logger.warning("Raw headers are only supported for Zarr v3. Skipping raw headers.")
         else: