Merge branch 'main' into numpy_ingestion

Author: BrianMichell

docs/configuration.md

@@ -0,0 +1,201 @@
+```{eval-rst}
+:tocdepth: 2
+```
+
+```{currentModule} mdio.core.config
+
+```
+
+# Configuration
+
+MDIO can be configured using environment variables to customize behavior for import, export,
+and validation operations. These variables provide runtime control without requiring code changes.
+You can find a summary of the available variables and their defaults below.
+
+| **Variable**                          | **Type** | **Default**                      |
+| ------------------------------------- | -------- | -------------------------------- |
+| `MDIO__IMPORT__CPU_COUNT`             | `int`    | Number of logical CPUs available |
+| `MDIO__EXPORT__CPU_COUNT`             | `int`    | Number of logical CPUs available |
+| `MDIO__GRID__SPARSITY_RATIO_WARN`     | `float`  | `2.0`                            |
+| `MDIO__GRID__SPARSITY_RATIO_LIMIT`    | `float`  | `10.0`                           |
+| `MDIO__IMPORT__SAVE_SEGY_FILE_HEADER` | `bool`   | `False`                          |
+| `MDIO__IMPORT__CLOUD_NATIVE`          | `bool`   | `False`                          |
+| `MDIO__IMPORT__RAW_HEADERS`           | `bool`   | `False`                          |
+| `MDIO_IGNORE_CHECKS`                  | `bool`   | `False`                          |
+
+## CPU and Performance
+
+### `MDIO__EXPORT__CPU_COUNT`
+
+Controls the number of CPUs used during SEG-Y export operations. Adjust this to balance
+performance with system resource availability.
+
+```shell
+$ export MDIO__EXPORT__CPU_COUNT=8
+$ mdio segy export input.mdio output.segy
+```
+
+### `MDIO__IMPORT__CPU_COUNT`
+
+Controls the number of CPUs used during SEG-Y import operations. Higher values can
+significantly speed up ingestion of large datasets.
+
+```shell
+$ export MDIO__IMPORT__CPU_COUNT=16
+$ mdio segy import input.segy output.mdio --header-locations 189,193
+```
+
+## Grid Validation
+
+### `MDIO__GRID__SPARSITY_RATIO_WARN`
+
+Sparsity ratio threshold that triggers warnings during grid validation. The sparsity ratio
+measures how sparse the trace grid is compared to a dense grid. Values above this threshold
+will log warnings but won't prevent operations.
+
+```shell
+$ export MDIO__GRID__SPARSITY_RATIO_WARN=3.0
+```
+
+### `MDIO__GRID__SPARSITY_RATIO_LIMIT`
+
+Sparsity ratio threshold that triggers errors and prevents operations. Use this to enforce
+quality standards and prevent ingestion of excessively sparse datasets that may indicate
+data quality issues.
+
+```shell
+$ export MDIO__GRID__SPARSITY_RATIO_LIMIT=15.0
+```
+
+## SEG-Y Processing
+
+### `MDIO__IMPORT__SAVE_SEGY_FILE_HEADER`
+
+**Accepted values:** `true`, `false`, `1`, `0`, `yes`, `no`, `on`, `off`
+
+When enabled, preserves the original SEG-Y textual file header during import.
+This is useful for maintaining full SEG-Y standard compliance and preserving survey metadata.
+
+```shell
+$ export MDIO__IMPORT__SAVE_SEGY_FILE_HEADER=true
+$ mdio segy import input.segy output.mdio --header-locations 189,193
+```
+
+### `MDIO__IMPORT__CLOUD_NATIVE`
+
+**Accepted values:** `true`, `false`, `1`, `0`, `yes`, `no`, `on`, `off`
+
+Enables buffered reads during SEG-Y header scans to optimize performance when reading from or
+writing to cloud object storage (S3, GCS, Azure). This mode balances bandwidth usage with read
+latency by processing the file twice: first to determine optimal buffering, then to perform the
+actual ingestion.
+
+```{note}
+This variable is designed for cloud storage I/O, regardless of where the compute is running.
+```
+
+**When to use:**
+
+- Reading from cloud storage (e.g., `s3://bucket/input.segy`)
+- Writing to cloud storage (e.g., `gs://bucket/output.mdio`)
+
+**When to skip:**
+
+- Local file paths on fast storage
+- Very slow network connections where bandwidth is the primary bottleneck
+
+```shell
+$ export MDIO__IMPORT__CLOUD_NATIVE=true
+$ mdio segy import s3://bucket/input.segy output.mdio --header-locations 189,193
+```
+
+## Development and Testing
+
+### `MDIO_IGNORE_CHECKS`
+
+**Accepted values:** `true`, `false`, `1`, `0`, `yes`, `no`, `on`, `off`
+
+Bypasses validation checks during MDIO operations. This is primarily intended for development,
+testing, or debugging scenarios where you need to work with non-standard data.
+
+```{warning}
+Disabling checks can lead to corrupted output or unexpected behavior. Only use this
+when you understand the implications and are working in a controlled environment.
+```
+
+```shell
+$ export MDIO_IGNORE_CHECKS=true
+$ mdio segy import input.segy output.mdio --header-locations 189,193
+```
+
+## Deprecated Features
+
+### `MDIO__IMPORT__RAW_HEADERS`
+
+**Accepted values:** `true`, `false`, `1`, `0`, `yes`, `no`, `on`, `off`
+
+```{warning}
+This is a deprecated feature and is expected to be removed without warning in a future release.
+```
+
+## Configuration Best Practices
+
+### Setting Multiple Variables
+
+You can configure multiple environment variables at once:
+
+```shell
+# Set for current session
+export MDIO__IMPORT__CPU_COUNT=16
+export MDIO__GRID__SPARSITY_RATIO_LIMIT=15.0
+export MDIO__IMPORT__CLOUD_NATIVE=true
+
+# Run MDIO commands
+mdio segy import input.segy output.mdio --header-locations 189,193
+```
+
+### Persistent Configuration
+
+To make environment variables permanent, add them to your shell profile:
+
+**Bash/Zsh:**
+
+```shell
+# Add to ~/.bashrc or ~/.zshrc
+export MDIO__IMPORT__CPU_COUNT=16
+export MDIO__IMPORT__CLOUD_NATIVE=true
+```
+
+**Windows:**
+
+```console
+# Set permanently in PowerShell (run as Administrator)
+[System.Environment]::SetEnvironmentVariable('MDIO__IMPORT__CPU_COUNT', '16', 'User')
+```
+
+### Project-Specific Configuration
+
+For project-specific settings, use a `.env` file with tools like `python-dotenv`:
+
+```python
+# example_import.py
+from dotenv import load_dotenv
+import mdio
+
+load_dotenv()  # Load environment variables from .env file
+# Your MDIO operations here
+```
+
+```shell
+# .env file
+MDIO__IMPORT__CPU_COUNT=16
+MDIO__GRID__SPARSITY_RATIO_LIMIT=15.0
+MDIO__IMPORT__CLOUD_NATIVE=true
+```
+
+## Reference
+
+```{eval-rst}
+.. autopydantic_model:: MDIOSettings
+    :inherited-members: BaseSettings
+```

docs/index.md

@@ -16,6 +16,7 @@ end-before: <!-- github-only -->
 
 installation
 cli_usage
+configuration
 ```
 
 ```{toctree}

docs/tutorials/custom_template.ipynb

@@ -5,7 +5,7 @@
    "id": "85114119ae7a4db0",
    "metadata": {},
    "source": [
-    "# Create and Register a Custom Template\n",
+    "# MDIO Template Usage\n",
     "\n",
     "```{article-info}\n",
     ":author: Altay Sansal\n",

pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "multidimio"
-version = "1.0.8"
+version = "1.0.9"
 description = "Cloud-native, scalable, and user-friendly multi dimensional energy data!"
 authors = [{ name = "Altay Sansal", email = "[email protected]" }]
 requires-python = ">=3.11,<3.14"
@@ -25,6 +25,7 @@ dependencies = [
     "pint>=0.25.0",
     "psutil>=7.1.0",
     "pydantic>=2.12.0",
+    "pydantic-settings>=2.6.1",
     "rich>=14.1.0",
     "segy>=0.5.3",
     "tqdm>=4.67.1",
@@ -182,7 +183,7 @@ init_typed = true
 warn_required_dynamic_aliases = true
 
 [tool.bumpversion]
-current_version = "1.0.8"
+current_version = "1.0.9"
 allow_dirty = true
 commit = false
 tag = false

src/mdio/builder/schemas/v1/units.py

@@ -59,7 +59,7 @@ class DensityUnitEnum(UnitEnum):
 class SpeedUnitEnum(UnitEnum):
     """Enum class representing units of speed."""
 
-    METER_PER_SECOND = ureg.meter / ureg.second
+    METERS_PER_SECOND = ureg.meter / ureg.second
     FEET_PER_SECOND = ureg.feet / ureg.second
 
 

src/mdio/builder/template_registry.py

@@ -20,13 +20,13 @@
 from typing import TYPE_CHECKING
 
 from mdio.builder.formatting_html import template_registry_repr_html
+from mdio.builder.templates.seismic_2d_cdp import Seismic2DCdpGathersTemplate
 from mdio.builder.templates.seismic_2d_poststack import Seismic2DPostStackTemplate
-from mdio.builder.templates.seismic_2d_prestack_cdp import Seismic2DPreStackCDPTemplate
-from mdio.builder.templates.seismic_2d_prestack_shot import Seismic2DPreStackShotTemplate
+from mdio.builder.templates.seismic_2d_streamer_shot import Seismic2DStreamerShotGathersTemplate
+from mdio.builder.templates.seismic_3d_cdp import Seismic3DCdpGathersTemplate
+from mdio.builder.templates.seismic_3d_coca import Seismic3DCocaGathersTemplate
 from mdio.builder.templates.seismic_3d_poststack import Seismic3DPostStackTemplate
-from mdio.builder.templates.seismic_3d_prestack_cdp import Seismic3DPreStackCDPTemplate
-from mdio.builder.templates.seismic_3d_prestack_coca import Seismic3DPreStackCocaTemplate
-from mdio.builder.templates.seismic_3d_prestack_shot import Seismic3DPreStackShotTemplate
+from mdio.builder.templates.seismic_3d_streamer_shot import Seismic3DStreamerShotGathersTemplate
 
 if TYPE_CHECKING:
     from mdio.builder.templates.base import AbstractDatasetTemplate
@@ -126,15 +126,15 @@ def _register_default_templates(self) -> None:
         # CDP/CMP Ordered Data
         for data_domain in ("time", "depth"):
             for gather_domain in ("offset", "angle"):
-                self.register(Seismic3DPreStackCDPTemplate(data_domain, gather_domain))
-                self.register(Seismic2DPreStackCDPTemplate(data_domain, gather_domain))
+                self.register(Seismic3DCdpGathersTemplate(data_domain, gather_domain))
+                self.register(Seismic2DCdpGathersTemplate(data_domain, gather_domain))
 
-        self.register(Seismic3DPreStackCocaTemplate("time"))
-        self.register(Seismic3DPreStackCocaTemplate("depth"))
+        self.register(Seismic3DCocaGathersTemplate("time"))
+        self.register(Seismic3DCocaGathersTemplate("depth"))
 
         # Field (shot) data
-        self.register(Seismic2DPreStackShotTemplate("time"))
-        self.register(Seismic3DPreStackShotTemplate("time"))
+        self.register(Seismic2DStreamerShotGathersTemplate())
+        self.register(Seismic3DStreamerShotGathersTemplate())
 
     def get(self, template_name: str) -> AbstractDatasetTemplate:
         """Get an instance of a template from the registry by its name.

src/mdio/builder/templates/base.py

@@ -148,14 +148,29 @@ def coordinate_names(self) -> tuple[str, ...]:
     @property
     def full_chunk_shape(self) -> tuple[int, ...]:
         """Returns the chunk shape for the variables."""
-        return copy.deepcopy(self._var_chunk_shape)
+        # If dimension sizes are not set yet, return the stored shape as-is
+        if len(self._dim_sizes) != len(self._dim_names):
+            return self._var_chunk_shape
+
+        # Expand -1 values to full dimension sizes
+        return tuple(
+            dim_size if chunk_size == -1 else chunk_size
+            for chunk_size, dim_size in zip(self._var_chunk_shape, self._dim_sizes, strict=False)
+        )
 
     @full_chunk_shape.setter
     def full_chunk_shape(self, shape: tuple[int, ...]) -> None:
         """Sets the chunk shape for the variables."""
-        if len(shape) != len(self._dim_sizes):
-            msg = f"Chunk shape {shape} does not match dimension sizes {self._dim_sizes}"
+        if len(shape) != len(self._dim_names):
+            msg = f"Chunk shape {shape} has {len(shape)} dimensions, expected {len(self._dim_names)}"
             raise ValueError(msg)
+
+        # Validate that all values are positive integers or -1
+        for chunk_size in shape:
+            if chunk_size != -1 and chunk_size <= 0:
+                msg = f"Chunk size must be positive integer or -1, got {chunk_size}"
+                raise ValueError(msg)
+
         self._var_chunk_shape = shape
 
     @property

src/mdio/builder/templates/seismic_2d_prestack_cdp.py renamed to src/mdio/builder/templates/seismic_2d_cdp.py

@@ -1,4 +1,4 @@
-"""Seismic2DPreStackCDPTemplate MDIO v1 dataset templates."""
+"""Seismic2DCDPGathersTemplate MDIO v1 dataset templates."""
 
 from typing import Any
 
@@ -7,7 +7,7 @@
 from mdio.builder.templates.types import SeismicDataDomain
 
 
-class Seismic2DPreStackCDPTemplate(AbstractDatasetTemplate):
+class Seismic2DCdpGathersTemplate(AbstractDatasetTemplate):
     """Seismic CDP pre-stack 2D time or depth Dataset template."""
 
     def __init__(self, data_domain: SeismicDataDomain, gather_domain: CdpGatherDomain):
@@ -26,7 +26,7 @@ def __init__(self, data_domain: SeismicDataDomain, gather_domain: CdpGatherDomai
     def _name(self) -> str:
         gather_domain_suffix = self._gather_domain.capitalize()
         data_domain_suffix = self._data_domain.capitalize()
-        return f"PreStackCdp{gather_domain_suffix}Gathers2D{data_domain_suffix}"
+        return f"Cdp{gather_domain_suffix}Gathers2D{data_domain_suffix}"
 
     def _load_dataset_attributes(self) -> dict[str, Any]:
         return {"surveyType": "2D", "gatherType": "cdp"}

src/mdio/builder/templates/seismic_2d_prestack_shot.py renamed to src/mdio/builder/templates/seismic_2d_streamer_shot.py

@@ -1,4 +1,4 @@
-"""Seismic2DPreStackShotTemplate MDIO v1 dataset templates."""
+"""Seismic2DStreamerShotGathersTemplate MDIO v1 dataset templates."""
 
 from typing import Any
 
@@ -9,10 +9,10 @@
 from mdio.builder.templates.types import SeismicDataDomain
 
 
-class Seismic2DPreStackShotTemplate(AbstractDatasetTemplate):
+class Seismic2DStreamerShotGathersTemplate(AbstractDatasetTemplate):
     """Seismic Shot pre-stack 2D time or depth Dataset template."""
 
-    def __init__(self, data_domain: SeismicDataDomain):
+    def __init__(self, data_domain: SeismicDataDomain = "time"):
         super().__init__(data_domain=data_domain)
 
         self._dim_names = ("shot_point", "channel", self._data_domain)
@@ -21,7 +21,7 @@ def __init__(self, data_domain: SeismicDataDomain):
 
     @property
     def _name(self) -> str:
-        return f"PreStackShotGathers2D{self._data_domain.capitalize()}"
+        return "StreamerShotGathers2D"
 
     def _load_dataset_attributes(self) -> dict[str, Any]:
         return {"surveyType": "2D", "ensembleType": "common_source"}