docs: clarify Rasteret's custom IO layer and obstore's transport role

  obstore is the HTTP transport for multi-cloud URL routing (S3/GCS/Azure),
  not the source of read performance. Performance comes from the index-first
  approach: pre-cached tile offsets in Parquet, no header round-trips, and
  asyncio concurrency across scenes and bands.

  Updated: design-decisions, architecture, benchmark, custom-cloud-provider,
  changelog, notebooks/README, COGReader docstring.

  Signed-off-by: print-sid8 sidsub94@gmail.com

Author: print-sid8

README.md

@@ -25,19 +25,21 @@ Rasteret parses those headers **once**, caches them in Parquet, and its
 own reader fetches pixels concurrently with no GDAL in the path.
 **Up to 20x faster** on cold starts.
 
-We call this pattern **index-first geospatial image retrieval**:
+Rasteret calls this pattern **index-first geospatial retrieval**:
 
 - **Control plane**: a queryable Parquet index (scene metadata, COG header metadata, user columns like splits/labels)
 - **Data plane**: on-demand tile reads from the original GeoTIFF/COG objects
 
 This keeps metadata and experiment logic in tables while leaving imagery bytes in source COGs.
 
+Key Features - 
 - **Easy** - three lines from STAC search or Parquet file to a TorchGeo-compatible dataset
-- **Zero downloads** - work with terabytes of imagery while storing only megabytes of metadata
-- **No STAC at training time** - query once at setup; zero API calls during training
-- **Reproducible** - same Parquet index = same records = same results
-- **Native dtypes** - uint16 stays uint16 in tensors; xarray promotes only when NaN fill requires it
-- **Shareable cache** - a few MB index can capture scene selection, band metadata, and split assignments
+- **20x faster, saves cloud LISTs and GETs** - Our custom IO gets chunks of images fast, and costs no overhead a Collection is built  
+- **Zero data downloads** - work with terabytes of imagery while storing only megabytes of metadata.
+- **No STAC at training time** - query once at setup; zero API calls during training with Collection you can extend.
+- **Reproducible** - same Parquet index = same records = same results 
+- **Native dtypes** - In our IO image chunks of uint16 stays uint16 in tensors; only xarray conversion promotes to float32 to fill NaNs
+- **Shareable cache** - enrich our Collection with your ML splits, patch geometries, custom data points for ML, and share it, don't write folders of image chips!
 
 Rasteret is an **opt-in accelerator** that integrates with TorchGeo by
 returning a standard `GeoDataset`. Your samplers, DataLoader, xarray
@@ -85,6 +87,7 @@ Rasteret ships with a growing catalog of datasets. Pick an ID and go:
 ```
 $ rasteret datasets list
 ID                          Name                                       Coverage       License              Auth
+aef/v1-annual               AlphaEarth Foundation Embeddings (Annual)  global         CC-BY-4.0            none
 earthsearch/sentinel-2-l2a  Sentinel-2 Level-2A                        global         proprietary(free)    none
 earthsearch/landsat-c2-l2   Landsat Collection 2 Level-2               global         proprietary(free)    required
 earthsearch/naip            NAIP                                       north-america  proprietary(free)    required
@@ -96,7 +99,6 @@ pc/alos-dem                 ALOS World 3D 30m DEM                      global
 pc/nasadem                  NASADEM                                    global         proprietary(free)    required
 pc/esa-worldcover           ESA WorldCover                             global         CC-BY-4.0            required
 pc/usda-cdl                 USDA Cropland Data Layer                   conus          proprietary(free)    required
-aef/v1-annual               AlphaEarth Foundation Embeddings (Annual)  global         CC-BY-4.0            none
 ```
 
 Each entry includes license metadata and a `commercial_use` flag for quick
@@ -326,7 +328,7 @@ the full GeoDataset contract:
 - Works with `RandomGeoSampler`, `GridGeoSampler`, and any custom sampler
 - Works with `IntersectionDataset` and `UnionDataset` for dataset composition
 
-Rasteret replaces the I/O backend (async obstore instead of rasterio/GDAL) but
+Rasteret replaces the I/O backend (custom IO instead of rasterio/GDAL) but
 speaks the same interface. Your samplers, DataLoader, transforms, and training
 loop do not change.
 

docs/changelog.md

@@ -38,9 +38,10 @@
   Cooperative exports, STAC GeoParquet, custom catalogs). No STAC API
   required for the table path. Optional `enrich_cog=True` parses COG
   headers for accelerated reads.
-- **Multi-cloud obstore backend**: S3, Azure Blob, and GCS routing via URL
-  auto-detection, with automatic fallback to anonymous access.
-- **`create_backend()`** for authenticated reads with obstore credential
+- **Multi-cloud support**: S3, Azure Blob, and GCS routing via URL
+  auto-detection, with automatic fallback to anonymous access. obstore replaces
+  direct aiohttp as the HTTP transport, adding unified multi-cloud routing.
+- **`create_backend()`** for authenticated reads with credential
   providers (e.g., Planetary Computer SAS tokens).
 - **TorchGeo adapter**: `collection.to_torchgeo_dataset()` returns a
   `GeoDataset` backed by Rasteret's async COG reader. Supports
@@ -101,7 +102,7 @@
 ### Other changes
 
 - Arrow-native geometry internals (GeoArrow replaces Shapely in hot paths).
-- obstore as base dependency (Rust-native async HTTP).
+- obstore as base dependency (multi-cloud support).
 - CLI: `rasteret collections build|list|info|delete|import`,
   `rasteret datasets list|info|build|register-local|export-local|unregister-local`.
 - TorchGeo `time_series=True` uses spatial-only intersection, matching

docs/contributing.md

@@ -86,7 +86,7 @@ src/rasteret/
 │   └── utils.py             Sync/async bridging (run_sync), CRS transforms, data source inference
 │
 ├── fetch/
-│   ├── cog.py               [COGReader](reference/fetch/cog.md): HTTP range reads, tile decompression, obstore backend
+│   ├── cog.py               [COGReader](reference/fetch/cog.md): HTTP range reads, tile decompression, aioHTTP via Obstore
 │   └── header_parser.py     TIFF/COG header parsing: IFD extraction, tile offset discovery
 │
 ├── ingest/

docs/explanation/architecture.md

@@ -2,6 +2,45 @@
 
 This page explains how Rasteret's components work together.
 
+## Category framing
+
+Rasteret uses an **index-first geospatial retrieval** architecture:
+
+- **Control plane (tables/Parquet)**: dataset discovery, filtering, train/val/test splits, patch metadata, and cached COG metadata.
+- **Data plane (COG object storage)**: on-demand TIFF byte reads from source GeoTIFF/COG assets.
+
+This split keeps metadata workflows table-native while avoiding payload-in-Parquet duplication.
+
+### Architecture
+
+```text
+┌──────────────────────────────────────────────────────────────────────────────┐
+│ Rasteret control plane (Collection lifecycle + index schema)                 │
+│ build* / load / as_collection / export / subset / where                      │
+│ outputs: Parquet-backed collection rows (scene metadata + COG header cache)  │
+└───────────────────────────────┬──────────────────────────────────────────────┘
+                                │ Parquet rows + geometry + user columns
+                                v
+┌──────────────────────────────────────────────────────────────────────────────┐
+│ Arrow ecosystem (optional compute + enrichment)                              │
+│ DuckDB / Polars / GeoPandas / pandas / PyArrow                               │
+│ operations: add split/label/patch/AOI columns, filter, joins, aggregations   │
+└───────────────────────────────┬──────────────────────────────────────────────┘
+                                │ filtered rows + geometry column
+                                v
+┌──────────────────────────────────────────────────────────────────────────────┐
+│ Rasteret IO engine (custom byte range fetches)                               │
+│ get_numpy() / get_xarray() / to_torchgeo_dataset()                           │
+│ consumes filtered rows + geometry + cached tile metadata                     │
+└───────────────────────────────┬──────────────────────────────────────────────┘
+                                │ async byte-range tile requests
+                                v
+┌──────────────────────────────────────────────────────────────────────────────┐
+│ Data plane: object storage with GeoTIFF/COG files                            │
+│ S3 / GCS / Azure / *.tif                                                     │
+└──────────────────────────────────────────────────────────────────────────────┘
+```
+
 ## Data flow
 
 ```text
@@ -33,7 +72,7 @@ Collection  (Arrow dataset wrapper)
     +--> iterate_rasters()      --> async RasterAccessor stream
     |
     v
-obstore (auto-routes to S3Store / AzureStore / GCSStore / HTTPStore)
+custom IO engine (async byte-range reads, tile decode, geometry mask) — obstore as HTTP transport (auto-routes to S3Store / AzureStore / GCSStore / HTTPStore)
 ```
 
 `DatasetRegistry` is the in-code dataset catalog. It powers `build()` and the
@@ -89,8 +128,9 @@ Rasteret's speedup over sequential approaches.
 tile-level byte-range reads. It:
 
 1. Merges nearby byte ranges to minimize HTTP round-trips.
-2. Delegates to `obstore` for all remote reads, with automatic URL routing
-   to native cloud stores (S3Store, AzureStore, GCSStore, or HTTPStore).
+2. Issues async byte-range requests via obstore (HTTP transport layer), with
+   automatic URL routing to native cloud stores (S3Store, AzureStore, GCSStore,
+   or HTTPStore).
 3. Decompresses tiles (deflate, LZW, zstd, LERC) in a thread pool.
    (TIFF JPEG is currently rejected with a hard error until implemented.)
 

docs/explanation/benchmark.md

@@ -45,7 +45,7 @@ before pixel reads begin.
 | Backend | Transport | Install |
 |---------|-----------|---------|
 | **TorchGeo** | GDAL `/vsicurl/` (sequential) | `uv pip install torchgeo` |
-| **Rasteret** | Rust-native HTTP via obstore (concurrent) | `uv pip install rasteret` |
+| **Rasteret** | Custom async IO engine (concurrent byte-range reads, obstore transport) | `uv pip install rasteret` |
 
 ## Cold start (first run)
 

docs/explanation/design-decisions.md

@@ -84,17 +84,16 @@ When does promotion happen?
 In both cases, the promotion is explicit at the call site (not an implicit
 side effect of masking).
 
-## obstore as the HTTP backend
+## obstore as the URL routing backend
 
-Rasteret uses [obstore](https://github.com/developmentseed/obstore) for all
-remote byte-range reads. obstore wraps the
-[`object_store`](https://docs.rs/object_store/) Rust crate. It is Rust-native,
-multi-cloud, and does not depend on GDAL.
+Rasteret uses [obstore](https://github.com/developmentseed/obstore) as the
+HTTP transport layer for multi-cloud URL routing. Rasteret's byte-range read
+logic, tile decode, and concurrency are custom; obstore replaced direct aiohttp
+to gain unified S3/GCS/Azure routing without adding per-cloud clients.
 
 Why obstore as a hard dependency rather than optional?
 
-- **Multi-cloud + auth**: Rasteret previously used a Python HTTP client
-  (aiohttp) for range reads. obstore gives a single, well-tested interface for
+- **Multi-cloud + auth**: Rasteret has custom IO for byte range reads. obstore gives a single, well-tested interface for
   S3/GCS/Azure/HTTPS plus credential providers (requester-pays, SAS signing,
   Earthdata, etc.).
 - **Single code path**: one backend means fewer branches to test and maintain.
@@ -103,13 +102,12 @@ Why obstore as a hard dependency rather than optional?
   `storage.googleapis.com`, AzureStore for `*.blob.core.windows.net`, and
   HTTPStore for everything else. Authenticated reads use obstore credential
   providers via `create_backend()`.
-- **Well-maintained lineage**: obstore is from Development Seed and it wraps
-  the same Rust crate [`object_store`](https://docs.rs/object_store/) that Databricks, InfluxDB, and many in Arrow
-  ecosystem depend on.
+- **Well-maintained lineage**: obstore wraps the same Rust crate [`object_store`](https://docs.rs/object_store/)
+  that Databricks, InfluxDB, and many in Arrow ecosystem depend on.
 
 ## Decoupled index and read layers
 
-The Parquet index is the stable contract. The read layer ([COGReader](../reference/fetch/cog.md), obstore)
+The Parquet index is the stable contract. The read layer ([COGReader](../reference/fetch/cog.md))
 is swappable via the [`StorageBackend`](../reference/cloud.md) protocol. Concretely:
 
 - Custom backends (e.g. a pre-configured `S3Store`) can be plugged in without

docs/explanation/interop.md

@@ -46,7 +46,7 @@ full contract that samplers and dataset composition rely on:
 | Samplers | Works with `RandomGeoSampler`, `GridGeoSampler`, and any sampler that reads `bounds`, `index`, and `res` |
 | Dataset composition | Works with `IntersectionDataset` and `UnionDataset`; the index is designed so `reset_index()` does not conflict |
 
-Rasteret replaces the I/O backend (async obstore instead of rasterio/GDAL)
+Rasteret replaces the I/O backend (custom IO instead of rasterio/GDAL)
 but speaks the same interface. Nothing downstream of the dataset object
 needs to change.
 
@@ -93,7 +93,7 @@ See [TorchGeo Integration](../tutorials/02_torchgeo_09_accelerator.ipynb) and
 
 ### xarray / GeoPandas / NumPy
 
-Rasteret handles the I/O (async byte-range reads via obstore), then hands
+Rasteret handles the I/O (async byte-range reads), then hands
 off to standard xarray, GeoPandas, or NumPy outputs:
 
 - [`Collection.get_xarray(...)`](../reference/core/collection.md) returns an `xr.Dataset`
@@ -137,7 +137,7 @@ Rasteret uses rasterio for geometry masking (`rasterio.features.geometry_mask`),
 multi-CRS reprojection (`rasterio.warp.reproject`), and TorchGeo query-grid
 placement (`rasterio.merge.merge` via `rio_semantics.py`). CRS transforms and
 coordinate operations use pyproj directly. Tile reads go through Rasteret's
-own async pipeline backed by obstore. No GDAL in the tile-read path.
+own async IO. No GDAL in the tile-read path.
 
 CRS encoding in xarray output uses pyproj's CF conventions (`CRS.to_cf()`,
 `CRS.to_wkt()`, `CRS.to_json()`), not rioxarray.

docs/how-to/custom-cloud-provider.md

@@ -3,9 +3,8 @@
 ## Do you need this page?
 
 Most datasets work **without any authentication or configuration**. Rasteret's
-obstore backend routes URLs to native cloud stores (S3, Azure Blob, GCS)
-automatically, and public data like Sentinel-2 on Earth Search is read
-anonymously.
+IO layer automatically routes URLs to native cloud stores (S3, Azure Blob, GCS),
+and public data like Sentinel-2 on Earth Search is read anonymously.
 
 | Data source | Auth needed | What to do |
 |---|---|---|
@@ -82,7 +81,7 @@ ds = collection.get_xarray(
 )
 ```
 
-Rasteret auto-creates an obstore backend from the config at read time.
+Rasteret auto-creates a backend from the config at read time.
 For requester-pays buckets, AWS credentials are resolved automatically
 from environment variables or `~/.aws/credentials` via boto3.
 
@@ -107,10 +106,10 @@ For catalog datasets, the `data_source` is the catalog ID (e.g.
 should use with `CloudConfig.get_config(...)` if you want to inspect or
 override built-in behavior.
 
-## Multi-cloud obstore backend
+## Multi-cloud URL routing (via obstore)
 
-Rasteret uses obstore for all remote reads and natively routes URLs to the
-correct cloud store:
+Rasteret's IO layer uses obstore as the HTTP transport and natively routes
+URLs to the correct cloud store:
 
 | URL pattern | Store type |
 |---|---|
@@ -128,7 +127,7 @@ This happens automatically -- no configuration needed for public data.
 
 Use `create_backend()` when your data source provides its own credential
 mechanism (Planetary Computer SAS tokens, Earthdata-style temporary S3 credentials).
-This passes the credential provider directly to the obstore native store
+This passes the credential provider to the underlying cloud store
 (S3Store, AzureStore, GCSStore):
 
 ### Planetary Computer

docs/reference/integrations/torchgeo.md

@@ -3,7 +3,7 @@
 TorchGeo `GeoDataset` adapter for Rasteret collections.
 
 `RasteretGeoDataset` is a standard TorchGeo `GeoDataset` subclass. It
-replaces the I/O backend (async obstore instead of rasterio/GDAL) while
+replaces the I/O backend (instead of rasterio/GDAL) while
 honoring the full GeoDataset contract: `index`, `crs`, `res`,
 `__getitem__(GeoSlice) -> Sample`. Compatible with all TorchGeo samplers,
 collation helpers (`stack_samples`, `concat_samples`), transforms, and