Add per-tile statistics columns (v0.5.0 spec, v0.9.0 CLI)

Optional pre-computed per-tile statistics (count, min, max, sum, mean,
stddev) as plain Parquet columns alongside each band. Enables UDF-free
analytics on any SQL engine — no decompression needed.

- Spec bumped to v0.5.0 with tile statistics section
- CLI: --tile-stats flag for convert raster command
- Validator updated to recognize stats columns
- Docs/changelog updated across site

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Author: jatorre

deforest_carbon.parquet

@@ -4,8 +4,8 @@ url: https://raquet.io
 baseurl: ""
 
 # Versions (auto-updated by CI)
-cli_version: "0.8.1"
-spec_version: "0.4.0"
+cli_version: "0.9.0"
+spec_version: "0.5.0"
 
 # Build settings
 markdown: kramdown

deforest_carbon.vrt

@@ -28,6 +28,7 @@ uv add raquet-io
 | `inspect` | Display metadata and statistics |
 | `validate` | Validate file structure and data integrity |
 | `export geotiff` | Export RaQuet back to GeoTIFF |
+| `partition` | Spatially partition a file for cloud storage |
 | `split-zoom` | Split by zoom level for optimized remote access |
 
 ---
@@ -56,6 +57,11 @@ raquet-io convert raster INPUT_FILE OUTPUT_FILE [OPTIONS]
 | `--block-size` | `256` | Block size in pixels: `256`, `512`, or `1024` (see [Block Size](#block-size)) |
 | `--target-size` | — | Target size for auto zoom calculation |
 | `--row-group-size` | `200` | Rows per Parquet row group (smaller = better remote pruning) |
+| `--overviews` | `auto` | Overview generation: `auto` (full pyramid) or `none` (native resolution only) |
+| `--min-zoom` | — | Minimum zoom level for overviews (overrides auto calculation) |
+| `--streaming` | — | Two-pass streaming mode for memory-safe conversion of large files |
+| `--workers` | `1` | Parallel worker processes (requires `--overviews none`) |
+| `--tile-stats` | — | Include per-tile statistics columns (count, min, max, sum, mean, stddev) |
 | `--band-layout` | `sequential` | Band storage: `sequential` or `interleaved` |
 | `--compression` | `gzip` | Compression: `gzip`, `jpeg`, `webp`, or `none` |
 | `--compression-quality` | `85` | Quality for lossy compression (1-100) |
@@ -76,8 +82,17 @@ raquet-io convert raster dem.tif dem.parquet --resampling bilinear
 # Larger blocks for dense data
 raquet-io convert raster satellite.tif output.parquet --block-size 512
 
-# Verbose output to monitor progress
-raquet-io convert raster large.tif output.parquet -v
+# Native resolution only (no overview pyramid), faster conversion
+raquet-io convert raster large.tif output.parquet --overviews none -v
+
+# Streaming mode for very large files (lower memory usage)
+raquet-io convert raster huge.tif output.parquet --streaming -v
+
+# Parallel conversion (4 workers, requires --overviews none)
+raquet-io convert raster huge.tif output.parquet --streaming --workers 4 --overviews none -v
+
+# Include per-tile statistics columns for UDF-free analytics
+raquet-io convert raster slope.tif slope.parquet --tile-stats --overviews none -v
 
 # Lossy compression for RGB satellite imagery (10-15x smaller files)
 raquet-io convert raster satellite.tif output.parquet \
@@ -86,6 +101,21 @@ raquet-io convert raster satellite.tif output.parquet \
   --compression-quality 85
 ```
 
+### Tile Statistics
+
+The `--tile-stats` flag adds pre-computed per-tile statistics as plain Parquet columns alongside each band. For each band, six columns are added: `{band}_count`, `{band}_min`, `{band}_max`, `{band}_sum`, `{band}_mean`, `{band}_stddev`.
+
+This enables **UDF-free analytics** on any SQL engine — no decompression needed:
+
+```sql
+-- Works on DuckDB, Snowflake, BigQuery, Databricks — no extensions required
+SELECT AVG(band_1_mean) AS avg_slope, MAX(band_1_max) AS steepest
+FROM 'slope.parquet'
+WHERE block != 0;
+```
+
+The overhead is negligible (typically <1% file size increase). See the [specification](https://github.com/CartoDB/raquet/blob/master/format-specs/raquet.md) for details.
+
 ### Block Size
 
 The `--block-size` option controls the pixel dimensions of each tile. The default is 256px (the web map standard), but 512px can be beneficial in certain scenarios.
@@ -197,7 +227,7 @@ raquet-io inspect landcover.parquet -v
 
 ```
 RaQuet File: spain_solar_ghi.parquet
-Version: 0.4.0
+Version: 0.5.0
 Size: 15.2 MB
 
 Dimensions: 9216 x 7936 pixels
@@ -259,7 +289,7 @@ raquet-io validate raster.parquet --json
 ```
 Validating: spain_solar_ghi.parquet
 ✓ Schema valid
-✓ Metadata valid (v0.4.0)
+✓ Metadata valid (v0.5.0)
 ✓ Pyramid complete (zoom 3-9)
 ✓ Band statistics valid
 ✓ Data integrity OK
@@ -299,6 +329,47 @@ raquet-io export geotiff raster.parquet output.tif -v
 
 ---
 
+## partition
+
+Spatially partition a RaQuet file into multiple files for optimized cloud storage access.
+
+```bash
+raquet-io partition INPUT_FILE OUTPUT_DIR [OPTIONS]
+```
+
+### Arguments
+
+| Argument | Description |
+|----------|-------------|
+| `INPUT_FILE` | Path to source RaQuet file |
+| `OUTPUT_DIR` | Directory for output partition files |
+
+### Options
+
+| Option | Default | Description |
+|--------|---------|-------------|
+| `--partition-zoom` | `auto` | QUADBIN zoom level for partitioning, or `auto` |
+| `--target-size-mb` | `128` | Target partition file size in MB (used with `auto`) |
+| `--row-group-size` | `200` | Rows per Parquet row group |
+| `-v, --verbose` | — | Enable verbose output |
+
+### Examples
+
+```bash
+# Auto partition (targets ~128 MB files)
+raquet-io partition slope.parquet ./partitioned/
+
+# Custom target size
+raquet-io partition slope.parquet ./partitioned/ --target-size-mb 256
+
+# Explicit partition zoom
+raquet-io partition slope.parquet ./partitioned/ --partition-zoom 12
+```
+
+Partitioning is recommended for large datasets (>1 GB) that will be queried from cloud storage. Each partition file is a valid standalone RaQuet file with its own metadata. Tile statistics columns are preserved automatically.
+
+---
+
 ## split-zoom
 
 Split a RaQuet file by zoom level for optimized remote access.

docs/_config.yml

@@ -324,13 +324,39 @@ See [CARTO Analytics Toolbox documentation](https://docs.carto.com/data-and-anal
 
 ---
 
+## UDF-Free Analytics with Tile Statistics
+
+RaQuet v0.5.0 files converted with `--tile-stats` include pre-computed per-tile statistics as plain Parquet columns. This means **any SQL engine** can perform raster analytics without UDFs, extensions, or decompression:
+
+```sql
+-- Works on ANY Parquet-compatible engine — no extensions needed
+-- Find average slope and steepest point across all tiles
+SELECT
+    AVG(band_1_mean) AS avg_slope,
+    MAX(band_1_max) AS steepest_slope,
+    SUM(band_1_count) AS total_pixels
+FROM 'slope.parquet'
+WHERE block != 0;
+
+-- Filter tiles by statistics (e.g., find flat areas for data centers)
+SELECT block, band_1_mean, band_1_max
+FROM 'slope.parquet'
+WHERE block != 0 AND band_1_max < 5.0;
+```
+
+Available columns per band: `{band}_count` (int64), `{band}_min`, `{band}_max`, `{band}_sum`, `{band}_mean`, `{band}_stddev` (all float64).
+
+---
+
 ## Performance Tips
 
-1. **Use spatial filtering** — Always include `ST_RasterIntersects` or equivalent to enable row group pruning
+1. **Use tile statistics for aggregates** — When you only need summary stats (mean, min, max), query the `{band}_mean` etc. columns directly instead of decompressing tile data
+
+2. **Use spatial filtering** — Always include `ST_RasterIntersects` or equivalent to enable row group pruning
 
-2. **Query remote files directly** — Parquet's columnar format enables efficient range requests; no need to download first
+3. **Query remote files directly** — Parquet's columnar format enables efficient range requests; no need to download first
 
-3. **Split by zoom for large files** — Use `raquet-io split-zoom` to create per-zoom files for optimal remote queries
+4. **Partition large files** — Use `raquet-io partition` to spatially split files for optimal cloud storage parallelism
 
-4. **Small row groups for remote access** — Use `--row-group-size 100-200` when converting files that will be queried remotely
+5. **Small row groups for remote access** — Use `--row-group-size 100-200` when converting files that will be queried remotely
 

docs/cli.md

@@ -112,12 +112,12 @@ Several options:
 
 ## What's the metadata format?
 
-RaQuet v0.4.0 stores metadata as JSON in the `block=0` row:
+RaQuet v0.5.0 stores metadata as JSON in the `block=0` row:
 
 ```json
 {
   "file_format": "raquet",
-  "version": "0.4.0",
+  "version": "0.5.0",
   "width": 32768,
   "height": 14848,
   "crs": "EPSG:3857",
@@ -186,7 +186,7 @@ RaQuet is ideal when you need to query rasters outside PostgreSQL, join with dat
 
 Yes. RaQuet is used in production at CARTO and is supported by the [Analytics Toolbox](https://carto.com/analytics-toolbox) across BigQuery, Snowflake, Databricks, and PostgreSQL.
 
-The format specification is at v0.4.0. Version 0.3.0 is stable for production use; v0.4.0 adds experimental interleaved band layout and lossy compression support.
+The format specification is at v0.5.0. The format is stable for production use. v0.4.0 added interleaved band layout and lossy compression; v0.5.0 adds optional per-tile statistics columns for UDF-free analytics on data warehouses.
 
 ---
 

docs/engines.md

@@ -187,7 +187,13 @@ GeoParquet brought vector data into the lakehouse. RaQuet does the same for rast
 
 ## Changelog
 
-### v0.4.0 (Experimental)
+### v0.5.0
+- **Per-Tile Statistics Columns**: Optional pre-computed statistics (`count`, `min`, `max`, `sum`, `mean`, `stddev`) as plain Parquet columns alongside each band. Enables UDF-free analytics on any SQL engine — no decompression needed.
+- **New CLI options**: `--tile-stats` flag for conversion, `--overviews none`, `--streaming`, `--workers` for parallel conversion, and `partition` command for spatial partitioning.
+- **Metadata signal**: `tile_statistics` and `tile_statistics_columns` fields in metadata JSON when tile stats are present.
+- **Negligible overhead**: Typically <1% file size increase for the statistics columns.
+
+### v0.4.0
 - **Interleaved Band Layout**: New `band_layout: "interleaved"` option stores all bands in a single `pixels` column using Band Interleaved by Pixel (BIP) format. This can reduce HTTP requests for RGB visualization by ~40%.
 - **Lossy Compression**: Support for JPEG and WebP compression for photographic imagery. Achieves 10-15x smaller files compared to gzip for satellite imagery.
 - **New metadata fields**: `band_layout`, `compression_quality` for controlling lossy compression.

docs/faq.md

@@ -1,4 +1,4 @@
-# RaQuet Specification v0.4.0
+# RaQuet Specification v0.5.0
 
 ## Overview
 
@@ -86,6 +86,45 @@ When `band_layout` is `"interleaved"`:
 **Primary Key with Time Dimension:**
 When `time_cf` is present, the combination of (`block`, `time_cf`) forms the unique key for each row. Multiple rows may have the same `block` value but different timestamps. Without time columns, `block` alone is unique (excluding the metadata row at `block = 0`).
 
+#### Tile Statistics Columns (Per-Tile Statistics)
+
+Optional pre-computed per-tile statistics for each band. These columns enable analytical queries without decompressing tile pixel data, which is especially valuable for cloud data warehouses (Snowflake, BigQuery, Databricks) where UDF-based decompression is expensive.
+
+**Column naming convention**: `{band_name}_{stat}` where `{band_name}` matches the band column name and `{stat}` is one of the supported statistics.
+
+**Supported statistics**:
+
+| Stat | Type | Description |
+|------|------|-------------|
+| `count` | int64 | Number of valid (non-nodata) pixels in the tile |
+| `min` | float64 | Minimum pixel value (excluding nodata) |
+| `max` | float64 | Maximum pixel value (excluding nodata) |
+| `sum` | float64 | Sum of all valid pixel values |
+| `mean` | float64 | Mean of valid pixel values |
+| `stddev` | float64 | Population standard deviation of valid pixel values |
+
+**Example columns** for a file with `band_1`:
+- `band_1_count` (int64)
+- `band_1_min` (float64)
+- `band_1_max` (float64)
+- `band_1_sum` (float64)
+- `band_1_mean` (float64)
+- `band_1_stddev` (float64)
+
+**Rules**:
+- The metadata row (`block = 0`) MUST have NULL values for all tile statistics columns.
+- Statistics are computed from raw (pre-compression) pixel data. For lossy compression (JPEG/WebP), statistics reflect source values, not post-decode values.
+- Empty tiles (all nodata) are excluded during conversion and have no statistics rows.
+- For time-series data with `time_cf`, statistics are per-tile-per-timestep (one set of stats per row).
+- Overview tiles have their own statistics computed from their (coarser resolution) pixel data.
+- For interleaved band layout, statistics columns use the original band names from metadata (e.g., `band_1_mean`, `band_2_mean`), not the `pixels` column name.
+
+**Metadata signal**: When tile statistics columns are present, the metadata JSON includes:
+- `"tile_statistics": true` — indicates that per-tile statistics columns are present.
+- `"tile_statistics_columns": ["count", "min", "max", "sum", "mean", "stddev"]` — lists which statistics are included.
+
+**Compatibility**: Tile statistics columns are plain Parquet columns. Any Parquet reader (pandas, DuckDB, Snowflake, BigQuery, Spark, Polars) can query them directly without UDFs or custom extensions. Files without tile statistics columns remain valid RaQuet.
+
 ## Tiling Scheme
 
 RaQuet uses the **QUADBIN** tiling scheme for spatial indexing. QUADBIN is a hierarchical geospatial index that encodes Web Mercator tile coordinates `(x, y, z)` into a single 64-bit integer. This encoding enables efficient spatial queries and Parquet row group pruning.
@@ -158,7 +197,7 @@ The metadata is stored as a JSON string in the `metadata` column where `block =
 ```json
 {
     "file_format": "raquet",
-    "version": "0.4.0",
+    "version": "0.5.0",
     "width": 9216,
     "height": 7936,
     "crs": "EPSG:3857",
@@ -213,7 +252,7 @@ The metadata is stored as a JSON string in the `metadata` column where `block =
 
 - **Format Identification**
   - `file_format`: String identifying this as a RaQuet file. MUST be `"raquet"`.
-  - `version`: String indicating the RaQuet specification version. Current version is "0.4.0".
+  - `version`: String indicating the RaQuet specification version. Current version is "0.5.0".
 
 - **Raster Dimensions**
   - `width`, `height`: Integers specifying full resolution raster dimensions in pixels.
@@ -338,7 +377,7 @@ The metadata is stored as a JSON string in the `metadata` column where `block =
 ```json
 {
     "file_format": "raquet",
-    "version": "0.4.0",
+    "version": "0.5.0",
     "width": 9216,
     "height": 7936,
     "crs": "EPSG:3857",
@@ -380,7 +419,7 @@ The metadata is stored as a JSON string in the `metadata` column where `block =
 ```json
 {
     "file_format": "raquet",
-    "version": "0.4.0",
+    "version": "0.5.0",
     "width": 1024,
     "height": 1024,
     "crs": "EPSG:3857",
@@ -435,7 +474,7 @@ The metadata is stored as a JSON string in the `metadata` column where `block =
 ```json
 {
     "file_format": "raquet",
-    "version": "0.4.0",
+    "version": "0.5.0",
     "width": 10980,
     "height": 10980,
     "crs": "EPSG:3857",
@@ -485,7 +524,7 @@ This example shows a Sentinel-2 TCI (True Color Image) stored with interleaved b
 ```json
 {
     "file_format": "raquet",
-    "version": "0.4.0",
+    "version": "0.5.0",
     "width": 32768,
     "height": 14848,
     "crs": "EPSG:3857",
@@ -521,7 +560,7 @@ This example shows a Sentinel-2 TCI (True Color Image) stored with interleaved b
 ```json
 {
     "file_format": "raquet",
-    "version": "0.4.0",
+    "version": "0.5.0",
     "width": 1440,
     "height": 721,
     "crs": "EPSG:3857",
@@ -571,7 +610,7 @@ This example represents 36 years (1980-2015) of monthly sea surface temperature
 ```json
 {
     "file_format": "raquet",
-    "version": "0.4.0",
+    "version": "0.5.0",
     "width": 400752,
     "height": 131072,
     "crs": "EPSG:3857",
@@ -686,7 +725,7 @@ Producers MAY extend the metadata with custom fields. To avoid conflicts with fu
    ```json
    {
        "file_format": "raquet",
-       "version": "0.4.0",
+       "version": "0.5.0",
        "custom": {
            "organization": "ACME Corp",
            "project_id": "climate-2024",

docs/index.md

@@ -1,6 +1,6 @@
 [project]
 name = "raquet-io"
-version = "0.8.1"
+version = "0.9.0"
 description = "RaQuet - Raster data in Parquet format with QUADBIN spatial indexing"
 readme = "README.md"
 license = {text = "BSD-3-Clause"}