Skip to content

Add support for Apache Iceberg REST Catalog specification#198

Open
HJLebbink wants to merge 2 commits intominio:masterfrom
HJLebbink:feature/s3tables
Open

Add support for Apache Iceberg REST Catalog specification#198
HJLebbink wants to merge 2 commits intominio:masterfrom
HJLebbink:feature/s3tables

Conversation

@HJLebbink
Copy link
Member

@HJLebbink HJLebbink commented Dec 1, 2025

This PR implements the MinIO S3 Tables API based on the Apache Iceberg REST Catalog specification, providing a complete Rust SDK for Iceberg table operations on MinIO.

Features

Core Iceberg REST Catalog API

Warehouse Management:

  • Create, list, get, delete warehouses
  • Warehouse-level encryption, policies, metrics, replication, storage class configuration

Namespace Management:

  • Create, list, get, delete namespaces (with multi-level namespace support)
  • Update namespace properties
  • Namespace existence checks

Table Management:

  • Create, delete, load, register, rename tables
  • Table existence checks and metadata operations
  • Table-level encryption, policies, replication, maintenance, expiration, storage class
  • Table metrics and monitoring

View Management:

  • Create, load, register, replace, rename, drop views
  • View versions and SQL dialect support
  • View existence checks

Scan Planning:

  • Plan table scans with filters and projections
  • Fetch scan tasks and planning results
  • Cancel ongoing planning operations

Advanced Operations:

  • Commit table transactions (atomic multi-statement operations)
  • Multi-table transactions
  • Tagging support (tag/untag/list tags for resources)

@HJLebbink HJLebbink requested a review from Copilot December 1, 2025 20:34
@HJLebbink HJLebbink self-assigned this Dec 1, 2025
@HJLebbink HJLebbink added the enhancement Used in release doc generation label Dec 1, 2025
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR introduces S3 Tables support to the MinIO Rust SDK, implementing a comprehensive Apache Iceberg REST Catalog client with warehouse management, namespace operations, table/view lifecycle, and query planning capabilities.

Key changes:

  • New s3tables module with full Iceberg REST API implementation
  • Builder pattern for all operations with typed-builder support
  • Pluggable authentication (SigV4, Bearer, NoAuth)
  • Performance optimizations (signing key cache, zero-copy streaming, multimap improvements)
  • Enhanced checksum support (CRC32/C, CRC64-NVME, SHA1/256)

Reviewed changes

Copilot reviewed 79 out of 221 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
src/s3tables/builders/*.rs Request builders for Tables API operations (warehouses, namespaces, tables, views)
src/s3tables/auth.rs Authentication providers for different catalog backends (SigV4, Bearer, NoAuth)
src/s3tables/advanced/*.rs Advanced operations for direct Iceberg metadata manipulation
src/s3/utils.rs Checksum utilities (CRC32/C/64, SHA1/256) for data integrity verification
src/s3/signer.rs Signing key cache for performance optimization
src/s3/client/mod.rs HTTP/2 support, connection pooling, fast-path GET, region lookup skip
src/s3/multimap_ext.rs Optimized header canonicalization (regex removal, pre-allocation)
examples/s3tables/*.rs Example programs and stress tests for Tables API
Comments suppressed due to low confidence (1)

src/s3/client/mod.rs:1

  • This TODO comment in test code should either be addressed or removed. If the test is known to be broken or unnecessary, it should be fixed or deleted rather than left with an unresolved TODO.
// MinIO Rust Library for Amazon S3 Compatible Cloud Storage

@HJLebbink HJLebbink force-pushed the feature/s3tables branch 4 times, most recently from fdd77b6 to 066d0ac Compare December 2, 2025 20:55
@HJLebbink HJLebbink requested a review from Copilot December 2, 2025 20:55
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 77 out of 231 changed files in this pull request and generated 5 comments.

@HJLebbink HJLebbink force-pushed the feature/s3tables branch 20 times, most recently from 53221cb to 781b702 Compare December 8, 2025 17:55
@HJLebbink HJLebbink force-pushed the feature/s3tables branch 5 times, most recently from 9a77319 to 10dcc20 Compare January 26, 2026 23:13
@shtripat shtripat force-pushed the feature/s3tables branch 4 times, most recently from 341190e to b6721d2 Compare January 27, 2026 11:39
@HJLebbink HJLebbink force-pushed the feature/s3tables branch 7 times, most recently from e0f8714 to cb66a97 Compare January 27, 2026 19:33
@HJLebbink HJLebbink requested a review from Copilot January 27, 2026 19:56
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 196 out of 313 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

src/s3tables/advanced/response/commit_multi_table_transaction.rs:1

  • This CommitMultiTableTransactionResponse definition is effectively a duplicate of the one in src/s3tables/response/commit_multi_table_transaction.rs, which increases maintenance burden and risk of the two versions diverging. Consider reusing a single definition (e.g., via a pub use or a type alias from the advanced module to the base response) instead of maintaining two separate but identical structs.

@HJLebbink HJLebbink changed the title Feature/s3tables Add support for Apache Iceberg REST Catalog specification Jan 27, 2026
This PR implements the MinIO S3 Tables API based on the Apache Iceberg
REST Catalog specification, providing a complete Rust SDK for Iceberg
table operations on MinIO.
@HJLebbink
Copy link
Member Author

Apache Iceberg support in MinIO-rs. If someone needs it, here it is.

@HJLebbink HJLebbink marked this pull request as ready for review January 27, 2026 20:14
@twuebi
Copy link
Contributor

twuebi commented Jan 28, 2026

Why would I use this over iceberg-rust?

For Iceberg Ecosystem Integration this PR introduces a standalone SigV4 authentication
crate (iceberg-sigv4) and adds optional iceberg-rust type compatibility, enabling better
integration with the Apache Iceberg ecosystem while keeping iceberg-rust as an
internal implementation detail invisible to SDK users.
It uses iceberg-rust as base for building this echo system.

Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement Used in release doc generation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants