[Proposal] Cloudberry Multi-Catalog System Design Proposal

[yjhjstz]

Proposers

@yjhjstz @my-ship-it

Proposal Status

Under Discussion

Abstract

Cloudberry Multi-Catalog System Design Proposal

Based on the comprehensive analysis provided and research into Apache Doris's Iceberg Catalog implementation, I'll create a detailed proposal for Cloudberry's multi-catalog system design. The key idea is that no CREATE FOREIGN TABLE is required. usage

Executive Summary

Apache Cloudberry, as an advanced open-source MPP database derived from PostgreSQL[1][2], has a unique opportunity to implement a cutting-edge multi-catalog system that extends its current two-tier naming (database.table) to a three-tier hierarchy (catalog.database.table)[3]. This proposal outlines a comprehensive design for a plugin-based catalog framework that enables federated queries across multiple data sources while maintaining PostgreSQL compatibility and leveraging Cloudberry's MPP architecture.

Motivation

Background and Motivation

Apache Cloudberry currently supports external data access through Foreign Data Wrappers (FDW)[4][5] and Platform Extension Framework (PXF)[6], but lacks the unified multi-catalog approach that modern data lakehouse architectures demand. Apache Doris has successfully implemented multi-catalog functionality[3][7] that enables seamless integration with external data sources like Iceberg, Hive, and JDBC databases. This capability is crucial for organizations building modern data platforms that need to query across heterogeneous data sources.

The proposed multi-catalog system addresses several key challenges:

• Data Silos: Enable unified access to data across multiple systems without data movement
• Vendor Lock-in: Provide open-source alternative to proprietary data federation solutions
• Performance: Leverage Cloudberry's MPP architecture for distributed query execution
• Standards Compliance: Maintain PostgreSQL/SQL standards while extending functionality

Implementation

Core Architecture Design

1. Three-Tier Naming Hierarchy

The system extends Cloudberry's current naming convention from database.table to catalog.database.table, enabling:

• Backward Compatibility: Default internal catalog maintains existing behavior
• Federated Queries: Cross-catalog JOINs and analytics
• Namespace Isolation: Clear separation between internal and external data sources

2. Plugin-Based Catalog Framework

The architecture follows a layered approach similar to Apache Doris's multi-catalog implementation[3]:

┌─────────────────────────────────────────┐
│         SQL Parser (3-part names)       │
│         catalog.database.table          │
└─────────────────┬───────────────────────┘
                  │
                  ▼
┌─────────────────────────────────────────┐
│      Metadata Virtualization            │
│        Subsystem (MVS)                  │
├─────────────────────────────────────────┤
│ • Catalog Registry                      │
│ • Name Resolution Hooks                 │
│ • Dynamic Object Factory                │
│ • Schema/Type Converters                │
└─────────────────┬───────────────────────┘
                  │
                  ▼
┌─────────────────────────────────────────┐
│        Plugin Interface Layer           │
├─────────────────────────────────────────┤
│ Rust+pgrx │ C++ Native │ Python/Java    │
│ Plugins   │ Plugins    │ Services       │
└─────────────────┬───────────────────────┘
                  │
                  ▼
┌─────────────────────────────────────────┐
│       External Data Sources             │
│ Iceberg │ Hive │ JDBC │ S3 │ Delta...   │
└─────────────────────────────────────────┘

3. Metadata Virtualization Subsystem (MVS)

The MVS serves as the core component that manages catalog registration, metadata caching, and dynamic schema resolution[8]. Key components include:

Catalog Registry

• Global Registry: Hash table storing registered catalog implementations
• Thread Safety: LWLock protection for concurrent access
• Health Monitoring: Background process for catalog availability checks

Name Resolution Engine

• Hook Chain: Pluggable resolution mechanisms for different catalog types
• Cache Management: Multi-level caching with TTL-based expiration
• Dynamic Discovery: On-demand metadata fetching from external sources

Virtual Object Factory

• Dynamic Relations: Generate PostgreSQL Relation objects from external schemas
• Type Mapping: Convert external data types to PostgreSQL types
• Memory Management: Proper cleanup and resource management

Rust+pgrx Iceberg Plugin Implementation

Technical Architecture

The Iceberg plugin leverages modern Rust capabilities through the pgrx framework[9][10], which provides safe and efficient PostgreSQL extension development:

// Core FFI interface for metadata retrieval
#[no_mangle]
pub extern "C" fn get_table_desc_dynamic(
    catalog_type: *const c_char,
    catalog_config: *const c_char,
    database_name: *const c_char,
    table_name: *const c_char,
    error_code: *mut c_int,
) -> *mut TableDesc

Key Advantages

1. Performance: Direct C FFI with zero JVM overhead
2. Safety: Rust's memory safety prevents common extension crashes
3. Ecosystem: Leverages official iceberg-rust library[11][12]
4. Maintainability: Modern language with excellent tooling

Iceberg Integration Features

Based on Apache Doris's comprehensive Iceberg support[13][14], the plugin will provide:

• Catalog Types: REST, Hive Metastore, AWS Glue, Hadoop
• Table Formats: V1/V2 with Position Delete and Equality Delete support
• Time Travel: Query historical snapshots
• Schema Evolution: Dynamic schema updates without downtime
• Partition Transforms: Hidden partitioning and layout evolution

FDW Integration Strategy

The multi-catalog system integrates seamlessly with Cloudberry's existing Foreign Data Wrapper infrastructure[4][5]:

Query Execution Flow

1. SQL Parsing: Extended parser recognizes three-part names
2. Analyze Phase: MVS hooks resolve external table metadata
3. Planning: Standard PostgreSQL planner with foreign table support
4. Execution: FDW-based data scanning with MPP distribution

MPP-Aware Design

• Segment Distribution: Metadata pushed to all query executors
• Parallel Scanning: Distributed reading of external data sources
• Plan Serialization: Schema information embedded in execution plans

SQL-DDL-USAGE

-- Create external catalog
CREATE CATALOG iceberg_catalog WITH (
    TYPE = 'iceberg',
    CATALOG_TYPE = 'rest',
    CATALOG_URI = 'http://localhost:8181',
    WAREHOUSE = 's3://my-bucket/warehouse',
    CACHE_TTL = 300
);

-- scan iceberg table
SELECT * FROM iceberg.db.table_name;

-- time traval
SELECT * FROM iceberg.db.table_name; 
FOR VERSION AS OF TIMESTAMPTZ '2024-01-01 12:00:00'
WHERE user_id = 123;

-- View table history
SELECT * FROM iceberg.db.table_name$history;

-- Query across catalogs
SELECT i.product_id, c.customer_name, i.sales_amount
FROM iceberg_catalog.sales.transactions i
JOIN internal.crm.customers c ON i.customer_id = c.id;

Performance Optimization

Multi-Level Caching Strategy

1. Process-Level Cache: Fast local metadata cache with TTL
2. Shared Memory Cache: Cross-process metadata sharing
3. Coordinator Distribution: QD pushes metadata to segments

Query Optimization

• Predicate Pushdown: Push filters to external data sources
• Partition Pruning: Leverage external partitioning information
• Statistics Integration: Use external table statistics for cost-based optimization

Implementation Roadmap

Phase 1: Core Infrastructure

• Metadata Virtualization Subsystem implementation
• Catalog registry and name resolution engine
• Basic three-part name SQL parsing support

Phase 2: Rust Iceberg Plugin

• pgrx-based Iceberg plugin development
• FFI interface and async runtime setup
• Basic REST catalog support

Phase 3: FDW Integration

• FDW handler implementation
• Query planning and execution integration
• MPP-aware metadata distribution

Phase 4: Advanced Features

• Additional catalog types (Hive, AWS Glue)
• Performance optimizations
• Comprehensive testing and documentation

Phase 5: Production Hardening

• Error handling and recovery
• Monitoring and observability
• Security and access control

Benefits and Impact

For Users

• Unified Data Access: Query across multiple data sources with SQL
• No Vendor Lock-in: Open-source alternative to proprietary solutions
• Performance: Leverage Cloudberry's MPP architecture
• PostgreSQL Compatibility: Familiar SQL interface and tooling

For Ecosystem

• Modern Data Architecture: Enable lakehouse patterns with open formats
• Community Growth: Attract users from data lake and analytics space
• Innovation Platform: Foundation for future data federation features

Technical Advantages

• Rust Safety: Memory-safe extensions prevent database crashes
• Async Performance: Non-blocking metadata operations
• Cloud Native: Designed for object storage and containerized deployments

Conclusion

This multi-catalog system design positions Apache Cloudberry as a leading open-source data analytics platform capable of competing with modern data lakehouse solutions. By leveraging proven architectural patterns from Apache Doris[13][3] and implementing them with modern technologies like Rust and pgrx[9][10], Cloudberry can provide a robust, performant, and maintainable solution for federated data analytics.

The proposed system maintains full backward compatibility while enabling powerful new capabilities for modern data architectures. The plugin-based approach ensures extensibility, while the Rust implementation provides safety and performance that traditional C extensions cannot match.

This proposal represents a significant step forward for Apache Cloudberry in the competitive data analytics landscape, providing users with the tools they need to build modern, scalable data platforms.

References

[1] Announcing Cloudberry Database Enters the Apache Incubator https://www.postgresql.org/about/news/announcing-cloudberry-database-enters-the-apache-incubator-2976/
[2] Apache Cloudberry (Incubating) - The Apache Software Foundation https://cloudberry.apache.org/
[3] Build a federated query solution with Apache Doris, Apache Flink ... https://dev.to/apachedoris/build-a-federated-query-solution-with-apache-doris-apache-flink-and-apache-hudi-40io
[4] ALTER FOREIGN DATA WRAPPER - Apache Cloudberry (Incubating) https://cloudberry.apache.org/docs/1.x/sql-stmts/alter-foreign-data-wrapper/
[5] CREATE FOREIGN DATA WRAPPER https://cloudberry.apache.org/docs/sql-stmts/create-foreign-data-wrapper/
[6] apache/cloudberry-pxf: Platform Extension Framework ... - GitHub https://github.com/apache/cloudberry-pxf
[7] Apache Doris speeds up data reporting, tagging, and data lake ... https://www.velodb.io/blog/152
[8] Metadata and Data Virtualization Explained - Altoros https://www.altoros.com/blog/metadata-and-data-virtualization/
[9] Develop Database Extensions Using PGRX https://cloudberry.apache.org/docs/developer/develop-extensions-using-rust/
[10] pgcentralfoundation/pgrx: Build Postgres Extensions with Rust! https://github.com/pgcentralfoundation/pgrx
[11] iceberg - Rust - Docs.rs https://docs.rs/iceberg
[12] apache/iceberg-rust - GitHub https://github.com/apache/iceberg-rust
[13] When Doris Meets Iceberg: A Data Engineer's Redemption - DZone https://dzone.com/articles/when-doris-meets-iceberg
[14] Using Doris and Iceberg - Apache Doris https://doris.apache.org/docs/dev/lakehouse/best-practices/doris-iceberg/

Rollout/Adoption Plan

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

[ostinru]

@yjhjstz - it is really cool feature!

Couple questions:

1. what it expected time to 'go live' with this feature?
2. does it mean that PXF will gradually deprecated? or it can be integrated as one of the catalogs?

PS: it seems that links in References paragraph had lost.

[yjhjstz]

Question 1: What is the expected time to 'go live' with this feature?

We expect to deliver an initial working version of the multi-catalog system within 3-6 months, depending on community participation and contribution levels.

Our implementation approach will be:

• Phase 1 (Months 1-2): Core MVS infrastructure and catalog registry
• Phase 2 (Months 2-3): Iceberg catalog as the first implementation - this gives us immediate value for modern data lakehouse use cases
• Phase 3 (Months 3-4): FDW integration and basic query execution
• Phase 4 (Months 4-6): Performance optimization and production hardening

Why Iceberg First?

• High Demand: Iceberg is the most requested lakehouse format in the community
• Mature Ecosystem: Official Rust client (iceberg-rust) and well-defined REST catalog spec
• Proven Architecture: Apache Doris has demonstrated successful Iceberg integration patterns we can adapt
• Immediate Value: Users can query S3/cloud data lakes directly without data movement

We're committed to delivering this feature incrementally with regular community updates and feedback integration. The Iceberg catalog will serve as our foundation for expanding to other catalog types (Hive, Delta Lake, etc.) in subsequent releases.

Question 2: Does this mean PXF will be gradually deprecated, or can it be integrated as one of the catalogs?

PXF will NOT be deprecated. Instead, it will be integrated as one of the catalog plugins within the multi-catalog framework. This approach:

• Preserves existing PXF investments and user workflows
• Provides unified access through catalog.database.table syntax
• Enables gradual migration from file-based to metadata-rich catalog systems
• Supports hybrid architectures where both legacy and modern data coexist

Users will benefit from both PXF's mature connector ecosystem and the new metadata-aware lakehouse catalogs under a single, consistent interface.

Links are fixed.

[MisterRaindrop]

@yjhjstz it is really cool feature! but I have some questions

1. plugin interface layer
   Can you elaborate on this level? How does it ensure support for multiple cross-language translations?
2. Catalog
   Will the catalog integrate different authentication systems? Such as Kerberos, S3 IAM, AWS Glue, KMS？
3. Cross Catalog
   If two cross-catalogs point to the same path, for example, deploying two Cloudberry instances but pointing to the same path, the cache may become invalid.
4. Dynamic Discovery: On-demand metadata fetching from external sources
   Can you elaborate on this? Is this part something the user needs to define for dynamic discovery themselves?

[yjhjstz]

Plugin Interface Layer

• Exposes a C ABI with function pointers (e.g. initialize_catalog, get_table_schema, list_tables)
• Uses FFI and protobuf/JSON for language-neutral data exchange
• Supports plugins in Rust, Java, Python, Go via standardized callbacks

Catalog Authentication

• Yes, Pluggable AuthProvider interface

Cross-Catalog Cache Coordination

• Use snapshot-id to check is it the latest .

Dynamic Discovery (On-Demand Metadata Fetching)

• Discovered schemas are cached with TTL and snapshot-ID checks, refreshed lazily