Introduce virtual tables in Storage abstraction

[brfrn169]

Description

This PR introduces virtual tables in the Storage abstraction layer. Virtual tables allow exposing a logical join of two source tables on their primary key, enabling related data stored in separate tables to be accessed as a single logical entity. Currently, only the JDBC adapter supports it.

Key Features:

• Virtual Table Creation: New API methods in DistributedStorageAdmin to create virtual tables with configurable join types (INNER, LEFT_OUTER)
• Primary Key Join: Automatically joins two source tables based on their primary key columns
• Column Composition: Resulting virtual table includes [primary key columns] + [left non-key columns] + [right non-key columns]
• Validation: Comprehensive validation ensuring:
  • Primary key schema compatibility (columns, order, types)
  • No non-key column name conflicts
  • Source tables within storage atomicity unit (namespace-level or storage-level)
  • Virtual tables cannot reference other virtual tables as sources
• Metadata Management: Virtual table information caching and retrieval via VirtualTableInfo

Notes:

• This feature is marked as internal use only and subject to breaking changes
• Virtual tables require storage atomicity unit at namespace level or higher
• Currently, nested virtual tables (using virtual tables as sources) are not supported

JDBC adapter implementation details

• createVirtualTable()
  • Create a view joining the source tables on their primary key columns
  • Store the metadata of the virtual table

For example, suppose we have the following two tables:

CREATE TABLE table1 (
    col1 INT,
    col2 TEXT,
    col3 BIGINT,
    col4 FLOAT,
    PRIMARY KEY (col1, col2)
);

CREATE TABLE table2 (
    col1 INT,
    col2 TEXT,
    col5 DOUBLE,
    col6 BOOLEAN,
    PRIMARY KEY (col1, col2)
);

Then we create the following view based on these source tables:

CREATE VIEW virtual_table AS
SELECT
    t1.col1 AS col1,
    t1.col2 AS col2,
    t1.col3 AS col3,
    t1.col4 AS col4,
    t2.col5 AS col5,
    t2.col6 AS col6
FROM table1 t1
INNER JOIN table2 t2
    ON t1.col1 = t2.col1 AND t1.col2 = t2.col2;

Whether to use INNER JOIN or LEFT OUTER JOIN depends on the specified join type.

• getVirtualTableInfo()

  • Return the metadata of the virtual table

• getTableMetadata() for virtual tables

  • Merge the table metadata of the source tables, and return it

• getNamespaceTableNames()

  • Return the table list including the virtual tables

• dropTable()

  • For virtual tables
    • Drop the view
    • Delete the metadata of the virtual table
  • Note that users can't drop the source tables

• truncateTable() for virtual tables

  • Truncate the source tables

• createIndex() for virtual tables

  • Create an index on the source table

• dropIndex() for virtual tables

  • Drop an index on the source table

• get() for virtual tables

  • Get a record from the view

• scan() for virtual tables

  • Scan records from the view

• put() for virtual tables

  • Divide the put into two puts for the source tables
  • Execute the divided puts for the source tables

• delete() for virtual tables

  • Divide the delete into two deletes for the source tables
  • Execute the divided deletes for the source tables

• mutate() for virtual tables

  • Same as put() and delete()

For read operations, we execute the reads against the view, but for write operations, we directly update the source tables. This is because most databases don't support updatable views or have significant limitations.

Currently, some administrative operations for virtual tables and source tables are not supported.

Related issues and/or PRs

N/A

Changes made

• New Methods in DistributedStorageAdmin:
  • createVirtualTable(): Multiple overloads with options for ifNotExists and custom options
  • getVirtualTableInfo(): Retrieve virtual table metadata
• Common Layer: CommonDistributedStorageAdmin provides comprehensive validation logic
• JDBC Support: Full implementation in JdbcAdmin with view creation
• New integration test base class: DistributedStorageVirtualTablesIntegrationTestBase
• Comprehensive test coverage for:
  • Virtual table creation with different join types
  • Schema validation
  • Error handling for invalid configurations
  • Get/Scan operations on virtual tables

Checklist

The following is a best-effort checklist. If any items in this checklist are not applicable to this PR or are dependent on other, unmerged PRs, please still mark the checkboxes after you have read and understood each item.

• I have commented my code, particularly in hard-to-understand areas.
• I have updated the documentation to reflect the changes.
• I have considered whether similar issues could occur in other products, components, or modules if this PR is for bug fixes.
• Any remaining open issues linked to this PR are documented and up-to-date (Jira, GitHub, etc.).
• Tests (unit, integration, etc.) have been added for the changes.
• My changes generate no new warnings.
• Any dependent changes in other PRs have been merged and published.

Additional notes (optional)

N/A

Release notes

Introduced virtual tables in the Storage abstraction layer. Virtual tables allow exposing a logical join of two source tables on their primary key, enabling related data stored in separate tables to be accessed as a single logical entity.

[gemini-code-assist]

Summary of Changes

Hello @brfrn169, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances the Storage abstraction by introducing virtual tables, enabling the logical joining of two source tables on their primary keys. This feature simplifies data access for related entities stored separately, providing a unified view. It includes new administrative APIs for creation and metadata retrieval, backed by extensive validation and a dedicated metadata management service. The JDBC adapter fully implements this functionality, while other adapters are updated to reflect its current unavailability.

Highlights

• Introduction of Virtual Tables: This pull request introduces the concept of virtual tables within the Storage abstraction layer, allowing users to define a logical join of two source tables based on their primary keys, which can then be accessed as a single entity.
• New API Methods: New API methods createVirtualTable() and getVirtualTableInfo() have been added to DistributedStorageAdmin to manage virtual table creation and retrieval of their metadata.
• Comprehensive Validation: The system now includes robust validation checks for virtual table creation, ensuring primary key schema compatibility, preventing non-key column name conflicts, verifying source tables reside within the same atomicity unit, and disallowing nested virtual tables (virtual tables as sources).
• JDBC Adapter Support: The JDBC adapter has been updated to fully support virtual tables, including the creation of database views and management of their metadata. Other storage adapters (Cassandra, Cosmos, Dynamo, ObjectStorage) have been updated to explicitly mark virtual tables as unsupported.
• Operation Handling for Virtual Tables: Operations like put, delete, and mutate are now handled for virtual tables by intelligently dividing them into corresponding operations on their underlying source tables, including conditional mutations.
• Metadata Management: A new VirtualTableInfoManager and VirtualTableMetadataService have been introduced to efficiently manage and cache virtual table metadata, ensuring consistent behavior across operations.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces virtual tables, a significant and valuable feature for the storage abstraction layer. The implementation is comprehensive, covering API changes, validation logic, JDBC support, and extensive testing. The code is well-structured, and the validation for creating virtual tables is thorough, covering schema compatibility, name conflicts, and atomicity constraints. The handling of operations like put, delete, scan, and get on virtual tables in the JDBC adapter is well-designed.

I have a couple of minor suggestions for improving maintainability.

[Copilot]

Pull Request Overview

This PR introduces virtual tables in the Storage abstraction layer, enabling a logical join of two source tables on their primary key. This allows related data stored in separate tables to be accessed as a single logical entity.

Key Changes:

• New API methods in DistributedStorageAdmin for creating and querying virtual tables with support for INNER and LEFT_OUTER joins
• Comprehensive validation logic in CommonDistributedStorageAdmin to ensure schema compatibility and prevent invalid configurations
• Full JDBC implementation via JdbcAdmin using database views, with metadata management through VirtualTableMetadataService
• Enhanced mutation handling in JdbcDatabase to automatically divide Put/Delete operations for virtual tables into corresponding operations on source tables
• New integration test suite DistributedStorageVirtualTablesIntegrationTestBase with comprehensive test coverage

Reviewed Changes

Copilot reviewed 26 out of 26 changed files in this pull request and generated 30 comments.

Show a summary per file

File	Description
DistributedStorageAdmin.java	Adds virtual table API methods with multiple overloads
VirtualTableInfo.java & VirtualTableJoinType.java	New interfaces defining virtual table metadata and join types
CommonDistributedStorageAdmin.java	Implements validation logic for virtual table creation
JdbcAdmin.java	JDBC implementation with view creation, metadata management, and special handling for truncate/drop/index operations
JdbcDatabase.java	Mutation handling that divides virtual table operations into source table operations
VirtualTableMetadataService.java	Manages virtual table metadata storage in JDBC
VirtualTableInfoManager.java	Caches virtual table information with configurable expiration
MultiStorageAdmin.java	Validates virtual tables and source tables reside in same storage
DistributedStorageVirtualTablesIntegrationTestBase.java	Comprehensive integration tests for virtual table functionality
CoreError.java	Adds error codes for virtual table validation failures

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[feeblefakie]

Overall, looking good!
Left some comments and suggestions.
PTAL!

[feeblefakie]

LGTM! Thank you!

[Torch3333]

LGTM, thank you!

[komamitsu]

LGTM, thank you!