DDL Implementation

DDL Implementation

Current State

Transactional Catalog

The existing transactional catalog solves several concurrency problems for us by leveraging the existing MVCC semantics of the system. Specifically, many concurrent DDL conflicts (name collisions, modifying the same table, etc.) are solved by our existing write-write and index conflict detection mechanisms. Therefore, the primary challenge we face is managing physical representations of logical elements (compiled functions, etc.) and implicit write-write conflicts (adding an index while another transaction modifies the underlying table).

Lazy DDL at Storage Layer

There was a proof of concept of how to implement this from last spring. However, the implementation has fallen significantly behind the current version of terrier. It was performant enough that it should probably serve as the reference point for implementing DDL at the storage layer, but much of the actual code has been obsoleted. The specific design points to bring forward are:

1. Fast-path for single version operations (common case)
2. Header mangling for projections (fast translation between versions)
3. Partial serialization of transactions (all transactions can be labeled as before or after a given DDL change)
4. SqlTable applies appropriate default values to "old" tables

Challenges

• Add index

• Concurrent Index DDL and Table DDL

• Maintaining logical-physical consistency