Merge pull request #12 from cmu-db/detailed-entity-docs

Detailed Entity Docs

Author: connortsui20

optd-persistent/src/migrator/memo/m20241029_000001_cascades_group.rs

@@ -1,3 +1,70 @@
+//! An entity representing a group / equivalence class in the Cascades framework.
+//!
+//! Quoted from the Microsoft article _Extensible query optimizers in practice_:
+//!
+//! > In the memo, each class of equivalent expressions is called an equivalent class or a group,
+//! > and all equivalent expressions within the class are called group expressions or simply
+//! > expressions.
+//!
+//! A Cascades group is defined as a class of equivalent logical or physical expressions. The
+//! Cascades framework uses these groups as a way of storing the best query sub-plans for use in the
+//! dynamic programming search algorithm.
+//!
+//! For example, a Cascades group could be the set of expressions containing the logical expressions
+//! `Join(A, B)` and `Join(B, A)`, as well as the physical expressions `HashJoin(A, B)` and
+//! `NestedLoopJoin(B, A)`.
+//!
+//! # Columns
+//!
+//! Each group is assigned a monotonically-increasing (unique) ID. This ID will be important since
+//! there are many foreign key references from other tables to `cascades_group`.
+//!
+//! We additionally store a `latest_winner` foreign key reference to a physical expression. See
+//! the [section](#best-physical-plan-winner) below for more details.
+//!
+//! Finally, we store `in_progress` and `is_optimized` flags that are used for quickly determining
+//! the state of optimization for this group during the dynamic programming search.
+//!
+//! # Entity Relationships
+//!
+//! ### Child Expressions (Logical and Physical)
+//!
+//! To retrieve all of a `cascades_group`'s equivalent expressions, you must query the
+//! [`logical_expression`] or the [`physical_expression`] entities via their foreign keys to
+//! `cascades_group`. The relationship between [`logical_expression`] and `cascades_group` is
+//! many-to-one, and the exact same many-to-one relationship is held for [`physical_expression`] to
+//! `cascades_group`.
+//!
+//! ### Parent Expressions (Logical and Physical)
+//!
+//! Additionally, each logical or physical expression can have any number of `cascades_group`s as
+//! children, and a group can be a child of any expression. Thus, `cascades_group` additionally has
+//! a many-to-many relationship with [`logical_expression`] and [`physical_expression`] via the
+//! [`logical_children`] and [`physical_children`] entities.
+//!
+//! To reiterate, `cascades_group` has **both** a one-to-many **and** a many-to-many relationship
+//! with both [`logical_expression`] and [`physical_expression`]. This is due to groups being both
+//! parents and children of expressions.
+//!
+//! ### Best Physical Plan (Winner)
+//!
+//! The `cascades_group` entity also stores a `latest_winner` _nullable_ foreign key reference to
+//! a physical expression. This represents the most recent best query plan we have computed. The
+//! reason it is nullable is because we may not have come up with any best query plan yet.
+//!
+//! ### Logical Properties
+//!
+//! Lastly, each `cascades_group` record will have a set of logical properties store in the
+//! [`logical_property`] entity, where there is an many-to-one relationship from
+//! [`logical_property`] to `cascades_group`. Note that we do not store physical properties directly
+//! on the `cascades_group`, but rather we store them for each [`physical_expression`] record.
+//!
+//! [`logical_expression`]: super::logical_expression
+//! [`physical_expression`]: super::physical_expression
+//! [`logical_children`]: super::logical_children
+//! [`physical_children`]: super::physical_children
+//! [`logical_property`]: super::logical_property
+
 use crate::migrator::memo::physical_expression::PhysicalExpression;
 use sea_orm_migration::{prelude::*, schema::*};
 

optd-persistent/src/migrator/memo/m20241029_000001_group_winner.rs

@@ -1,3 +1,35 @@
+//! An entity representing a the best physical plan (or "winner") of a Cascades group.
+//!
+//! In the Cascades framework, query optimization is done through dynamic programming that is based
+//! on the assumption that the cost model satisfies the _principle of optimality_. Quoted from the
+//! Microsoft article _Extensible query optimizers in practice_:
+//!
+//! > ... in the search space of linear sequence of joins, the optimal plan for a join of n
+//! > relations can be found by extending the optimal plan of a sub-expression of n - 1 joins with
+//! > an additional join.
+//!
+//! By storing the best sub-plans / [`physical_expression`]s of smaller Cascades groups, we can
+//! build up an optimal query plan.
+//!
+//! This entity represents the best plan sub-tree for a specific group. However, we store multiple
+//! winners over different epochs, as changes to the database may require us to re-evaluate what the
+//! optimal sub-plan is.
+//!
+//! # Columns
+//!
+//! Other than the primary key, all of the columns in this relation are foreign keys to other
+//! tables.
+//!
+//! A group winner is defined by the [`cascades_group`] it belongs to (`group_id`), the unique ID of
+//! the [`physical_expression`] (`physical_expression_id`), the ID of the cost record in the
+//! [`plan_cost`] table (`cost_id`), and the monotonically-increasing epoch ID in the [`event`]
+//! table (`epoch_id`).
+//!
+//! [`cascades_group`]: super::cascades_group
+//! [`physical_expression`]: super::physical_expression
+//! [`plan_cost`]: super::super::cost_model::plan_cost
+//! [`event`]: super::super::cost_model::event
+
 use crate::migrator::cost_model::{event::Event, plan_cost::PlanCost};
 use crate::migrator::memo::{
     cascades_group::CascadesGroup, physical_expression::PhysicalExpression,

optd-persistent/src/migrator/memo/m20241029_000001_logical_children.rs

@@ -1,3 +1,15 @@
+//! An entity representing the [`cascades_group`] children of every [`logical_expression`].
+//!
+//! Formally, this entity is a junction which allows us to represent a many-to-many relationship
+//! between [`logical_expression`] and [`cascades_group`]. Expressions can have any number of child
+//! groups, and every group can be a child of many different expressions, hence the many-to-many
+//! relationship.
+//!
+//! See [`cascades_group`] for more details.
+//!
+//! [`cascades_group`]: super::cascades_group
+//! [`logical_expression`]: super::logical_expression
+
 use crate::migrator::memo::{cascades_group::CascadesGroup, logical_expression::LogicalExpression};
 use sea_orm_migration::{prelude::*, schema::*};
 

optd-persistent/src/migrator/memo/m20241029_000001_logical_expression.rs

@@ -1,3 +1,40 @@
+//! An entity representing a logical plan expression in the Cascades framework.
+//!
+//! Quoted from the Microsoft article _Extensible query optimizers in practice_:
+//!
+//! > A logical expression is defined as a tree of logical operators, and corresponds to a
+//! > relational algebraic expression.
+//!
+//! In the Cascades query optimization framework, the memo table stores equivalence classes of
+//! expressions (see [`cascades_group`]). These equivalence classes, or "groups", store both
+//! `logical_expression`s and [`physical_expression`]s.
+//!
+//! Optimization starts by "exploring" equivalent logical expressions within a group. For example,
+//! the logical expressions `Join(A, B)` and `Join(B, A)` are contained in the same group. The
+//! logical expressions are defined as a `Join` operator with the groups representing a scan of
+//! table `A` and a scan of table `B` as its children.
+//!
+//! # Columns
+//!
+//! Each `logical_expression` has a unique primary key ID, but it holds little importance other than
+//! helping distinguish between two different expressions.
+//!
+//! The more interesting column is the `fingerprint` column, in which we store a hashed fingerprint
+//! value that can be used to efficiently check equality between two potentially equivalent logical
+//! expressions (hash-consing). See ???TODO??? for more information on expression fingerprints.
+//!
+//! Finally, since there are many different types of operators, we store a variant tag and a data
+//! column as JSON to represent the semi-structured data fields of logical operators.
+//!
+//! # Entity Relationships
+//!
+//! The only relationship that `logical_expression` has is to [`cascades_group`]. It has **both** a
+//! one-to-many **and** a many-to-many relationship with [`cascades_group`], and you can see more
+//! details about this in the module-level documentation for [`cascades_group`].
+//!
+//! [`cascades_group`]: super::cascades_group
+//! [`physical_expression`]: super::physical_expression
+
 use crate::migrator::memo::cascades_group::CascadesGroup;
 use sea_orm_migration::{prelude::*, schema::*};
 

optd-persistent/src/migrator/memo/m20241029_000001_logical_property.rs

@@ -1,3 +1,8 @@
+//! An entity representing a logical property of a Cascades group.
+//!
+//! TODO what exactly are we storing in here?
+//! TODO why is it linked to only cascades groups and not logical expressions?
+
 use crate::migrator::memo::cascades_group::CascadesGroup;
 use sea_orm_migration::{prelude::*, schema::*};
 

optd-persistent/src/migrator/memo/m20241029_000001_physical_children.rs

@@ -1,3 +1,15 @@
+//! An entity representing the [`cascades_group`] children of every [`physical_expression`].
+//!
+//! Formally, this entity is a junction which allows us to represent a many-to-many relationship
+//! between [`physical_expression`] and [`cascades_group`]. Expressions can have any number of child
+//! groups, and every group can be a child of many different expressions, hence the many-to-many
+//! relationship.
+//!
+//! See [`cascades_group`] for more details.
+//!
+//! [`cascades_group`]: super::cascades_group
+//! [`physical_expression`]: super::physical_expression
+
 use crate::migrator::memo::{
     cascades_group::CascadesGroup, physical_expression::PhysicalExpression,
 };

optd-persistent/src/migrator/memo/m20241029_000001_physical_expression.rs

@@ -1,3 +1,41 @@
+//! An entity representing a logical plan expression in the Cascades framework.
+//!
+//! Quoted from the Microsoft article _Extensible query optimizers in practice_:
+//!
+//! > A physical expression is a tree of physical operators, which is also referred to as the
+//! > _physical plan_ or simply _plan_.
+//!
+//! In the Cascades query optimization framework, the memo table stores equivalence classes of
+//! expressions (see [`cascades_group`]). These equivalence classes, or "groups", store both
+//! [`logical_expression`]s and `physical_expression`s.
+//!
+//! Optimization starts by exploring equivalent logical expressions within a group, and then it
+//! proceeds to implement / optimize those logical operators into physical operators. For example,
+//! the logical expression `Join(A, B)` could be implemented into a `HashJoin(A, B)` or a
+//! `NestedLoopJoin(A, B)`, and both of these new physical expressions would be contained in the
+//! same group.
+//!
+//! # Columns
+//!
+//! Each `physical_expression` has a unique primary key ID, and other tables will store a foreign
+//! key reference to a specific `physical_expression`s.
+//!
+//! The more interesting column is the `fingerprint` column, in which we store a hashed fingerprint
+//! value that can be used to efficiently check equality between two potentially equivalent physical
+//! expressions (hash-consing). See ???TODO??? for more information on expression fingerprints.
+//!
+//! Finally, since there are many different types of operators, we store a variant tag and a data
+//! column as JSON to represent the semi-structured data fields of logical operators.
+//!
+//! # Entity Relationships
+//!
+//! The only relationship that `physical_expression` has is to [`cascades_group`]. It has **both** a
+//! one-to-many **and** a many-to-many relationship with [`cascades_group`], and you can see more
+//! details about this in the module-level documentation for [`cascades_group`].
+//!
+//! [`cascades_group`]: super::cascades_group
+//! [`logical_expression`]: super::logical_expression
+
 use crate::migrator::memo::cascades_group::CascadesGroup;
 use sea_orm_migration::{prelude::*, schema::*};
 

optd-persistent/src/migrator/memo/m20241029_000001_physical_property.rs

@@ -1,3 +1,8 @@
+//! An entity representing a physical property of a physical expression in the Cascades framework.
+//!
+//! TODO what exactly are we storing in here?
+//! TODO why is it linked to only physical expressions and not cascades groups?
+
 use crate::migrator::memo::physical_expression::PhysicalExpression;
 use sea_orm_migration::{prelude::*, schema::*};
 

optd-persistent/src/migrator/memo/mod.rs

@@ -1,3 +1,6 @@
+//! Entities related to the memo table used for dynamic programming in the Cascades query
+//! optimization framework.
+
 pub(crate) mod m20241029_000001_cascades_group;
 pub(crate) mod m20241029_000001_group_winner;
 pub(crate) mod m20241029_000001_logical_children;