Improve the DML / DDL Documentation (#16115)

* Update documentation about DDL and DML

* Improve the DML Documentation

* Apply suggestions from code review

Co-authored-by: Oleks V <[email protected]>

* Fix docs

* Fix docs

---------

Co-authored-by: Oleks V <[email protected]>

Author: alamb

benchmarks/README.md

@@ -33,8 +33,8 @@ benchmarks that compare performance with other engines. For example:
 - [ClickBench] scripts are in the [ClickBench repo](https://github.com/ClickHouse/ClickBench/tree/main/datafusion)
 - [H2o.ai `db-benchmark`] scripts are in [db-benchmark](https://github.com/apache/datafusion/tree/main/benchmarks/src/h2o.rs)
 
-[ClickBench]: https://github.com/ClickHouse/ClickBench/tree/main
-[H2o.ai `db-benchmark`]: https://github.com/h2oai/db-benchmark
+[clickbench]: https://github.com/ClickHouse/ClickBench/tree/main
+[h2o.ai `db-benchmark`]: https://github.com/h2oai/db-benchmark
 
 # Running the benchmarks
 

benchmarks/queries/clickbench/README.md

@@ -9,7 +9,7 @@ ClickBench is focused on aggregation and filtering performance (though it has no
 - `queries.sql` - Actual ClickBench queries, downloaded from the [ClickBench repository]
 - `extended.sql` - "Extended" DataFusion specific queries.
 
-[ClickBench repository]: https://github.com/ClickHouse/ClickBench/blob/main/datafusion/queries.sql
+[clickbench repository]: https://github.com/ClickHouse/ClickBench/blob/main/datafusion/queries.sql
 
 ## "Extended" Queries
 

datafusion/expr/src/logical_plan/dml.rs

@@ -89,8 +89,28 @@ impl Hash for CopyTo {
     }
 }
 
-/// The operator that modifies the content of a database (adapted from
-/// substrait WriteRel)
+/// Modifies the content of a database
+///
+/// This operator is used to perform DML operations such as INSERT, DELETE,
+/// UPDATE, and CTAS (CREATE TABLE AS SELECT).
+///
+/// * `INSERT` - Appends new rows to the existing table. Calls
+///   [`TableProvider::insert_into`]
+///
+/// * `DELETE` - Removes rows from the table. Currently NOT supported by the
+///   [`TableProvider`] trait or builtin sources.
+///
+/// * `UPDATE` - Modifies existing rows in the table. Currently NOT supported by
+///   the [`TableProvider`] trait or builtin sources.
+///
+/// * `CREATE TABLE AS SELECT` - Creates a new table and populates it with data
+///   from a query. This is similar to the `INSERT` operation, but it creates a new
+///   table instead of modifying an existing one.
+///
+/// Note that the structure is adapted from substrait WriteRel)
+///
+/// [`TableProvider`]: https://docs.rs/datafusion/latest/datafusion/datasource/trait.TableProvider.html
+/// [`TableProvider::insert_into`]: https://docs.rs/datafusion/latest/datafusion/datasource/trait.TableProvider.html#method.insert_into
 #[derive(Clone)]
 pub struct DmlStatement {
     /// The table name
@@ -177,11 +197,18 @@ impl PartialOrd for DmlStatement {
     }
 }
 
+/// The type of DML operation to perform.
+///
+/// See [`DmlStatement`] for more details.
 #[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Hash)]
 pub enum WriteOp {
+    /// `INSERT INTO` operation
     Insert(InsertOp),
+    /// `DELETE` operation
     Delete,
+    /// `UPDATE` operation
     Update,
+    /// `CREATE TABLE AS SELECT` operation
     Ctas,
 }
 

docs/source/library-user-guide/catalogs.md

@@ -23,11 +23,14 @@ This section describes how to create and manage catalogs, schemas, and tables in
 
 ## General Concepts
 
-CatalogProviderList, Catalogs, schemas, and tables are organized in a hierarchy. A CatalogProviderList contains catalog providers, a catalog provider contains schemas and a schema contains tables.
+Catalog providers, catalogs, schemas, and tables are organized in a hierarchy. A `CatalogProviderList` contains `CatalogProvider`s, a `CatalogProvider` contains `SchemaProviders` and a `SchemaProvider` contains `TableProvider`s.
 
 DataFusion comes with a basic in memory catalog functionality in the [`catalog` module]. You can use these in memory implementations as is, or extend DataFusion with your own catalog implementations, for example based on local files or files on remote object storage.
 
+DataFusion supports DDL queries (e.g. `CREATE TABLE`) using the catalog API described in this section. See the [TableProvider] section for information on DML queries (e.g. `INSERT INTO`).
+
 [`catalog` module]: https://docs.rs/datafusion/latest/datafusion/catalog/index.html
+[tableprovider]: ./custom-table-providers.md
 
 Similarly to other concepts in DataFusion, you'll implement various traits to create your own catalogs, schemas, and tables. The following sections describe the traits you'll need to implement.
 

docs/source/library-user-guide/custom-table-providers.md

@@ -19,17 +19,22 @@
 
 # Custom Table Provider
 
-Like other areas of DataFusion, you extend DataFusion's functionality by implementing a trait. The `TableProvider` and associated traits, have methods that allow you to implement a custom table provider, i.e. use DataFusion's other functionality with your custom data source.
+Like other areas of DataFusion, you extend DataFusion's functionality by implementing a trait. The [`TableProvider`] and associated traits allow you to implement a custom table provider, i.e. use DataFusion's other functionality with your custom data source.
 
-This section will also touch on how to have DataFusion use the new `TableProvider` implementation.
+This section describes how to create a [`TableProvider`] and how to configure DataFusion to use it for reading.
 
 ## Table Provider and Scan
 
-The `scan` method on the `TableProvider` is likely its most important. It returns an `ExecutionPlan` that DataFusion will use to read the actual data during execution of the query.
+The [`TableProvider::scan`] method reads data from the table and is likely the most important. It returns an [`ExecutionPlan`] that DataFusion will use to read the actual data during execution of the query. The [`TableProvider::insert_into`] method is used to `INSERT` data into the table.
 
 ### Scan
 
-As mentioned, `scan` returns an execution plan, and in particular a `Result<Arc<dyn ExecutionPlan>>`. The core of this is returning something that can be dynamically dispatched to an `ExecutionPlan`. And as per the general DataFusion idea, we'll need to implement it.
+As mentioned, [`TableProvider::scan`] returns an execution plan, and in particular a `Result<Arc<dyn ExecutionPlan>>`. The core of this is returning something that can be dynamically dispatched to an `ExecutionPlan`. And as per the general DataFusion idea, we'll need to implement it.
+
+[`tableprovider`]: https://docs.rs/datafusion/latest/datafusion/datasource/trait.TableProvider.html
+[`tableprovider::scan`]: https://docs.rs/datafusion/latest/datafusion/datasource/trait.TableProvider.html#tymethod.scan
+[`tableprovider::insert_into`]: https://docs.rs/datafusion/latest/datafusion/datasource/trait.TableProvider.html#tymethod.insert_into
+[`executionplan`]: https://docs.rs/datafusion/latest/datafusion/physical_plan/trait.ExecutionPlan.html
 
 #### Execution Plan