docs: add how to write a system table

Signed-off-by: Chojan Shang <[email protected]>

Author: PsiACE

docs/doc/90-contributing/10-how-to-write-a-system-table.md

@@ -0,0 +1,176 @@
+---
+title: How to Write a System Table
+---
+
+System tables are special tables that provide information about Databend's internal state, such as databases, tables, functions, settings, etc. In this document, we will show you how to write a new system table for Databend using the credits table as an example.
+
+The credits table returns information about the upstream dependencies used by Databend, including their names, versions and licenses.
+
+## Prerequisites
+
+To write a new system table for Databend, you need to have some basic knowledge of Rust programming language and Databend's code structure.
+
+## Location
+
+The existing system tables for Databend are located in the `query/storage` directory. You should place your new system table file in this directory as well, unless there are some special build reasons that prevent you from doing so. In that case, you can temporarily place it in the `service/databases/system` directory (not recommended).
+
+## Definition
+
+The definition of a system table mainly focuses on two aspects: one is the table information, which includes the table `name` and `schema`, etc.; the other is the data generation/retrieval logic for the table content. These two aspects correspond to two traits: `SyncSystemTable` and `AsyncSystemTable`. You need to implement one of these traits depending on whether your data retrieval involves asynchronous function calls or not.
+
+## Implementation
+
+In this section, we will walk through the implementation of the credits table step by step. The code file is located at `credits_table.rs`.
+
+Firstly, you need to define a struct for your system table that contains only the fields for storing the table information. For example:
+
+```rust
+pub struct CreditsTable {
+    table_info: TableInfo,
+}
+```
+
+Next, you need to implement a create method for your system table struct that takes a `table_id` as an argument and returns an `Arc<dyn Table>`. The `table_id` is generated by `sys_db_meta.next_table_id()` when creating a new system table.
+
+```rust
+pub fn create(table_id: u64) -> Arc<dyn Table>
+```
+
+Inside this method, you need to define a schema for your system table using `TableSchemaRefExt` and `TableField`. The schema describes the structure of your system table with field names and types depending on the data you want to store in it.
+
+For example:
+
+```rust
+let schema = TableSchemaRefExt::create(vec![
+    TableField::new("name", TableDataType::String),
+    TableField::new("version", TableDataType::String),
+    TableField::new("license", TableDataType::String),
+]);
+```
+
+For string-type data, you can use `TableDataType::String`; other basic types are similar. But if you need to allow null values in your field, such as an optional 64-bit unsigned integer field, you can use `TableDataType::Nullable(Box::new(TableDataType::Number(NumberDataType::UInt64)))` instead. `TableDataType::Nullable` indicates that null values are allowed; `TableDataType::Number(NumberDataType::UInt64)` represents that the type is 64-bit unsigned integer.
+
+After defining the schema, you need to define some metadata for your system table, such as description (`desc`), `name`, `meta`, etc. You can follow other existing examples and fill in these fields accordingly.
+
+For example:
+
+```rust
+let table_info = TableInfo {
+    desc: "'system'.'credits'".to_string(),
+    name: "credits".to_string(),
+    ident: TableIdent::new(table_id, 0),
+    meta: TableMeta {
+        schema,
+        engine: "SystemCredits".to_string(),
+        ..Default::default()
+    },
+   ..Default::default()
+};
+
+SyncOneBlockSystemTable::create(CreditsTable { table_info })
+```
+
+Finally, you need to create an instance of your system table struct with these fields and wrap it with either `SyncOneBlockSystemTable` or `AsyncOneBlockSystemTable` depending on whether your data retrieval logic is synchronous or asynchronous.
+
+Next, you need to implement either `SyncSystemTable` or `AsyncSystemTable` trait for your system table struct. `SyncSystemTable` requires you to define `NAME` constant and implement four methods: `get_table_info()`, `get_full_data()`, `get_partitions()` and `truncate()`. However, the last two methods have default implementations, so you don't need to implement them yourself in most cases. (`AsyncSystemTable` is similar, but it doesn't have `truncate()` method.)
+
+`NAME` constant follows the format of `system.<name>`.
+
+```rust
+const NAME: &'static str = "system.credits";
+```
+
+`get_table_info()` method returns the table information stored in the struct.
+
+```rust
+fn get_table_info(&self) -> &TableInfo {
+    &self.table_info
+}
+```
+
+`get_full_data()` method is the most important part, because it contains the logic for generating or retrieving the data for your system table. The credits table has three fields that are similar, so we will only show the license field as an example.
+
+The license field information is obtained from an environment variable named `DATABEND_CREDITS_LICENSES` (see `common-building`). Each data item is separated by a comma.
+
+String-type columns are eventually converted from `Vec<Vec<u8>>`, where each string needs to be converted to `Vec<u8>`. So we use `.as_bytes().to_vec()` to do this conversion when iterating over the data.
+
+```rust
+let licenses: Vec<Vec<u8>> = env!("DATABEND_CREDITS_LICENSES")
+    .split_terminator(',')
+    .map(|x| x.trim().as_bytes().to_vec())
+    .collect();
+```
+
+After getting all the data, you can return them in a `DataBlock` format. For non-null types, use `from_data`; for nullable types, use `from_opt_data`.
+
+For example:
+
+```rust
+Ok(DataBlock::new_from_columns(vec![
+    StringType::from_data(names),
+    StringType::from_data(versions),
+    StringType::from_data(licenses),
+]))
+```
+
+Lastly, if you want to integrate your system table into Databend, you also need to edit `system_database.rs` and register it to `SystemDatabase`.
+
+```rust
+impl SystemDatabase {
+    pub fn create(sys_db_meta: &mut InMemoryMetas, config: &Config) -> Self {
+    ...
+        CreditsTable::create(sys_db_meta.next_table_id()),
+    ...
+    }
+}
+```
+
+## Testing
+
+The tests for system tables are currently located at `tests/it/storages/system.rs`.
+
+For tables whose content does not change frequently, you can use Golden File testing. Its logic is to write the corresponding table into a specified file and compare it with an expected file. If they match, then the test passes; otherwise, it fails.
+
+For example:
+
+```rust
+#[tokio::test(flavor = "multi_thread")]
+async fn test_columns_table() -> Result<()> {
+    let (_guard, ctx) = crate::tests::create_query_context().await?;
+
+    let mut mint = Mint::new("tests/it/storages/testdata");
+    let file = &mut mint.new_goldenfile("columns_table.txt").unwrap();
+    let table = ColumnsTable::create(1);
+
+    run_table_tests(file, ctx, table).await?;
+    Ok(())
+}
+```
+
+For tables whose content may change dynamically or depend on external factors, there is a lack of sufficient testing methods. You can choose to test the parts that have relatively fixed patterns, such as the number of rows and columns; or you can verify whether the output contains specific content.
+
+For example:
+
+```rust
+#[tokio::test(flavor = "multi_thread")]
+async fn test_metrics_table() -> Result<()> {
+	...
+    let result = stream.try_collect::<Vec<_>>().await?;
+    let block = &result[0];
+    assert_eq!(block.num_columns(), 4);
+    assert!(block.num_rows() >= 1);
+
+    let output = pretty_format_blocks(result.as_slice())?;
+    assert!(output.contains("test_test_metrics_table_count"));
+    #[cfg(feature = "enable_histogram")]
+    assert!(output.contains("test_test_metrics_table_histogram"));
+
+    Ok(())
+}
+```
+
+## Summary
+
+In this document, we have shown you how to write a new system table for Databend using the credits table as an example. We hope this document helps you understand the basic steps and principles of creating a system table for Databend. If you have any questions or feedback, please feel free to contact us on GitHub or Slack. Thank you for your interest and contribution to Databend!
+
+