|
2 | 2 | title: Storage |
3 | 3 | weight: 1 |
4 | 4 | --- |
| 5 | + |
5 | 6 | The research conducted with LingoDB does not focus on storage aspects of database systems. |
6 | 7 | Thus, LingoDB does not come with an optimized storage backend and currently does not provide transactional semantics. |
7 | 8 |
|
8 | 9 | ## In-Memory Format: Apache Arrow |
| 10 | + |
9 | 11 | The Apache Arrow columnar layout is used for the in-memory representation of tabular data. |
10 | | -Thus, LingoDB can exchange data with existing libraries and frameworks withoug any overhead and can directly query Apache Arrow tables. |
| 12 | +Thus, LingoDB can exchange data with existing libraries and frameworks withoug any overhead and can directly query |
| 13 | +Apache Arrow tables. |
11 | 14 |
|
12 | 15 | ## Persistent Storage |
| 16 | + |
13 | 17 | For many practical purposes, persistent storage is required. |
14 | 18 | We chose a pragmatic approach: |
15 | 19 |
|
16 | 20 | 1. Each database is represented by multiple files placed in one *database directory* |
17 | | -2. In this directory, each table is represented by multiple files, each starting with the name of the table: |
18 | | - 1. *name*`.metadata.json`: stores metadata relevant to LingoDB. This includes basic informations like column names and internal column types, but also statistics and available indices |
19 | | - 2. *name*`.arrow`: Stores the contents of the table using Apache Arrow's IPC-Format |
20 | | - 3. *name*`.arrow.sample`: Optionally stores an sample of up to 1024 rows randomly selected from the table. |
| 21 | +2. The main database file is called `db.lingodb` and contains the database catalog and metadata in a binary format |
| 22 | +3. At the moment, further files are used to store the table data (e.g., in Apache Arrow format) |
21 | 23 |
|
22 | | -Given the database directory, LingoDB automatically detects the available tables, loads the metadata, data, and samples. |
| 24 | +Given the database directory, LingoDB automatically loads the database catalog and metadata from the `db.lingodb` file. |
0 commit comments