Version doc for v0.0.4

Author: jungmair

versioned_docs/version-0.0.4/Design/Storage.md

@@ -0,0 +1,24 @@
+---
+title: Storage
+weight: 1
+---
+
+The research conducted with LingoDB does not focus on storage aspects of database systems.
+Thus, LingoDB does not come with an optimized storage backend and currently does not provide transactional semantics.
+
+## In-Memory Format: Apache Arrow
+
+The Apache Arrow columnar layout is used for the in-memory representation of tabular data.
+Thus, LingoDB can exchange data with existing libraries and frameworks withoug any overhead and can directly query
+Apache Arrow tables.
+
+## Persistent Storage
+
+For many practical purposes, persistent storage is required.
+We chose a pragmatic approach:
+
+1. Each database is represented by multiple files placed in one *database directory*
+2. The main database file is called `db.lingodb` and contains the database catalog and metadata in a binary format
+3. At the moment, further files are used to store the table data (e.g., in Apache Arrow format)
+
+Given the database directory, LingoDB automatically loads the database catalog and metadata from the `db.lingodb` file.

versioned_docs/version-0.0.4/Design/_index.md

@@ -0,0 +1,7 @@
+---
+title: Design
+type: docs
+weight: 4
+---
+
+This section gives an overview over the overall design of LingoDB.

versioned_docs/version-0.0.4/ForDevelopers/Contributing.md

@@ -0,0 +1,44 @@
+
+LingoDB is an open-source project that welcomes contributions from the community.
+However, it is also a research project that still undergoes major changes (often not in public repositories) that might conflict with your contributions.
+Furthermore, the project is developed by a very small team of researchers and students, which means that we have limited resources to review and merge pull requests.
+Finally, we have to ensure that the codebase stays maintainable and that the project's goals are met.
+Thus, please follow the guidelines below when planning to contribute to LingoDB.
+
+### Micro-Changes such as fixing typos, etc
+If you find a small typo or similar in one of the LingoDB repositories, please open an *Issue* in the respective repository.
+We won't accept pull requests for such small changes, but we will be happy to fix them ourselves as soon as possible.
+
+Examples:
+* Typos
+* Slight rephrasing of existing sentences
+* Updating npm dependencies
+* ...
+
+### Medium-sized Changes: Create a Pull Request
+If you want to contribute a medium-sized change, please create a pull request in the respective repository.
+
+Examples:
+* Any changes to the documentation
+* Bug-Fixes that do not require large changes/redesign (e.g., fixing a segfault)
+* Smallish new features (e.g., adding a new command line option, adding a new SQL function (e.g., `sin`))
+* Adding new tests
+
+### Large Changes: Discuss first
+If you want to contribute a larger change, please open an issue in the respective repository first.
+This way, we can discuss the change before you start working on it and we can avoid situations like:
+* You working on a feature that is already in development
+* You working on a feature that is not in line with the project's goals and won't be merged
+* You working on a feature that will not be working soon due to other changes in the project
+
+Examples:
+* Add a new compilation backend/target
+* Refactor the SQL parser
+* Refactorings
+* Larger features that touch the code base in many places
+* Anything that is more "researchy"
+
+### Before Creating a Pull Request
+Before creating a pull request, please make sure that
+* the CI pipeline passes and the coverage does not decrease.
+* the code is formatted according to the `.clang-format` file in the repository

versioned_docs/version-0.0.4/ForDevelopers/Debugging.md

@@ -0,0 +1,60 @@
+---
+title: Debugging & Profiling
+---
+
+Compared to interpreted execution engines, compiling engines come with many advantages but also some challenges.
+Especially debugging and profiling can become a challenge, as one not only needs to debug and profile the engine code, but also the generated code.
+Possible solutions to these problems have been discussed before for debugging [Hyper](https://ieeexplore.ieee.org/document/8667737) and [Umbra](https://dl.acm.org/doi/abs/10.1145/3395032.3395321) and [profiling Umbra](https://dl.acm.org/doi/abs/10.1145/3447786.3456254).
+
+## Guide: Profiling queries
+For profiling queries LingoDB comes with a *ct* tool that collects several metrics.
+For the following instructions, we assume that LingoDB was built in Release mode with debugging informations (`build/lingodb-relwithdebinfo/.buildstamp` ).
+
+1. Run the ct.py script with query and dataset: `python3 tools/ct/ct.py resources/sql/tpch/1.sql resources/data/tpch-1/`. If the build directory is not `build/lingodb-relwithdebinfo`, it can be supplied with the `BIN_DIR` environment variable
+2. Open the resulting `ct.json` file with the [CT viewer](https://ct.lingo-db.com) and explore it in detail
+
+## Guide: Debugging
+* If the compilation fails: Use [Snapshotting](#snapshotting) to identify the broken/problematic pass. Then run the pass isolated with [mlir-db-opt](../GettingStarted/CommandLineTools.md#performing-optimizations-and-lowerings) for detailed debugging (e.g., with gdb). 
+* If compilation succeeds but execution fails in/because generated code: First check if the error persists when switching to the [C++-Backend](#c-backend) if possible (i.e., all MLIR operations are supported)
+  * If yes: debug with this backend. 
+  * If not: you should use the [LLVM Debug Backend](#llvm-debug-backend)
+
+## Components for Debugging and Profiling
+### Location Tracking in MLIR
+In MLIR, every operation is associated with a *Location*, that must be provided during operation creation.
+While it is possible to provide a *Unknown Location*, it should be avoided.
+When parsing a MLIR file, MLIR automatically annotates the parsed operations with the corresponding file locations.
+When new operations are created during a pass they are usually annotated with the location of the current operation that is transformed or lowered.
+**All passes in LingoDB ensure that correct locations are set afterwards.**
+
+### Snapshotting
+MLIR already comes with a `LocationSnapshotPass` that takes an operation (e.g. a MLIR Module) and writes it to disk, including the annotated locations.
+Then, this file is now read back in, now annotating the locations *according to the location inside this newly written file*.
+
+If enabled (cf [Settings](Settings.md) ), LingoDB performs multiple location snapshots on after every or selected (important) MLIR passes.
+
+Using this snapshot files, we can track the origin of any operation, by recursively following the following steps
+1. get the origin location of the current operation by looking in the appropriate snapshot file
+2. find the origin operation by going to this location
+
+### Special Compiler Backends
+In addition to location tracking and snapshotting, LingoDB implements two special compiler backends for debugging.
+
+#### LLVM-Debug Backend
+Instead of using the standard LLVM backend, another LLVM-based backend can be used that adds debug information and performs no optimizations.
+This backend is selected by setting the environment variable `LINGODB_EXECUTION_MODE=DEBUGGING`.
+During the execution, standard debuggers like `gdb` will then point to the corresponding operation in the last snapshot that was performed
+This enables basic tracking of problematic operations, but advanced debugging will remain difficult.
+
+#### C++-Backend
+For more advanced debugging, a *C++-Backend* can be used by setting `LINGODB_EXECUTION_MODE=C`.
+This backend directly translates a fixed set of low-level generic MLIR operations to C++ statements and functions that are written to a file called `mlir-c-module.cpp`.
+Next, LingoDB automatically invokes `clang++` (must be installed!) with `-O0` and `-g` to compile this C++ file into a shared library with debug informations.
+This shared library is then loaded with `dlopen` and the main function is called.
+Thus, the generated code can be debugged as any usual C++ program.
+To help with tracking an error to higher-level MLIR operations, each C++ statement is preceeded with a comment containing the original operation and it's location.
+
+
+### Lightweight Tracing
+When compiled as `RelWithDebInfo`, LingoDB will produce a trace file with events (type, start timestamp, duration, thread) as trace.json.
+This trace file can then be opened with the [CT Viewer](https://ct.lingo-db.com)

versioned_docs/version-0.0.4/ForDevelopers/Dependencies.md

@@ -0,0 +1,147 @@
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+LingoDB relies on three main external dependencies:
+* [LLVM/MLIR 20](https://github.com/llvm/llvm-project)
+* [Apache Arrow 20](https://arrow.apache.org/release/19.0.0.html)
+* [Boost Context 1.83](https://www.boost.org/doc/libs/1_83_0/libs/context/doc/html/index.html)
+
+**Additional tools and libraries required:**
+* C++ compiler supporting C++ 20
+* CMake 3.13.4 or newer
+* Ninja
+* lit (optional, for testing), can be e.g., installed via `pip install lit`
+
+We also provide a [Dockerfile](https://github.com/lingo-db/lingo-db/pkgs/container/lingodb-dev) that contains all dependencies and tools required to build LingoDB.
+
+When building dependencies from source, make sure that either the cmake config files are installed in a system-wide locations, or for example, the `CMAKE_PREFIX_PATH` is set accordingly.
+
+## LLVM/MLIR
+
+<Tabs groupId="os-tabs">
+<TabItem value="linux" label="Ubuntu/Linux">
+Follow the instructions on [https://apt.llvm.org/](https://apt.llvm.org/) to install the repository on your system.
+Then install the following packages: `clang-20 llvm-20 libclang-20-dev llvm-20-dev libmlir-20-dev mlir-20-tools clang-tidy-20`
+
+### Binaries
+For other recent Linux distributions, you can also rely on the pre-built binaries provided by the LLVM project on the Github release pages.
+
+### Building from Source
+
+```shell
+wget https://github.com/llvm/llvm-project/releases/download/llvmorg-20.1.0-rc1/llvm-project-20.1.0-rc1.src.tar.xz
+tar -xf llvm-project-20.1.0-rc1.src.tar.xz
+mkdir llvm-project-20.1.0-rc1.src/build
+cd llvm-project-20.1.0-rc1.src
+export INSTALL_PREFIX=[install_prefix]
+env VIRTUAL_ENV=/venv cmake -B build  -DLLVM_ENABLE_PROJECTS="llvm;mlir;clang;clang-tools-extra" -DLLVM_TARGETS_TO_BUILD="X86" -DLLVM_BUILD_EXAMPLES=OFF -DCMAKE_BUILD_TYPE=Release -G Ninja -DLLVM_ENABLE_ASSERTIONS=OFF  -DLLVM_BUILD_TESTS=OFF -DLLVM_BUILD_LLVM_DYLIB=ON -DLLVM_LINK_LLVM_DYLIB=OFF -DLLVM_ENABLE_DUMP=ON -DLLVM_ENABLE_FFI=ON -DCMAKE_CXX_FLAGS="-fno-omit-frame-pointer -mno-omit-leaf-frame-pointer" -DLLVM_PARALLEL_LINK_JOBS=1 -DLLVM_PARALLEL_TABLEGEN_JOBS=10 -DBUILD_SHARED_LIBS=OFF -DLLVM_INSTALL_UTILS=ON  -DLLVM_ENABLE_ZLIB=OFF -DCMAKE_INSTALL_PREFIX=$INSTALL_PREFIX llvm/
+cmake --build build --target install -j$(nproc)
+```
+
+</TabItem>
+<TabItem value="macos" label="MacOS">
+Install LLVM/MLIR using Homebrew and make it available system-wide:
+
+```shell
+brew install llvm@20
+brew link --force llvm@20
+```
+
+### Binaries
+⚠️ **Caution**: the pre-built binaries provided by the LLVM project on the Github release pages **DO NOT** serve as a replacement, since they lack the required MLIR support.
+
+### Building from Source
+
+1. Install XCode (through the App Store).
+2. Install the build requisites: `brew install cmake ninja z3`
+2. Make sure to replace [install_prefix] with your preferred install path of LLVM.
+```shell
+wget https://github.com/llvm/llvm-project/releases/download/llvmorg-20.1.4/llvm-project-20.1.4.src.tar.xz
+tar -xf llvm-project-20.1.4.src.tar.xz
+mkdir llvm-project-20.1.4.src/build
+cd llvm-project-20.1.4.src
+export INSTALL_PREFIX=[install_prefix]
+export SDKROOT=$(xcrun --sdk macosx --show-sdk-path)
+env VIRTUAL_ENV=/venv cmake -B build  -DLLVM_ENABLE_PROJECTS="clang;clang-tools-extra;mlir;polly;lldb" -DLLVM_ENABLE_RUNTIMES="compiler-rt;libcxx;libcxxabi;libunwind;pstl;openmp" -DLLVM_TARGETS_TO_BUILD="AArch64" -DLLVM_BUILD_EXAMPLES=OFF -DCMAKE_BUILD_TYPE=Release -G Ninja -DLLVM_ENABLE_ASSERTIONS=ON  -DLLVM_BUILD_TESTS=OFF -DLLVM_BUILD_LLVM_DYLIB=ON -DLLVM_LINK_LLVM_DYLIB=ON -DLLVM_ENABLE_DUMP=ON -DLLVM_ENABLE_FFI=ON -DCMAKE_CXX_FLAGS="-fno-omit-frame-pointer -mno-omit-leaf-frame-pointer" -DLLVM_PARALLEL_LINK_JOBS=1 -DLLVM_PARALLEL_TABLEGEN_JOBS=10 -DBUILD_SHARED_LIBS=OFF -DLLVM_INSTALL_UTILS=ON  -DLLVM_ENABLE_ZLIB=OFF -DLLVM_POLLY_LINK_INTO_TOOLS=ON -DLLVM_BUILD_EXTERNAL_COMPILER_RT=ON -DLLVM_ENABLE_EH=OFF -DLLVM_ENABLE_RTTI=ON -DLLVM_INCLUDE_DOCS=OFF -DLLVM_INCLUDE_TESTS=OFF -DLLVM_OPTIMIZED_TABLEGEN=ON -DLLVM_USE_RELATIVE_PATHS_IN_FILES=ON -DLLVM_SOURCE_PREFIX=. -DLLDB_USE_SYSTEM_DEBUGSERVER=ON -DLIBOMP_INSTALL_ALIASES=OFF -DLIBCXX_INSTALL_MODULES=ON -DLLVM_CREATE_XCODE_TOOLCHAIN=OFF -DCLANG_FORCE_MATCHING_LIBCLANG_SOVERSION=OFF -DLLVM_BUILD_LLVM_C_DYLIB=ON -DLLVM_ENABLE_LIBCXX=ON -DLIBCXX_PSTL_BACKEND=libdispatch -DCMAKE_INSTALL_LIBDIR=lib -DCMAKE_FIND_FRAMEWORK=LAST -DCMAKE_VERBOSE_MAKEFILE=ON -DCMAKE_PROJECT_TOP_LEVEL_INCLUDES=/opt/homebrew/Library/Homebrew/cmake/trap_fetchcontent_provider.cmake -Wno-dev -DCMAKE_OSX_SYSROOT=$(xcrun --sdk macosx --show-sdk-path) -DLLVM_ENABLE_Z3_SOLVER=ON -DFFI_INCLUDE_DIR=$(xcrun --sdk macosx --show-sdk-path)/usr/include/ffi -DFFI_LIBRARY_DIR=$(xcrun --sdk macosx --show-sdk-path)/usr/lib -DLIBCXX_INSTALL_LIBRARY_DIR=$INSTALL_PREFIX/lib/c++ -DLIBUNWIND_INSTALL_LIBRARY_DIR=$INSTALL_PREFIX/lib/unwind -DLIBCXXABI_INSTALL_LIBRARY_DIR=$INSTALL_PREFIX/c++ -DRUNTIMES_CMAKE_ARGS="-DCMAKE_INSTALL_RPATH=@loader_path|@loader_path/../unwind" -DBUILTINS_CMAKE_ARGS="-DCOMPILER_RT_ENABLE_IOS=OFF;-DCOMPILER_RT_ENABLE_WATCHOS=OFF;-DCOMPILER_RT_ENABLE_TVOS=OFF" -DCMAKE_PREFIX_PATH=/opt/homebrew -DCMAKE_INSTALL_PREFIX=$INSTALL_PREFIX llvm/
+cmake --build build --target install -j$(sysctl -n hw.logicalcpu)
+```
+</TabItem>
+</Tabs>
+
+## Apache Arrow
+
+<Tabs groupId="os-tabs">
+<TabItem value="linux" label="Ubuntu/Linux">
+
+```shell
+wget https://apache.jfrog.io/artifactory/arrow/$(lsb_release --id --short | tr 'A-Z' 'a-z')/apache-arrow-apt-source-latest-$(lsb_release --codename --short).deb
+apt install -y -V ./apache-arrow-apt-source-latest-$(lsb_release --codename  --short).deb
+apt-get update
+apt-get install libarrow-dev=20.*
+```
+
+### Binaries
+For other recent Linux distributions, you can also rely on the pre-built binaries provided by the Apache Arrow project.
+
+### Building from Source
+
+```shell
+wget https://github.com/apache/arrow/releases/download/apache-arrow-20.0.0/apache-arrow-20.0.0.tar.gz
+tar -xf apache-arrow-20.0.0.tar.gz
+cd apache-arrow-20.0.0/cpp
+cmake -B build -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=[output-dir] -DARROW_DEPENDENCY_SOURCE=BUNDLED -DARROW_BUILD_STATIC=ON -DARROW_CSV=ON -DARROW_JSON=ON -DARROW_COMPUTE=ON apache-arrow-20.0.0/cpp
+cmake --build build --target install -j$(nproc)
+```
+
+</TabItem>
+<TabItem value="macos" label="MacOS">
+
+Install Apache Arrow using Homebrew:
+
+```shell
+brew tap lingo-db/homebrew https://github.com/lingo-db/homebrew.git
+brew install lingo-db/homebrew/apache-arrow@20
+```
+
+### Building from Source
+
+```shell
+wget https://github.com/apache/arrow/releases/download/apache-arrow-20.0.0/apache-arrow-20.0.0.tar.gz
+tar -xf apache-arrow-20.0.0.tar.gz
+cd apache-arrow-20.0.0/cpp
+cmake -B build -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=[output-dir] -DARROW_DEPENDENCY_SOURCE=BUNDLED -DARROW_BUILD_STATIC=ON -DARROW_CSV=ON -DARROW_JSON=ON -DARROW_COMPUTE=ON -DCMAKE_PREFIX_PATH=/opt/homebrew/ -DCMAKE_CXX_COMPILER=/opt/homebrew/bin/clang++ -DCMAKE_C_COMPILER=/opt/homebrew/bin/clang
+cmake --build build --target install -j$(sysctl -n hw.logicalcpu)
+```
+
+</TabItem>
+</Tabs>
+
+## Boost Context
+
+<Tabs groupId="os-tabs">
+<TabItem value="linux" label="Ubuntu/Linux">
+
+```shell
+apt-get install libboost-context1.83-dev
+```
+
+### Build from Source
+```shell
+wget https://archives.boost.io/release/1.83.0/source/boost_1_83_0.tar.gz
+tar -xf boost_1_83_0.tar.gz
+cd boost_1_83_0
+ ./bootstrap.sh --prefix=/usr # or any other directory in the PATH/LD_LIBRARY_PATH
+ ./b2 install --with-context
+```
+
+</TabItem>
+<TabItem value="macos" label="MacOS">
+
+Install Boost Context using Homebrew:
+
+```shell
+brew install boost
+```
+
+</TabItem>
+</Tabs>