docs: move and adapt the other chapters of Stan's docs suitable for public docs

alehander92 · alehander92 · commit 5678ac6f3885 · 2025-03-11T21:32:05.000+02:00
also added `just serve-docs`
diff --git a/docs/experimental-documentation/Introduction/Backends/CoreBackend.md b/docs/experimental-documentation/Introduction/Backends/CoreBackend.md
@@ -1,22 +1,32 @@
-The Core backend is used for systems programming languages, like:
+## RR backend
+
+The RR backend is used for systems programming languages, like:
 
 1. [C & C++](https://dev-docs.codetracer.com/Introduction/Backends/CoreBackend/CAndCpp)
 1. [Goland](https://dev-docs.codetracer.com/Introduction/Backends/CoreBackend/Golang)
 1. [Nim](https://dev-docs.codetracer.com/Introduction/Backends/CoreBackend/Nim)
 1. [Rust](https://dev-docs.codetracer.com/Introduction/Bacends/CoreBackend/Rust)
 
-Under the hood, it uses a patched version of the [rr debugger](https://rr-project.org/), as well as the standard
+Under the hood, it uses a patched version of the [RR debugger](https://rr-project.org/), as well as the standard
 GDB debugger.
 
+We are still not distributing it, but we might need to rethink the GDB usage before that.
+
 The Core backend is based on a dispatcher that maintains a pool of rr replay processes. These processes allow you to jump back and forward in time, while also
 preloading information, such as the information, needed for omniscience and calltrace, as well as managing other features, such as tracepoints. This logic is
-implemented using python code using the gdb API.
+currently implemented using python code using the gdb API. 
+(If required, this can be changed to use the lldb API or something more custom, like a dwarf lib-based solution. We are currently using the rust `gimli` DWARF lib for a tool, that helps with recording/metadata in the RR backend).
+
+These are the main differences, when compared to the DB Backend: based on the way RR works:
 
-These are the main differences, when compared to the DB Backend:
+1. Only non-deterministic events are recorded.
+1. The program's different states are navigated through re-execution.
+1. More practical for recording real-world software(RR was originally created by Mozilla to debug Firefox)
+1. Some features of CodeTracer are more challenging to implement, compared to the [DB Backend](./backends/db-backend.md)
 
-1. Only non-deterministic events are recorded
-1. The program's different sates are navigated through re-execution
-1. Practical for recording real-world software(rr was originally created by Mozilla to debug Firefox)
-1. Some features of CodeTracer are more challenging to implement, compared to the [DB Backend](https://dev-docs.codetracer.com/Introduciton/Backends/DBBackend)
+> ![NOTE]
+> This backend is in active development, and is yet to be released to the wider public. 
+>
+> It is currently proprietary and it might remain closed source. 
+> Open sourcing it is also possible, if we find a suitable business model.
 
-This backend is in active development, and is yet to be released to the wider public. 
diff --git a/docs/experimental-documentation/Introduction/Backends/DBBackend.md b/docs/experimental-documentation/Introduction/Backends/DBBackend.md
@@ -1,9 +1,10 @@
-The DB backend is used for interpreted languages, like:
+The DB backend can be used for interpreted languages, like:
 
-1. [Noir](https://dev-docs.codetracer.com/Introduction/Backends/DBBackend/Noir)
-1. [Ruby](https://dev-docs.codetracer.com/Introduction/Backends/DBBackend/Ruby)
-1. [Lua](https://dev-docs.codetracer.com/Introduction/Backends/DBBackend/Lua)
-1. [SmallLang](https://dev-docs.codetracer.com/Introduction/Backends/DBBackend/SmallLang)
+1. [Noir](./backends/db_backend/noir.md)
+1. [Ruby](./backends/db_backend/ruby.md)
+1. [Python(prototype not finished yet)](./backends/db_backend/python.md)
+1. [Lua(not done: just a plan)](./backends/db_backend/lua.md)
+1. [small(a toy interpreter for dogfooding)](./backends/db_backend/small.md)
 
 As the name suggests, this backend is implemented as a large database. Due to its properties, this backend provides
 a more complete feature set - one of the reasons it was open-sourced first.
diff --git a/docs/experimental-documentation/Introduction/Troubleshooting.md b/docs/experimental-documentation/Introduction/Troubleshooting.md
@@ -4,15 +4,18 @@ If you find any bug, please report it as an issue on GitHub.
 In the meantime, you can use this page to fix some issues that are somewhat common.
 
 ## Fixing outdated configuration/layout files
+
 You can find more information [here](https://dev-docs.codetracer.com/Introduction/Configuration#Layout).
 
 ## Resetting the local trace database
+
 There are 2 commands that can be used to completely wipe all traces from your user's data:
 
 1. `just reset-db` - Resets the local user's trace database
 1. `just clear-local-traces` - Clears the local user's traces
 
 ## Broken local build
+
 Sometimes your local build might break. In most cases, a simple `direnv reload` and `just build` should be able to fix it.
 Otherwise, you can ask for help on our [issues tracker](github.com/metacraft-labs/codetracer), or in our [chat server](https://discord.gg/aH5WTMnKHT)
 
diff --git a/docs/experimental-documentation/Misc/BuildingDocs.md b/docs/experimental-documentation/Misc/BuildingDocs.md
@@ -1,7 +1,8 @@
 # Building the documentation
+
 The documenation for codetracer is written in [GitHub Flavoured Markdown](https://docs.github.com/en/get-started/writing-on-github/getting-started-with-writing-and-formatting-on-github/basic-writing-and-formatting-syntax) 
 that's converted to HTML using [pandoc](https://pandoc.org). This makes the documentation both easy to write, maintain, deploy and build.
 
 To build the documentation run `just build-docs`. If you want to build the documentation for local development, run `just build-docs localhost` and a web server will be started on port `5000`.
 
-The built is stored under `docs/experimental-documentation/build`, while the markdown files are under `docs/experimental-documentation` and its child directories.
+The built doc files are stored under `docs/experimental-documentation/build`, while the markdown files are under `docs/experimental-documentation` and its child directories.
diff --git a/docs/experimental-documentation/mdbook-ct/src/SUMMARY.md b/docs/experimental-documentation/mdbook-ct/src/SUMMARY.md
@@ -1,12 +1,35 @@
-# Summary
+## Summary
 
-- [Introduction](./chapter_1.md)
+- [Introduction](./introduction.md)
 
-# Usage guide
+## Usage guide
 
 - [Intro to usage guide](./usage_guide/intro_to_usage.md)
 - [CLI](./usage_guide/CLI.md)
 - [Basic GUI](./usage_guide/basic_gui.md)
 - [Tracepoints](./usage_guide/tracepoints.md)
 - [CodeTracer Shell](./usage_guide/codetracer_shell.md)
 
+## Backends
+
+- [DB backend](./backends/db_backend.md)
+  - [Noir](./backends/db-backend/noir.md)
+  - [Ruby](./backends/db-backend/ruby.md)
+  - [Python](./backends/db-backend/py.md)
+  - [Lua](./backends/db-backend/lua.md)
+  - [small](./backends/db-backend/small.md)
+  
+- [RR backend](./backends/rr_backend.md)
+  - [C & C++](./backends/rr-backend/c_and_cpp.md)
+  - [Rust](./backends/rr-backend/rust.md)
+  - [Nim](./backends/rr-backend/nim.md)
+  - [Go](./backends/rr-backend/go.md)
+
+## Building & packaging
+
+- [Build systems](./building_and_packaging/build_systems.md)
+
+### Misc
+
+- [Troubleshooting](./misc/troubleshooting.md)
+- [Building the documentation](./misc/building_docs.md)
diff --git a/docs/experimental-documentation/mdbook-ct/src/backends/db-backend/lua.md b/docs/experimental-documentation/mdbook-ct/src/backends/db-backend/lua.md
@@ -0,0 +1,5 @@
+## Lua
+
+We still don't have Lua support, however we've had a very minimal proof of concept experiment with supporting Lua (the interpreter, not LuaJIT) with our RR backend years ago.
+
+We can change that patch to produce traces suitable for the DB backend, or at least publish some kind of basis for that(the RR approach remains possible, but with different tradeoffs)
diff --git a/docs/experimental-documentation/mdbook-ct/src/backends/db-backend/noir.md b/docs/experimental-documentation/mdbook-ct/src/backends/db-backend/noir.md
@@ -0,0 +1,7 @@
+## Noir
+
+The initial release of CodeTracer has MVP support for the Noir programming language. It has been developed in collaboration with the Blocksense team and currently requires the use of the [Blocksense Noir Compiler](https://github.com/blocksense-network/noir), which is included in the CodeTracer distribution.
+
+We support many of Noir's features, but not all: e.g. we don't support mutable references currently, we don't serialize struct values and some other cases.
+
+We are planning on adding support for the missing features.
diff --git a/docs/experimental-documentation/mdbook-ct/src/backends/db-backend/py.md b/docs/experimental-documentation/mdbook-ct/src/backends/db-backend/py.md
@@ -0,0 +1,8 @@
+## python
+
+We currently have started adding support for recording python programs for the DB backend, but the prototype isn't finished yet. 
+
+The recorder is currently hosted in the [codetracer-python-recorder](https://github.com/metacraft-labs/codetracer-python-recorder) repo.
+
+You can read its README for more details. We are welcoming contributors!
+
diff --git a/docs/experimental-documentation/mdbook-ct/src/backends/db-backend/python.md b/docs/experimental-documentation/mdbook-ct/src/backends/db-backend/python.md
@@ -0,0 +1 @@
+# Python
diff --git a/docs/experimental-documentation/mdbook-ct/src/backends/db-backend/ruby.md b/docs/experimental-documentation/mdbook-ct/src/backends/db-backend/ruby.md
@@ -0,0 +1,9 @@
+## Ruby
+
+We currently have partial support for the Ruby programming language in the DB backend. 
+We support many cases, you can use it like `ct record <path to rb file> [<args>]` and `ct replay <name of rb file>` (or directly `ct run <path to rb file> [<args>]`)
+
+The recorder for Ruby is currently hosted in the [codetracer-ruby-recorder](https://github.com/metacraft-labs/codetracer-ruby-recorder) repo.
+
+You can read its README for more details. We are welcoming contributors!
+
diff --git a/docs/experimental-documentation/mdbook-ct/src/backends/db-backend/small.md b/docs/experimental-documentation/mdbook-ct/src/backends/db-backend/small.md
@@ -0,0 +1,8 @@
+## small lang
+
+small lang is, as the name says, a small, lisp-like language, that's used for general dogfooding of new tracing features and testing of the DB backend.
+It's purely experimental.
+
+You can find its source code under `src/small-lang`. There, you can also find multiple files with the `.small`
+extension, which can be used as examples of the language.
+
diff --git a/docs/experimental-documentation/mdbook-ct/src/backends/db_backend.md b/docs/experimental-documentation/mdbook-ct/src/backends/db_backend.md
@@ -0,0 +1,30 @@
+## DB backend
+
+The DB backend is currently used for interpreted languages, like scripting and blockchain/zero knowledge languages:
+
+1. [Noir](./db-backend/noir.md)
+1. [Ruby](./db-backend/ruby.md)
+1. [Python(prototype not finished yet)](./db-backend/py.md)
+1. [Lua(not done: just a plan)](./db-backend/lua.md)
+1. [small(a toy interpreter for dogfooding)](./db-backend/small.md)
+
+If you're interested in db-backend support for those languages or for now ones, you can discuss with us on our Github issue tracker,
+in the discussions or in our [chat server](https://discord.gg/aH5WTMnKHT). We welcome contributors!
+
+However it might be used for emulators(and this way for native languages) in the future.
+
+Also, languages can have both native(AOT) and interpreted implementations: 
+so we can more easily support e.g. interpreters/VM-s of certain languages with this backend currently.
+
+As the name suggests, this backend is implemented as a database. Due to its properties, this backend provides
+a more complete feature set - one of the reasons it was ready first.
+
+For each language, we define a tracer, which instruments or hooks into the interpreter, VM or program and 
+records all the data, that we need in a trace folder. 
+
+We postprocess and index that trace into a database-like structure. Currently this happens in the beginning of the replay.
+For now the data in the database is represented by a runtime-only Rust structure, however
+we plan on storing it as a file, which can be memory-mapped and on separating the postprocessing/indexing step as 
+an optionally independent action.
+
+
diff --git a/docs/experimental-documentation/mdbook-ct/src/backends/db_backend/noir.md b/docs/experimental-documentation/mdbook-ct/src/backends/db_backend/noir.md
@@ -0,0 +1 @@
+# Noir
diff --git a/docs/experimental-documentation/mdbook-ct/src/backends/rr-backend/c_and_cpp.md b/docs/experimental-documentation/mdbook-ct/src/backends/rr-backend/c_and_cpp.md
@@ -0,0 +1,6 @@
+## C & C++
+
+We have partial support for C and C++ in the RR backend. C generally has good support, though it is less advanced than
+the Rust implementation.
+
+Support for C++ is still experimental, due to missing support for many C++ constructs.
diff --git a/docs/experimental-documentation/mdbook-ct/src/backends/rr-backend/go.md b/docs/experimental-documentation/mdbook-ct/src/backends/rr-backend/go.md
@@ -0,0 +1,4 @@
+# Go
+
+We have developed а basic integration that supports small parts of the Golang language in the Core backend. 
+Development on a more complete revision is scheduled to start soon!
diff --git a/docs/experimental-documentation/mdbook-ct/src/backends/rr-backend/nim.md b/docs/experimental-documentation/mdbook-ct/src/backends/rr-backend/nim.md
@@ -0,0 +1,18 @@
+# Nim
+
+We have partial support for the Nim backend, with it being somewhat behind in features compared to C, but ahead,
+compared to C++.
+
+The Nim backend only supports Nim 1 as of today, but there are plans to add support for Nim 2.
+
+
+Some additional future goals that have been announced in the Nim forum are:
+
+* First-class support for macro-generated code - macro generated code should be indistinguishable from regular code. You can expand it layer by layer, set breakpoints and tracepoints anywhere within it, and step through it in all the familiar ways. (or at least part of those)
+* Precise debug info - you should be able to set breakpoints and tracepoints at the level of individual sub-expressions and you can switch between debugging Nim, C or assembly code at any time.
+* Compile-time debugging - with few clicks in your text editor, you should be able to trace the execution of the Nim VM itself.
+
+> [!NOTE]
+> For now we're focused on the beta and initial releases of the RR backend though, and on initial releases for some of the language supports.
+> This means the initial support for Nim would be focused on the same features as the other languages: stepping, tracepoints, event log, calltrace, omniscience, state, history. We do hope we'll be able to continue with some of the more advanced goals afterwards.
+
diff --git a/docs/experimental-documentation/mdbook-ct/src/backends/rr-backend/rust.md b/docs/experimental-documentation/mdbook-ct/src/backends/rr-backend/rust.md
@@ -0,0 +1,3 @@
+# Rust
+
+Our backend for Rust is the most developed out of all languages in the RR backend, with it being closest to MVP status(but still non trivial amount of work left).
diff --git a/docs/experimental-documentation/mdbook-ct/src/backends/rr_backend.md b/docs/experimental-documentation/mdbook-ct/src/backends/rr_backend.md
@@ -0,0 +1,32 @@
+## RR backend
+
+The RR backend is used for systems programming languages, like:
+
+1. [C & C++](./rr-backend/c_and_cpp.md)
+1. [Rust](./rr-backend/rust.md)
+1. [Nim](./rr-backend/nim.md)
+1. [Go](./rr-backend/go.md)
+
+Under the hood, it uses a patched version of the [RR debugger](https://rr-project.org/), as well as the standard
+GDB debugger.
+
+We are still not distributing it, but we might need to rethink the GDB usage before that.
+
+The Core backend is based on a dispatcher that maintains a pool of rr replay processes. These processes allow you to jump back and forward in time, while also
+preloading information, such as the information, needed for omniscience and calltrace, as well as managing other features, such as tracepoints. This logic is
+currently implemented using python code using the gdb API. 
+(If required, this can be changed to use the lldb API or something more custom, like a dwarf lib-based solution. We are currently using the rust `gimli` DWARF lib for a tool, that helps with recording/metadata in the RR backend).
+
+These are the main differences, when compared to the DB Backend: based on the way RR works:
+
+1. Only non-deterministic events are recorded.
+1. The program's different states are navigated through re-execution.
+1. More practical for recording real-world software(RR was originally created by Mozilla to debug Firefox)
+1. Some features of CodeTracer are more challenging to implement, compared to the [DB Backend](./backends/db-backend.md)
+
+> ![NOTE]
+> This backend is in active development, and is yet to be released to the wider public. 
+>
+> It is currently proprietary and it might remain closed source. 
+> Open sourcing it is also possible, if we find a suitable business model.
+
diff --git a/docs/experimental-documentation/mdbook-ct/src/building_and_packaging/build_systems.md b/docs/experimental-documentation/mdbook-ct/src/building_and_packaging/build_systems.md
@@ -0,0 +1,56 @@
+## Build systems
+
+Codetracer uses nix, just, tup and direnv as parts of its build system.
+
+## Breakdown of the different components
+### Nix
+The Nix package manager/build system deals with managing all dependencies, required by CodeTracer, as well as packaging
+it for NixOS. It can package for other operating systems through an AppImage generator utility, however currently we are building our
+appimages with custom shell scripts in `appimage-scripts`.
+
+### Just
+
+The `just` tool is used to trigger build commands and various other actions. It's semi-analogous to a Makefile. The following commands are
+the most widely-used:
+
+1. `just build` - Builds the project with Tup and starts the automatic build process, which is used for active development
+1. `just build-once` - Just builds the project using Tup
+1. `just build-nix` - Builds the project and packages it for Nix
+1. `just build-docs` - Builds the documentation. More info can be found [here](https://dev-docs.codetracer.com/Misc/BuildDocs)
+1. `just cachix-push-nix-package` - Pushes nix package artefacts to cachix
+1. `just cachix-push-devshell` - Pushes the current dev shell to cachix
+1. `just reset-db` - Resets the local user's trace database
+1. `just clear-local-traces` - Clears the local user's traces
+1. `just reset-layout` - Resets the GUI window arrangements layout if your user's layout is incompatible with the latest version of CodeTracer. Further [documentation](https://dev-docs.codetracer.com/Introduction/Configuration)
+1. `just reset-config` - Resets the user's configuration if it's incompatible with the latest version of CodeTracer. Further [documentation](https://dev-docs.codetracer.com/Introduction/Configuration)
+
+### Tup
+The Tup build system is used for local builds of CodeTracer and deals with calling the actual low-level build instructions.
+
+### Direnv
+The `direnv` utility sets up your local environment for using CodeTracer.
+
+## Packaging
+
+### More detailed breakdown of the Nix package
+
+Coming soon!
+
+### Packaging for non-NixOS distributions
+
+> [!TIP]
+> If you're a user that wants a package for your distribution contact us. We're currently in the process of creating
+> packages for popular distributions, such as Debian/Ubuntu, Fedora/RHEL, Arch Linux, Gentoo, Void, etc. 
+
+To package for another Linux distribution with a ports-based package manager, you can utilise our AppImage(currently unreleased), 
+which you can install to `/usr/bin`. Along with it, you should also install our icon and desktop file 
+from `resources/` to the needed directories, such as `/usr/share/pixmaps` and `/usr/share/applications`.
+
+### Packaging for Windows(DB-backend only)
+
+Coming soon!
+
+### Packaging for macOS(DB-backend only)
+
+Coming soon!
+
diff --git a/docs/experimental-documentation/mdbook-ct/src/introduction.md b/docs/experimental-documentation/mdbook-ct/src/introduction.md
@@ -3,13 +3,9 @@
 Welcome to the codetracer-desktop wiki. Here you can find information on almost every topic
 regarding codetracer development and usage.
 
-## What is codetracer
-Codetracer is a debugging environment, based on the concept of record and replay, developed as a powerful tool to easily
-debug complex applications.
-
 ## Installation
 > [!CAUTION]
-> Codetracer can only be installed on Linux and macOS.
+> Codetracer can only be installed on Linux and macOS currently.
 
 ### Prerequisites
 On systems that are not NixOS, you need to install `direnv` and `nix`.
@@ -50,39 +46,14 @@ TODO:
 > 1. Run `direnv allow`
 
 ### Building and running the tests
-To start running tests do the following:
 
-1. Run `tester build` - Builds tester
-1. Run `tester parallel` - Runs the tests
+Coming soon!
 
 ### Enabling `cachix`
-> [!NOTE]
-> This step is optional
-
-<!-- TODO: Scrap or completely rewrite the cachix instructions for end users. Alternatively, make this an internal developer guide -->
-
-Cachix is a cache for nix that allows you to save time on compiling codetracer and related projects. To enable `cachix` do the following:
-
-1. Log into [cachix](https://www.cachix.org/) with your personal GitHub account
-1. Create an authentication token
-1. Run `cachix authtoken --stdin`
-1. Paste the token and click enter
-1. Press `CTRL + D` to save the token
-1. Run `cachix use metacraft-labs-codetracer`
-1. Run `direnv allow`, `nix develop`, or `just build-nix` to auto-download cached binaries if available
-
-### Explicit `cachix` setup
-> [!NOTE]
-> You have to be an admin for the private cache. Ask an administrator to get added.
-
-Our current `cachix` setup pushes binaries from CI, but if you want to manually push to `cachix` as well, or want to know how pushing works, you can do the following:
-
-1. Go to [this page](https://app.cachix.org/organization/metacraft-labs/cache/metacraft-labs-codetracer/settings/authtokens)
-1. Create an auth token with `Read+Write` permissions
-1. Locally register it as described in the above heading
 
-To push the dev shell to `cachix` use either one of the following commands:
+<!-- TODO(alexander): I removed the detailed cachix guide, as it's sensitive, and we don't have a public codetracer cache yet  -->
+<!-- either include it in an internal docs in the rr-backend, or re-include here when this is discussed again -->
 
-1. Automatically: `just cachix-push-devshell`
-1. Manually: `cachix push metacraft-labs-codetracer "$(nix build --print-out-paths .#devshells.x86_64-linux.default)"`
+Cachix is a cache for nix that allows you to save time on compiling codetracer and related projects. We'll discuss using a public codetracer cache
+for the open sourced parts, however this is available only internally for now.
 
diff --git a/docs/experimental-documentation/mdbook-ct/src/misc/building_docs.md b/docs/experimental-documentation/mdbook-ct/src/misc/building_docs.md
diff --git a/docs/experimental-documentation/mdbook-ct/src/misc/troubleshooting.md b/docs/experimental-documentation/mdbook-ct/src/misc/troubleshooting.md
diff --git a/examples/array.rb b/examples/array.rb
diff --git a/justfile b/justfile

-Original file line number
+Diff line change
@@ @@ -0,0 +1,5 @@ @@
 +## Lua
++
 +We still don't have Lua support, however we've had a very minimal proof of concept experiment with supporting Lua (the interpreter, not LuaJIT) with our RR backend years ago.
++
 +We can change that patch to produce traces suitable for the DB backend, or at least publish some kind of basis for that(the RR approach remains possible, but with different tradeoffs)