docs: move Stan's docs book to a mdbook

.gitignore

@@ -11,7 +11,7 @@ build-*/
 .direnv/
 result
 
-docs/experimental-documentation/build/*
+docs/book/book
 src/index.js
 ui.js
 debugger.js

CONTRIBUTING.md

@@ -0,0 +1,35 @@
+<!-- copied as both top-level codetracer repo CONTRIBUTING.md and a contributing page in the docs book  -->
+
+## Contributing
+
+We welcome contributors!
+
+If you want to fix something smaller, feel free to open an issue or a a PR.
+
+For bigger changes it's advised to first open an issue/discussion in the relevant Github repo or to discuss it our team in our [Discord chat server](https://discord.gg/aH5WTMnKHT).
+
+### Contributors guide
+
+Here are some recommendations, however if you want more info, our docs/contributor guide are hosted [on the CodeTracer site](https://contributors-guide.codetracer.com/) !
+
+The guide is written as a set of markdown documents and built using [mdbook](https://rust-lang.github.io/mdBook/) and [mdbook-alerts](https://crates.io/crates/mdbook-alerts) . 
+
+You can contribute to the documentation itself, by editing it in `docs/book` and making a pull request. You can iterate on it locally , by cloning the repo, activating it's nix devshell and running `just serve-docs [hostname port]`: it will serve it locally(by default on http://localhost:3000).
+
+### Style guide
+
+For Rust, we are using `cargo fmt` to autoformat our code, and `cargo clippy` with some custom allow/deny rules inside the code.
+We have a `cargo clippy` check in our CI, but one can also run it locally.
+
+For Nim, we still haven't written down a guide or list of rules and principles that we agree on, so this is something that we hope to do.
+We might also link to an existing document. 
+
+### Commits/Pull Requests
+
+We are using [the "Conventional Commits" strategy](https://www.conventionalcommits.org/). 
+
+We use or are ok with using more "types", not only those included by default in their official page: e.g. `cleanup:`, `tooling:`, `examples:` etc.
+
+We use `git rebase`, not merge and currently use the github pull requests as the main way to add code. Any pull request would need at least one review
+from someone from the CodeTracer team.
+

README.md

@@ -134,7 +134,7 @@ Run `ct --help` to see the full list of supported subcommands, but the most comm
    - `ct replay` - Opens a simple console-based dialog to choose what recording you want to replay.
    - `ct replay <program-name>` - Opens the last trace of an application.
    - `ct replay --id=<trace-id>` - Opens a trace by its trace id.
-   - `ct replay --trace-folde=<trace-folder>` - Opens a trace by its trace folder.
+   - `ct replay --trace-folder=<trace-folder>` - Opens a trace by its trace folder.
 1. `ct` - Launches the startup screen of the CodeTracer GUI.
 1. `ct help / ct --help` - Gives you a help message.
 1. `ct version` - Returns the current version of CodeTracer.

ci/deploy/docs.sh

@@ -2,10 +2,18 @@
 
 set -e
 
-pushd  docs/experimental-documentation
+pushd docs/book/
+
+# TODO: eventually: a more fine-grained nix shell/env?
+# nix-shell --command "mdbook build"
+
+# for now depending on global project devshell 
+mdbook build # build output is in the `book` directory
+
+# TODO: deploy book to codetracer.com ? ask Zahary?
 
 echo '###############################################################################'
-echo TODO
+echo "TODO upload book"
 echo '###############################################################################'
 
 popd

docs/book/book.toml

@@ -0,0 +1,8 @@
+[book]
+authors = ["CodeTracer team"]
+language = "en"
+multilingual = false
+src = "src"
+title = "codetracer docs"
+
+[preprocessor.alerts]

docs/experimental-documentation/shell.nix → docs/book/shell.nix

@@ -4,14 +4,12 @@ in
   pkgs.mkShell {
     buildInputs = with pkgs; [
       bash
-      terser
-      pandoc
-      parallel
+      mdbook
+      mdbook-alerts     
     ];
     nativeBuildInputs = with pkgs; [
       bash
-      terser
-      pandoc
-      parallel
+      mdbook
+      mdbook-alerts
     ];
   }

docs/book/src/CONTRIBUTING.md

@@ -0,0 +1,35 @@
+<!-- copied as both top-level codetracer repo CONTRIBUTING.md and a contributing page in the docs book  -->
+
+## Contributing
+
+We welcome contributors!
+
+If you want to fix something smaller, feel free to open an issue or a a PR.
+
+For bigger changes it's advised to first open an issue/discussion in the relevant Github repo or to discuss it our team in our [Discord chat server](https://discord.gg/aH5WTMnKHT).
+
+### Contributors guide
+
+Here are some recommendations, however if you want more info, our docs/contributor guide are hosted [on the CodeTracer site](https://contributors-guide.codetracer.com/) !
+
+The guide is written as a set of markdown documents and built using [mdbook](https://rust-lang.github.io/mdBook/) and [mdbook-alerts](https://crates.io/crates/mdbook-alerts) . 
+
+You can contribute to the documentation itself, by editing it in `docs/book` and making a pull request. You can iterate on it locally , by cloning the repo, activating it's nix devshell and running `just serve-docs [hostname port]`: it will serve it locally(by default on http://localhost:3000).
+
+### Style guide
+
+For Rust, we are using `cargo fmt` to autoformat our code, and `cargo clippy` with some custom allow/deny rules inside the code.
+We have a `cargo clippy` check in our CI, but one can also run it locally.
+
+For Nim, we still haven't written down a guide or list of rules and principles that we agree on, so this is something that we hope to do.
+We might also link to an existing document. 
+
+### Commits/Pull Requests
+
+We are using [the "Conventional Commits" strategy](https://www.conventionalcommits.org/). 
+
+We use or are ok with using more "types", not only those included by default in their official page: e.g. `cleanup:`, `tooling:`, `examples:` etc.
+
+We use `git rebase`, not merge and currently use the github pull requests as the main way to add code. Any pull request would need at least one review
+from someone from the CodeTracer team.
+

docs/book/src/SUMMARY.md

@@ -0,0 +1,37 @@
+## Summary
+
+- [Introduction](./introduction.md)
+- [Contributing](./CONTRIBUTING.md)
+
+## Usage guide
+
+- [Intro to usage guide](./usage_guide/intro_to_usage.md)
+- [CLI](./usage_guide/CLI.md)
+- [Basic GUI](./usage_guide/basic_gui.md)
+- [Tracepoints](./usage_guide/tracepoints.md)
+- [CodeTracer Shell](./usage_guide/codetracer_shell.md)
+
+## Backends
+
+- [DB backend](./backends/db_backend.md)
+  - [Noir](./backends/db-backend/noir.md)
+  - [Ruby](./backends/db-backend/ruby.md)
+  - [Python](./backends/db-backend/py.md)
+  - [Lua](./backends/db-backend/lua.md)
+  - [small](./backends/db-backend/small.md)
+
+- [RR backend](./backends/rr_backend.md)
+  - [C & C++](./backends/rr-backend/c_and_cpp.md)
+  - [Rust](./backends/rr-backend/rust.md)
+  - [Nim](./backends/rr-backend/nim.md)
+  - [Go](./backends/rr-backend/go.md)
+
+## Building & packaging
+
+- [Build systems](./building_and_packaging/build_systems.md)
+
+### Misc
+
+- [Troubleshooting](./misc/troubleshooting.md)
+- [Environment variables](./misc/environment_variables.md)
+- [Building the documentation](./misc/building_docs.md)

docs/book/src/backends/db-backend/lua.md

@@ -0,0 +1,5 @@
+## Lua
+
+We still don't have Lua support, however we've had a very minimal proof of concept experiment with supporting Lua (the interpreter, not LuaJIT) with our RR backend years ago.
+
+We can change that patch to produce traces suitable for the DB backend, or at least publish some kind of basis for that(the RR approach remains possible, but with different tradeoffs)

docs/book/src/backends/db-backend/noir.md

@@ -0,0 +1,7 @@
+## Noir
+
+The initial release of CodeTracer has MVP support for the Noir programming language. It has been developed in collaboration with the Blocksense team and currently requires the use of the [Blocksense Noir Compiler](https://github.com/blocksense-network/noir), which is included in the CodeTracer distribution.
+
+We support many of Noir's features, but not all: e.g. we don't support mutable references currently, we don't serialize struct values and some other cases.
+
+We are planning on adding support for the missing features.

docs/book/src/backends/db-backend/py.md

@@ -0,0 +1,8 @@
+## Python language
+
+We currently have started adding support for recording python programs for the DB backend, but the prototype isn't finished yet. 
+
+The recorder is currently hosted in the [codetracer-python-recorder](https://github.com/metacraft-labs/codetracer-python-recorder) repo.
+
+You can read its README for more details. We are welcoming contributors!
+

docs/book/src/backends/db-backend/ruby.md

@@ -0,0 +1,9 @@
+## Ruby
+
+We currently have partial support for the Ruby programming language in the DB backend. 
+We support many cases, you can use it like `ct record <path to rb file> [<args>]` and `ct replay <name of rb file>` (or directly `ct run <path to rb file> [<args>]`)
+
+The recorder for Ruby is currently hosted in the [codetracer-ruby-recorder](https://github.com/metacraft-labs/codetracer-ruby-recorder) repo.
+
+You can read its README for more details. We welcome contributors!
+

docs/experimental-documentation/Introduction/Backends/DBBackend/SmallLang.md → docs/book/src/backends/db-backend/small.md

@@ -1,5 +1,8 @@
-SmallLang is, as the name says, a small, lisp-like language, that's used for general testing of the DB backend.
+## small lang
+
+small lang is, as the name says, a small, lisp-like language, that's used for general dogfooding of new tracing features and testing of the DB backend.
 It's purely experimental.
 
 You can find its source code under `src/small-lang`. There, you can also find multiple files with the `.small`
 extension, which can be used as examples of the language.
+

docs/book/src/backends/db_backend.md

@@ -0,0 +1,36 @@
+## DB backend
+
+### Languages
+
+The DB backend is currently used for interpreted languages, like scripting and blockchain/zero knowledge languages:
+
+1. [Noir](./db-backend/noir.md)
+1. [Ruby](./db-backend/ruby.md)
+1. [Python(prototype not finished yet)](./db-backend/py.md)
+1. [Lua(not done: just a plan)](./db-backend/lua.md)
+1. [small(a toy interpreter for dogfooding)](./db-backend/small.md)
+
+If you're interested in db-backend support for those languages or for now ones, you can discuss with us on our [GitHub issue tracker](https://github.com/metacraft-labs/codetracer/issues),
+in the [GitHub repo discussions forum](https://github.com/metacraft-labs/codetracer/discussions) or in our [discord chat server](https://discord.gg/aH5WTMnKHT). We welcome contributors!
+
+### Additional directions
+
+However we're exploring using this method for emulators(and conversely, for emulating native languages) in the future.
+
+Also, languages can have both native(AOT) and interpreted implementations: 
+so we can more easily support e.g. interpreters/VM-s of certain languages with this backend currently.
+
+### Implementation
+
+As the name suggests, this backend is implemented as a database. Due to its properties, this backend provides
+a more complete feature set - one of the reasons it was ready first.
+
+For each language, we define a tracer, which instruments or hooks into the interpreter, VM or program and 
+records all the data, that we need in a trace folder. 
+
+We postprocess and index that trace into a database-like structure. Currently this happens in the beginning of the replay.
+For now the data in the database is represented by a runtime-only Rust structure, however
+we plan on storing it as a file, which can be memory-mapped and on separating the postprocessing/indexing step as 
+an optionally independent action.
+
+

docs/book/src/backends/db_backend/noir.md

@@ -0,0 +1 @@
+# Noir

docs/book/src/backends/rr-backend/c_and_cpp.md

@@ -0,0 +1,6 @@
+## C & C++
+
+We have partial support for C and C++ in the RR backend. C generally has good support, though it is less advanced than
+the Rust implementation.
+
+Support for C++ is still experimental, due to missing support for many C++ constructs.

docs/experimental-documentation/Introduction/Backends/CoreBackend/Golang.md → docs/book/src/backends/rr-backend/go.md

@@ -1,2 +1,4 @@
+# Go
+
 We have developed а basic integration that supports small parts of the Golang language in the Core backend. 
-Development on a more complete revision is scheduled to start soon!
+Development of a more complete revision is scheduled to start soon!

docs/book/src/backends/rr-backend/nim.md

@@ -0,0 +1,18 @@
+# Nim
+
+We have partial support for the Nim backend, with it being somewhat behind in features compared to C, but ahead,
+compared to C++.
+
+The Nim backend only supports Nim 1 as of today, but there are plans to add support for Nim 2.
+
+
+Some additional future goals that have been announced in the Nim forum are:
+
+* First-class support for macro-generated code - macro generated code should be indistinguishable from regular code. You can expand it layer by layer, set breakpoints and tracepoints anywhere within it, and step through it in all the familiar ways. (or at least part of those)
+* Precise debug info - you should be able to set breakpoints and tracepoints at the level of individual sub-expressions and you can switch between debugging Nim, C or assembly code at any time.
+* Compile-time debugging - with few clicks in your text editor, you should be able to trace the execution of the Nim VM itself.
+
+> [!NOTE]
+> For now we're focused on the beta and initial releases of the RR backend though, and on initial releases for some of the language supports.
+> This means the initial support for Nim would be focused on the same features as the other languages: stepping, tracepoints, event log, calltrace, omniscience, state, history. We do hope we'll be able to continue with some of the more advanced goals afterwards.
+

docs/book/src/backends/rr-backend/rust.md

@@ -0,0 +1,3 @@
+# Rust
+
+Our backend for Rust is the most developed out of all languages in the RR backend, with it being closest to MVP status(but still non trivial amount of work left).

docs/book/src/backends/rr_backend.md

@@ -0,0 +1,32 @@
+## RR backend
+
+The RR backend is used for systems programming languages, like:
+
+1. [C & C++](./rr-backend/c_and_cpp.md)
+1. [Rust](./rr-backend/rust.md)
+1. [Nim](./rr-backend/nim.md)
+1. [Go](./rr-backend/go.md)
+
+Under the hood, it uses a patched version of the [RR debugger](https://rr-project.org/), as well as the standard
+GDB debugger.
+
+We are still not distributing it, but we might need to rethink the GDB usage before that.
+
+The Core backend is based on a dispatcher that maintains a pool of rr replay processes. These processes allow you to jump back and forward in time, while also
+preloading information, such as the information, needed for omniscience and calltrace, as well as managing other features, such as tracepoints. This logic is
+currently implemented using python code using the gdb API. 
+(If required, this can be changed to use the lldb API or something more custom, like a dwarf lib-based solution. We are currently using the rust `gimli` DWARF lib for a tool, that helps with recording/metadata in the RR backend).
+
+These are the main differences, when compared to the DB Backend, based on the way RR works:
+
+1. Only non-deterministic events are recorded.
+1. The program's different states are navigated through re-execution.
+1. More practical for recording real-world software(RR was originally created by Mozilla to debug Firefox)
+1. Some features of CodeTracer are more challenging to implement, compared to the [DB Backend](./backends/db-backend.md)
+
+> ![NOTE]
+> This backend is in active development, and is yet to be released to the wider public. 
+>
+> It is currently proprietary and it might remain closed source. 
+> Open sourcing it is also possible, if we find a suitable business model.
+

docs/experimental-documentation/Introduction/BuildSystem.md → docs/book/src/building_and_packaging/build_systems.md

@@ -1,12 +1,18 @@
+## Build systems
+
 Codetracer uses nix, just, tup and direnv as parts of its build system.
 
 ## Breakdown of the different components
+
 ### Nix
-The Nix package manager/build system deals with managing all dependencies, required by CodeTracer, as well as packaging
-it for NixOS, and other operating systems through an AppImage generator utility.
+
+The [Nix](https://nixos.org/) package manager/build system deals with managing all dependencies, required by CodeTracer, as well as packaging
+it for NixOS. It can package for other operating systems through an AppImage generator utility, however currently we are building our
+appimages with custom shell scripts in `appimage-scripts`.
 
 ### Just
-The Just build system is used to trigger build commands. It's semi-analogous to a Makefile. The following commands are
+
+The [just](https://just.systems/) task runner is used to trigger build commands and various other actions. It's semi-analogous to a Makefile. The following commands are
 the most widely-used:
 
 1. `just build` - Builds the project with Tup and starts the automatic build process, which is used for active development
@@ -21,16 +27,20 @@ the most widely-used:
 1. `just reset-config` - Resets the user's configuration if it's incompatible with the latest version of CodeTracer. Further [documentation](https://dev-docs.codetracer.com/Introduction/Configuration)
 
 ### Tup
-The Tup build system is used for local builds of CodeTracer and deals with calling the actual low-level build instructions.
+
+The [Tup](https://gittup.org/tup/) build system is used for local builds of CodeTracer and deals with calling the actual low-level build instructions.
 
 ### Direnv
-The `direnv` utility sets up your local environment for using CodeTracer.
+The [direnv](https://direnv.net/) utility sets up your local environment for using CodeTracer.
 
 ## Packaging
-### Detailed breakdown of the Nix package
+
+### More detailed breakdown of the Nix package
+
 Coming soon!
 
 ### Packaging for non-NixOS distributions
+
 > [!TIP]
 > If you're a user that wants a package for your distribution contact us. We're currently in the process of creating
 > packages for popular distributions, such as Debian/Ubuntu, Fedora/RHEL, Arch Linux, Gentoo, Void, etc. 
@@ -40,7 +50,10 @@ which you can install to `/usr/bin`. Along with it, you should also install our
 from `resources/` to the needed directories, such as `/usr/share/pixmaps` and `/usr/share/applications`.
 
 ### Packaging for Windows(DB-backend only)
+
 Coming soon!
 
 ### Packaging for macOS(DB-backend only)
+
 Coming soon!
+