Rework documentation

Rework outline
Add getting started with interactive example

Author: LMG

docs/book.toml

@@ -1,5 +1,5 @@
 [book]
-authors = ["Christian Schilling"]
+authors = ["Christian Schilling", "Louis-Marie Givel"]
 language = "en"
 multilingual = false
 src = "src"

docs/src/.usecases.md

@@ -0,0 +1 @@
+# Use cases

docs/src/SUMMARY.md

@@ -2,7 +2,19 @@
 
 - [Intro](./intro.md)
 - [Use cases](./usecases.md)
+# Guide
+- [Getting started](./gettingstarted.md)
+- [Working with workspaces]()
+- [Importing projects]()
+- [Integrating with CI]()
+# Reference
 - [Filter syntax](./filters.md)
-- [Command Line usage](./cli.md)
 - [Proxy](./proxy.md)
 - [Workspaces](./workspace.md)
+- [Command line tools](./cli.md)
+- [Graphql API]()
+- [Handlebar generator]()
+- [The josh UI]()
+# Contributing
+- [Development tools]()
+- [Tracing]()

docs/src/gettingstarted.md

@@ -0,0 +1,65 @@
+# Getting Started
+
+This book will guide you into setting up the josh
+proxy to serve your own git repository.
+
+---
+***NOTE***
+
+All the commands in this book are included from the file `tutorial.t`
+which can be run with `cram tutorial.t`.
+
+---
+
+## Setting up the proxy
+
+Josh is distributed via [docker hub](https://hub.docker.com/r/esrlabs/josh-proxy),
+and is installed and started with the following command:
+
+```shell
+{{#include tutorial.t:docker_github}}
+```
+
+This starts josh as a proxy to github.com, in a docker container, 
+mounting the ./git\_data folder to the image for use by josh.
+
+## Cloning a repository
+
+Once josh is running, we can clone a repository through it.
+For example, let's clone josh:
+
+```shell
+{{#include tutorial.t:clone_full}}
+```
+
+As we can see, this repository is simply the normal josh one:
+
+```shell
+{{#include tutorial.t:ls_full}}
+```
+
+## Extracting a module
+
+Josh becomes interesting when we want to extract a module.
+Let's check out the josh repository again, but this time let's filter
+only the documentation out:
+
+```shell
+{{#include tutorial.t:clone_doc}}
+```
+
+Note the addition of `:/docs` at the end of the url.
+This is called a filter, and it instructs josh to only check out the
+given folder.
+
+Looking inside the repository, we now see that the history is quite
+different. Indeed, it contains only the commits pertaining to the 
+subfolder that we checked out.
+
+```shell
+{{#include tutorial.t:ls_doc}}
+```
+
+This repository is a real repository in which we can pull, commit, push,
+as with a regular one. Josh will take care of synchronizing it with
+the main one in a transparent fashion.

docs/src/intro.md

@@ -1,11 +1,8 @@
-# Just One Single History
+![Just One Single History](../banner.png)
 
 Josh combines the advantages of monorepos with those of multirepos by leveraging a blazingly-fast,
 incremental, and reversible implementation of git history filtering.
 
-This documentation describes the filtering mechanism, as well as
-the tools provided by Josh: the josh library, `josh-proxy` and `josh-filter`.
-
 ## Concept
 
 Traditionally, history filtering has been viewed as an expensive operation that should only be

docs/src/tutorial.t

@@ -0,0 +1,86 @@
+  $ export TESTTMP=${PWD}
+  $ mkdir git_data
+
+# starting josh
+ANCHOR: docker_github
+  $ docker run -d -p 8000:8000 -e JOSH_REMOTE=https://github.com -v josh-vol:$(pwd)/git_data esrlabs/josh-proxy:latest > josh.out
+ANCHOR_END: docker_github
+
+# waiting for josh to be running
+  $ until curl -s http://localhost:8000/
+  > do
+  >     sleep 0.1
+  > done
+
+# cloning josh
+ANCHOR: clone_full
+  $ git clone http://localhost:8000/esrlabs/josh.git
+  Cloning into 'josh'...
+  $ cd josh
+ANCHOR_END: clone_full
+
+# checking out a release tag to make the output dependable
+  $ git checkout r21.03.19.1 1>/dev/null 2>/dev/null
+ANCHOR: ls_full
+  $ ls
+  Cargo.lock
+  Cargo.toml
+  Dockerfile
+  Dockerfile.tests
+  LICENSE
+  Makefile
+  README.md
+  docs
+  josh-proxy
+  run-josh.sh
+  run-tests.sh
+  rustfmt.toml
+  scripts
+  src
+  static
+  tests
+  $ git log -2
+  commit fc6af1e10c865f790bff7135d02b1fa82ddebe29
+  Author: Christian Schilling <[email protected]>
+  Date:   Fri Mar 19 11:15:57 2021 +0100
+  
+      Update release.yml
+  
+  commit 975581064fa21b3a3d6871a4e888fd6dc1129a13
+  Author: Christian Schilling <[email protected]>
+  Date:   Fri Mar 19 11:11:45 2021 +0100
+  
+      Update release.yml
+ANCHOR_END: ls_full
+
+# cloning doc
+ANCHOR: clone_doc
+  $ cd ..
+  $ git clone http://localhost:8000/esrlabs/josh.git:/docs.git
+  Cloning into 'docs'...
+  $ cd docs
+ANCHOR_END: clone_doc
+
+# checking out a release tag to make the output dependable
+  $ git checkout r21.03.19.1 1>/dev/null 2>/dev/null
+ANCHOR: ls_doc
+  $ ls
+  book.toml
+  src
+  $ git log -2
+  commit dd26c506f6d6a218903b9f42a4869184fbbeb940
+  Author: Christian Schilling <[email protected]>
+  Date:   Mon Mar 8 09:22:21 2021 +0100
+  
+      Update docs to use docker for default setup
+  
+  commit ee6abba0fed9b99c9426f5224ff93cfee2813edc
+  Author: Louis-Marie Givel <[email protected]>
+  Date:   Fri Feb 26 11:41:37 2021 +0100
+  
+      Update proxy.md
+ANCHOR_END: ls_doc
+
+# cleanup
+  $ cd ${TESTTMP}
+  $ docker stop $(cat josh.out) >/dev/null

docs/src/usecases.md

@@ -1,9 +1,36 @@
 
 # Use cases
 
-## Workspaces in a mono-repo
+### Partial cloning
 
-Multiple projects, depending on a shared set of libraries, can live together in a single repository.
+Reduce scope and size of clones by treating subdirectories of the monorepo
+as individual repositories.
+
+```
+$ git clone http://josh/monorepo.git:/path/to/library.git
+```
+
+The partial repo will act as a normal git repository but only contain the files
+found in the subdirectory and only commits affecting those files.
+The partial repo supports both fetch as well as push operation.
+
+This helps not just to improve performace on the client due to having fever files in
+the tree,
+it also enables collaboration on parts of the monorepo with other parties
+utilizing git's normal distributed development features.
+For example, this makes it easy to mirror just selected parts of your
+repo to public github repositories or specific customers.
+
+### Project composition / Workspaces
+
+Simplify code sharing and dependency management. Beyond just subdirectories,
+Josh supports filtering, re-mapping and composition of arbitrary virtual repositories
+from the content found in the monorepo.
+
+The mapping itself is also stored in the repository and therefore versioned alongside
+the code.
+
+Multiple projects, depending on a shared set of libraries, can thus live together in a single repository.
 This approach is commonly referred to as “monorepo”, and was popularized by
 [Google](https://people.engr.ncsu.edu/ermurph3/papers/seip18.pdf), Facebook or Twitter to name a
 few.
@@ -38,10 +65,65 @@ dependencies = :/modules:[
     </tbody>
 </table>
 
+Workspaces act as normal git repos:
+
+```
+$ git clone http://josh/central.git:workspace=workspaces/project1.git
+```
+
 Each of the subprojects defines a `workspace.josh` file, defining the mapping between the original central.git repository and the hierarchy in use inside of the project.
 
 In this setup, project1 and project2 can seemlessly depend on the latest version of library1, while only checking out the part of the central monorepo that's needed for their purpose.
 What's more, any changes to a shared module will be synced in both directions.
 
 If a developper of the library1 pushed a new update, both projects will get the new version, and the developper will be able to check if they broke any test.
 If a developper of project1 needs to update the library, the changes will be automatically shared back into central, and project2.
+
+### Simplified CI/CD
+
+With everything stored in one repo, CI/CD systems only need to look into one source for each particular
+deliverable.
+However in traditional monorepo environments dependency mangement is handled by the build system.
+Build systems are usually taylored to specific languages and need their input already checked
+out on the filesystem.
+So the question:
+
+> "What deliverables are affected by a given commit and need to be rebuild?"
+
+cannot be answered without cloning the entire repository and understanding how the languages
+used handle dependencies.
+
+In particular when using C familiy languages, hidden dependencies on header files are easy to miss.
+For this reason limiting the visibility of files to the compiler by sandboxing is pretty much a requirement
+for reproducible builds.
+
+With Josh, each deliverable gets it's own virtual git repository with dependencies declared in the `workspace.josh`
+file. This means answering the above question becomes as simple as comparing commit ids.
+Furthermore due to the tree filtering each build is guaranteed to be perfectly sandboxed
+and only sees those parts of the monorepo that have actually been mapped.
+
+This also means the deliverables to be re-built can be determined without cloning any repos like
+typically necessary with normal build tools.
+
+### GraphQL API
+
+It is often desireable to access content stored in git without requiring a clone of the repository.
+This is usefull for CI/CD systems or web frontends such as dashboards.
+
+Josh exposes a GraphQL API for that purpose. For example, it can be used to find all workspaces currently
+present in the tree:
+
+```
+query {
+  rev(at:"refs/heads/master", filter:"::**/workspace.josh") {
+    files { path }
+  }
+}
+```
+
+### Caching proxy
+
+Even without using the more advanced features like partial cloning or workspaces,
+`josh-proxy` can act as a cache to reduce traffic between locations or keep your CI from
+performing many requests to the main git host.
+