chore(readme): split README and CONTRIBUTING

Author: jguerreiro

CONTRIBUTING.md

@@ -0,0 +1,79 @@
+# Contributing
+
+## Architecture
+
+### Main overview
+
+All the git information can be found inside commits that are located inside git repositories
+Our tree element steps are the following:
+
+- Collect all repository URL's from an object (org, user, group).
+- Clone them with the appropriate authentication.
+- Run git commands to extract the information we need on each repository.
+- Gather data and store this information in a json file.
+
+### Implementation
+
+The root package is the abstract implementation of the extractor.
+
+It contains a Pipeline that extracts git information for every git artifact
+(currently a git file but we could support commit), of every repository of an organization.
+
+The cmd/src-fingerprint package contains the binary code.
+It reads from CLI and environment the configuration and run the Pipeline on an organization.
+
+## Development build and testing
+
+- Build binary
+
+  ```sh
+  go build ./cmd/src-fingerprint
+  ```
+
+- Set env var `VCS_TOKEN` to the GitHub Token or GitLab Token
+
+  ```sh
+  export VCS_TOKEN="<token>"
+  ```
+
+- Run and read doc
+
+  ```sh
+  ./src-fingerprint
+  ```
+
+- Run on a given user/group
+  ```sh
+  ./src-fingerprint --provider github --object Uber
+  ./src-fingerprint --provider-url http://gitlab.example.com --provider gitlab --object Groupe
+  ```
+
+## Performance considerations
+
+Streaming is prefered in this scenario to avoid accumulation in memory of objects.
+
+What we have done for now to improve performance:
+
+- Write object by object to output/file by using jsonl format by default
+- Clone using the native git executable. Natively written libraries tend to clone
+  in memory at some point.
+
+### To consider
+
+- Limiting go channel numbers
+
+## Libraries we use
+
+#### Providers
+
+- GitHub wrapper: "github.com/google/go-github/v18/github"
+- Gitlab go wrapper: "github.com/xanzy/go-gitlab"
+- Bitbucket wrapper: "github.com/suhaibmujahid/go-bitbucket-server/bitbucket"
+- Repository: None
+
+#### Cloning
+
+- native wrapped git command
+
+Using go-git resulted in in-memory cloning (stream to memory and then to directory).
+This caused too high peaks of memory unsuitable for small VMs.

README.md

@@ -77,67 +77,3 @@ src-fingerprint -p repository -o 'https://user:[email protected]/GitGuardian/g
 ```sh
 src-fingerprint -p repository -o 'https://github.com/GitGuardian/gg-shield.git'
 ```
-
-## Architecture
-
-### Main overview
-
-All the git information can be found inside commit that are located inside git repositories
-Our tree element step are the following:
-
-- Collect all repositories URL from the company.
-- Clone them with the appropriate authentication.
-- Run git commands to extract the information we need on each repository.
-- Gather data and store this information in a json file.
-
-### Implementation
-
-The root package is the abstract implementation of the extractor. It contains a Cloner, that clones a git repository.
-It contains a Pipeline that extracts git information for every git artifact (currently a git file but we could support commit), of every repository of an organization.
-
-The github package contains the implementation of the Github Provider.
-The gitlab package contains the implementation of the Gitlab Provider.
-
-The cmd/src-fingerprint package contains the binary code. It reads from CLI and environment the configuration and run the Pipeline on an organization.
-
-## Development build and testing
-
-- Build binary
-
-  ```sh
-  go build ./cmd/src-fingerprint
-  ```
-
-- Set env var `VCS_TOKEN` to the GitHub Token or GitLab Token
-
-  ```sh
-  export VCS_TOKEN="<token>"
-  ```
-
-- Run and read doc
-
-  ```sh
-  ./src-fingerprint
-  ```
-
-- Run on a given user/group
-  ```sh
-  ./src-fingerprint --provider github --object Uber
-  ./src-fingerprint --provider-url http://gitlab.example.com --provider gitlab --object Groupe
-  ```
-
-### Libraries we use
-
-#### Providers
-
-- GitHub wrapper: "github.com/google/go-github/v18/github"
-- Gitlab go wrapper: "github.com/xanzy/go-gitlab"
-- Bitbucket wrapper: "github.com/suhaibmujahid/go-bitbucket-server/bitbucket"
-- Repository: None
-
-#### Cloning
-
-- native wrapped git command
-
-Using go-git resulted in in-memory cloning (stream to memory and then to directory).
-This caused too high peaks of memory unsuitable for small VMs.