Git secrets windows and linux (#276)

* Rename deny rules file

* Added Linux and Windows install scripts and docs

* Reinstated changes

Author: karstenabc

tools/nhsd-git-secrets/README-linux-workstation.md

@@ -0,0 +1,65 @@
+# Setup (Linux/WSL only)
+
+## Setup
+
+Make sure to copy the `nhd-git-secrets` folder into the root of the project repository, and then navigate the terminal to the repo root
+
+* `cd nhsd-git-secrets`
+* `cp .gitallowed-base ../.gitallowed`
+* `./install-linux.sh`
+
+Next time you do a commit the git secrets hook should be invoked.
+
+### Troubleshooting
+
+You should have 3 new files in the `.git/hooks` folder in the repository. If these are not present, then make sure you have ran the install script and that this script ran successfully.
+If you get an output containing:
+
+```bash
+[3/5] Adding Git Hooks
+./install-linux.sh: line 18: git-secrets: command not found
+```
+
+* Run this command anywhere: `export PATH="$HOME/git-secrets/bin":$PATH`
+* Then re-run the install script (`./nhsd-git-secrets/install-linux.sh`)
+
+### Custom configuration (per repo / per service team)
+
+* Add individual regex expressions to the existing `repo_root/nhsd-git-secrets-nhsd-rules-deny.txt` file
+* Or, create your own file for regex rules and add it as a provider within the [pre-commit script](pre-commit.sh) e.g.:
+ `./nhsd-git-secrets/git-secrets --add-provider -- cat nhsd-git-secrets/nhsd-rules-deny.txt`
+
+* Add file/dir excludes within the `repo_root/.gitallowed`, e.g. `.*terraform.tfstate.*:*`
+
+* Control full scan vs staged files scan within [pre-commit script](pre-commit.sh) by commenting/uncommenting the mode to run e.g.:
+
+ ```bash
+ # Just scan the files changed in this commit
+ # ./nhsd-git-secrets/git-secrets --pre_commit_hook
+
+ # Scan all files within this repo for this commit
+ ./nhsd-git-secrets/git-secrets --scan
+ ```
+
+## Testing and Usage
+
+To test that the hooks have been enabled correctly:
+
+* make sure you have done git add if you have changed anything within git-Secrets
+* create a file containing one or more patterns from the `git-secrets/nhsd-rules-deny.txt` file (e.g.: `password = “test”`)
+* stage and commit the file
+
+You should see an output similar to: `“[ERROR] Matched one or more prohibited patterns…”`.
+
+**Note** This message may appear differently depending on the tools used.
+
+> If you have a *false-positive* match, and your changes do not contain sensitive credentials then you can add the `--no-verify` flag to the commit command to **skip the checking**.
+
+## Docker version
+
+Alternatively, you might find this [dockerfile](nhsd-git-secrets.dockerfile) convenient, which:
+
+1. Copies your source code into a docker image
+1. Downloads latest version of the secret scanner tool
+1. Downloads latest regex patterns from software-engineering-quality-framework
+1. Runs a scan

tools/nhsd-git-secrets/README-mac-workstation.md

@@ -1,8 +1,6 @@
 # Setup (Mac only)
 
-Ensure you have the pre-commit framework set up first:
-
-https://pre-commit.com/
+Ensure you have the [pre-commit framework](https://pre-commit.com/) set up first.
 
 ## TL;DR
 
@@ -33,33 +31,34 @@ Then:
 
 Next time you do a commit the git secrets hook should be invoked.
 
-# Custom configuration (per repo / per service team)
+## Custom configuration (per repo / per service team)
 
-* Add individual regex expressions to nhsd-rules.txt
-* Add regex rules files within wrapper.sh e.g.
+* Add individual regex expressions to the existing `repo_root/nhsd-git-secrets-nhsd-rules-deny.txt` file
+* Or, create your own file for regex rules and add it as a provider within the [pre-commit script](pre-commit.sh) e.g.:
 
- `git secrets --add-provider -- cat git-secrets/nhsd-rules.txt`
+ `git secrets --add-provider -- cat git-secrets/nhsd-rules-deny.txt`
 
-* Add file/dir excludes within .gitallowed, e.g. `.*terraform.tfstate.*:*`
+* Add file/dir excludes within the `repo_root/.gitallowed` file, e.g. `.*terraform.tfstate.*:*`
 
-* Control full scan vs staged files scan within wrapper.sh by commenting/uncommenting the mode to run e.g.:
+* Control full scan vs staged files scan within [pre-commit (mac) script](pre-commit-mac.sh) by commenting/uncommenting the mode to run e.g.:
 
- ```
+ ```bash
  # Just scan the files changed in this commit
  # git secrets --pre_commit_hook
 
  # Scan all files within this repo for this commit
  git secrets --scan
  ```
 
-# Testing-a
+## Testing-a
 
 * make sure you have done git add if you have changed anything within git-Secrets
 * Run: `pre-commit run git-secrets`
 
-# Docker version
+## Docker version
 
 Alternatively, you might find this [dockerfile](nhsd-git-secrets.dockerfile) convenient, which:
+
 1. Copies your source code into a docker image
 1. Downloads latest version of the secret scanner tool
 1. Downloads latest regex patterns from software-engineering-quality-framework

tools/nhsd-git-secrets/README-windows-workstation.md

@@ -0,0 +1,61 @@
+# Setup (Windows only)
+
+## Prerequisites
+
+Ensure you have permissions to exceute powershell scripts. If this is not possible, then you may be able to use WSL instead and setup git-secrets following the [guide for linux](README-linux-workstation.md).
+
+## Setup
+
+Make sure to copy the `nhd-git-secrets` folder into the root of the project repository, and then navigate the terminal (powershell) to the repo root
+
+* `cd .\nhsd-git-secrets\`
+* `cp .\gitallowed-base ../.gitallowed`
+* `.\install-windows.ps1`
+
+Next time you do a commit the git secrets hook should be invoked.
+
+### Troubleshooting
+
+You should have 3 new files in the `.git/hooks` folder in the repository. If these are not present, then make sure you have ran the install script.
+
+### Custom configuration (per repo / per service team)
+
+* Add individual regex expressions to the existing `repo_root/nhsd-git-secrets-nhsd-rules-deny.txt` file
+* Or, create your own file for regex rules and add it as a provider within the [pre-commit script](pre-commit.sh) e.g.:
+
+ `./nhsd-git-secrets/git-secrets --add-provider -- cat nhsd-git-secrets/nhsd-rules-deny.txt`
+
+* Add file/dir excludes within the `repo_root/.gitallowed`, e.g. `.*terraform.tfstate.*:*`
+
+* Control full scan vs staged files scan within [pre-commit script](pre-commit.sh) by commenting/uncommenting the mode to run e.g.:
+
+ ```bash
+ # Just scan the files changed in this commit
+ # ./nhsd-git-secrets/git-secrets --pre_commit_hook
+
+ # Scan all files within this repo for this commit
+ ./nhsd-git-secrets/git-secrets --scan
+ ```
+
+## Testing and Usage
+
+To test that the hooks have been enabled correctly:
+
+* make sure you have done git add if you have changed anything within git-Secrets
+* create a file containing one or more patterns from the `git-secrets/nhsd-rules-deny.txt` file (e.g.: `password = “test”`)
+* stage and commit the file
+
+You should see an output similar to: `“[ERROR] Matched one or more prohibited patterns…”`.
+
+**Note** This message may appear differently depending on the tools used.
+
+> If you have a *false-positive* match, and your changes do not contain sensitive credentials then you can add the `--no-verify` flag to the commit command to **skip the checking**.
+
+## Docker version
+
+Alternatively, you might find this [dockerfile](nhsd-git-secrets.dockerfile) convenient, which:
+
+1. Copies your source code into a docker image
+1. Downloads latest version of the secret scanner tool
+1. Downloads latest regex patterns from software-engineering-quality-framework
+1. Runs a scan

tools/nhsd-git-secrets/README.md

@@ -13,36 +13,44 @@ Essentially, we want to avoid the NHS facing [potentially dire consequences](htt
 
 ## How to get started
 
-If your team isn't doing secrets scanning at all yet, the fundamental first step is to understand the current state of the art. Use the [Macbook](README-mac-workstation.md) or Windows (coming soon...) guides to set up and run Git-Secrets for a nominated team member. Run the tooling, and ascertain whether there's any immediate actions to be taken.
+If your team isn't doing secrets scanning at all yet, the fundamental first step is to understand the current state of the art. Use the following guides to set up and run Git-Secrets for a nominated team member:
 
-## Getting to green
+* [macOS](README-mac-workstation.md)
+* [Linux/WSL](README-linux-workstation.md)
+* [Windows](README-windows-workstation.md)
+
+Run the tooling, and ascertain whether there's any immediate actions to be taken.
+
+## Ongoing checks
 
 Once you've verified there's no urgent actions on your code, the next steps towards getting to green are:
 
 1. Ensure every team member is doing local scans. Stopping secrets before code has been committed is cheap, removing them from git history is expensive.
 2. Run these same scripts as part of your deployment pipelines as a second line of defence.
 
-## Consider using OIDC instead of secrets
+## Other ways of keeping credentials out of your code
+
+### Consider using OIDC Authentication instead of passwords
 
 OpenID Connect allows federated authentication from pipeline workflows to AWS and Azure without storing credentials in repository secrets at all, so no expiry to manage. GitHub documentation for achieving this for:
 
-- [AWS](https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/configuring-openid-connect-in-amazon-web-services)
-- [Azure](https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/configuring-openid-connect-in-azure)
-- [Google](https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/configuring-openid-connect-in-google-cloud-platform)
+* [AWS](https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/configuring-openid-connect-in-amazon-web-services)
+* [Azure](https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/configuring-openid-connect-in-azure)
+* [Google](https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/configuring-openid-connect-in-google-cloud-platform)
 
 Configuration for OIDC is light. See the below example GitHub Actions workflow excerpt which connects to both AWS and Azure:
 
 ```yaml
 steps:
   - name: Checkout
     uses: actions/checkout@v3
-
+​
   - name: Configure AWS STS credentials via OIDC
     uses: aws-actions/configure-aws-credentials@v1
     with:
       role-to-assume: ${{ secrets.AWS_ROLE_ID }}
       aws-region: eu-west-2
-
+​
   - name: Configure Azure identity token via OIDC
     uses: azure/login@v1
     with:

tools/nhsd-git-secrets/full-history-scan.sh

@@ -6,7 +6,7 @@ export PATH=$PATH:.
 
 # # These only need to be run once per workstation/slave/agent but are included to try and ensure they are present
 ./git-secrets --register-aws
-./git-secrets --add-provider -- cat nhsd-git-secrets/nhsd-rules-linux.txt
+./git-secrets --add-provider -- cat nhsd-git-secrets/nhsd-rules-deny.txt
 ./git-secrets --add --allowed '.*git-secrets/.*:*'
 ./git-secrets --add --allowed '.*terraform.tfstate.*:*'
 

tools/nhsd-git-secrets/install-linux.sh

@@ -0,0 +1,31 @@
+echo "[1/5] Cloning Git Secrets"
+installPath="$HOME/git-secrets-temp"
+if [ -d "$installPath" ];
+then
+  echo "Git secrets already cloned"
+else
+  git clone https://github.com/awslabs/git-secrets.git $installPath
+fi
+
+echo "" && echo "[2/5] Installing Git Secrets and Adding to PATH"
+pushd $installPath
+make install PREFIX="$HOME/git-secrets"
+echo 'export PATH="$HOME/git-secrets/bin":$PATH' >> ~/.bashrc
+source ~/.bashrc
+popd
+
+echo "" && echo "[3/5] Adding Git Hooks"
+git-secrets --install -f
+
+echo "" && echo "[4/5] Removing Temp Git Secrets Repo"
+rm -rf $installPath
+
+echo "" && echo "[5/5] Updating Pre-Commit Hook"
+projectRoot=$(dirname $PWD)
+preCommitHook="$projectRoot/.git/hooks/pre-commit"
+hookScript="nhsd-git-secrets/pre-commit.sh"
+replaceString='git secrets --pre_commit_hook -- "$@"'
+sed -i -e "s,$replaceString,./$hookScript," "$preCommitHook"
+chmod +x "$projectRoot/$hookScript"
+
+echo "" && echo "Git Secrets Installation Complete"

tools/nhsd-git-secrets/install-windows.ps1

@@ -0,0 +1,55 @@
+Write-Host "[1/6] Cloning Git Secrets"
+$installPath = $($Env:USERPROFILE + "\git-secrets-temp")
+if (-not (Test-Path $installPath))
+{
+  git clone https://github.com/awslabs/git-secrets.git $installPath
+}
+else
+{
+  Write-Host "Git secrets already cloned"
+}
+
+Write-Host "`n[2/6] Installing Git Secrets"
+Push-Location $installPath
+& ".\install.ps1"
+Pop-Location
+
+Write-Host "`n[3/6] Adding Git Hooks"
+git secrets --install
+
+Write-Host "`n[4/6] Removing Temp Git Secrets Repo"
+Remove-Item $installPath -Recurse -Force
+
+Write-Host "`n[5/6] Adding Git Bash to PATH"
+Write-Host "Checking if git bash already exists in path..."
+$currentPath = [Environment]::GetEnvironmentVariable("PATH", "User")
+$gitInstallDirectory = "C:\Program Files\Git\bin"
+if ($currentPath -notlike "*$gitInstallDirectory*")
+{
+    Write-Host "Adding to path..."
+    $newPath = $currentPath
+    if (-not ($newPath.EndsWith(";")))
+    {
+        $newPath = $newPath + ";"
+    }
+    $newPath = $newPath + $gitInstallDirectory
+    [Environment]::SetEnvironmentVariable("PATH", $newPath, "User")
+    Write-Host "Added to path"
+}
+else
+{
+    Write-Host "Already in Path"
+}
+
+Write-Host "`n[6/6] Updating Pre-Commit Hook"
+$projectRoot = Split-Path -Path $PSScriptRoot -Parent
+$preCommitHook = $projectRoot + '\.git\hooks\pre-commit'
+(Get-Content $preCommitHook) | ForEach-Object { 
+  if ($_.ReadCount -eq 2) { 
+    $_ -replace '^.*$','./nhsd-git-secrets/pre-commit.sh' 
+  } else { 
+    $_ 
+  } 
+} | Set-Content $preCommitHook
+
+Write-Host "`nGit Secrets Installation Complete"

tools/nhsd-git-secrets/nhsd-git-secrets.dockerfile

@@ -44,7 +44,7 @@ WORKDIR /secrets-scanner/source
 RUN git init
 
 RUN echo "Downloading regex files from engineering-framework"
-RUN curl https://codeload.github.com/NHSDigital/software-engineering-quality-framework/tar.gz/main | tar -xz --strip=3 software-engineering-quality-framework-main/tools/nhsd-git-secrets/nhsd-rules-linux-mac.txt
+RUN curl https://codeload.github.com/NHSDigital/software-engineering-quality-framework/tar.gz/main | tar -xz --strip=3 software-engineering-quality-framework-main/tools/nhsd-git-secrets/nhsd-rules-deny.txt
 
 RUN echo "Copying allowed secrets list"
 COPY .gitallowed .
@@ -53,7 +53,7 @@ RUN echo .gitallowed
 # Register additional providers: adds AWS by default
 RUN echo "Configuring secrets scanner"
 RUN /secrets-scanner/git-secrets --register-aws
-RUN /secrets-scanner/git-secrets --add-provider -- cat nhsd-rules-linux-mac.txt
+RUN /secrets-scanner/git-secrets --add-provider -- cat nhsd-rules-deny.txt
 
 # build will fail here, if secrets are found
 RUN echo "Running scan..."

tools/nhsd-git-secrets/nhsd-rules-linux-mac.txt renamed to tools/nhsd-git-secrets/nhsd-rules-deny.txt

@@ -3,7 +3,7 @@
 # Note that this will be invoked by the git hook from the repo root, so cd .. isn't required
 
 # These only need to be run once per workstation but are included to try and ensure they are present
-./git-secrets --add-provider -- cat nhsd-git-secrets/nhsd-rules-linux-mac.txt
+./git-secrets --add-provider -- cat nhsd-git-secrets/nhsd-rules-deny.txt
 
 # Just scan the files changed in this commit
 ./git-secrets --pre_commit_hook