Skip to content

Commit 017c8a0

Browse files
wildjameswildjames
authored
New: [AEA-4540] - Add secret scanning (#5)
## Summary - ✨ New Feature ### Details Add a step which runs the secret scanning container given [here](https://github.com/NHSDigital/software-engineering-quality-framework/blob/main/tools/nhsd-git-secrets/nhsd-git-secrets.dockerfile)
1 parent 5bc62f5 commit 017c8a0

File tree

8 files changed

+190
-522
lines changed

8 files changed

+190
-522
lines changed

‎.devcontainer/Dockerfile‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@ RUN apt-get update \
99
jq apt-transport-https ca-certificates gnupg-agent \
1010
software-properties-common bash-completion python3-pip make libbz2-dev \
1111
libreadline-dev libsqlite3-dev wget llvm libncurses5-dev libncursesw5-dev \
12-
xz-utils tk-dev liblzma-dev netcat libyaml-dev
12+
xz-utils tk-dev liblzma-dev netcat libyaml-dev pre-commit
1313

1414
USER vscode
1515

‎.devcontainer/devcontainer.json‎

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,15 @@
1515
"source=${env:HOME}${env:USERPROFILE}/.npmrc,target=/home/vscode/.npmrc,type=bind"
1616
],
1717
"containerUser": "vscode",
18-
"features": {},
18+
"remoteEnv": { "LOCAL_WORKSPACE_FOLDER": "${localWorkspaceFolder}" },
19+
"postAttachCommand": "docker build -f /workspaces/eps-workflow-quality-checks/dockerfiles/nhsd-git-secrets.dockerfile -t git-secrets . && pre-commit install --install-hooks -f",
20+
"features": {
21+
"ghcr.io/devcontainers/features/docker-outside-of-docker:1": {
22+
"version": "latest",
23+
"moby": "true",
24+
"installDockerBuildx": "true"
25+
}
26+
},
1927
"customizations": {
2028
"vscode": {
2129
"extensions": [

‎.github/workflows/quality-checks.yml‎

Lines changed: 54 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -4,18 +4,62 @@ on:
44
workflow_call:
55
secrets:
66
SONAR_TOKEN:
7-
required: true
7+
required: false
8+
inputs:
9+
install_java:
10+
type: boolean
11+
description: "If true, the action will install java into the runner, separately from ASDF."
12+
default: false
13+
required: false
814

915
jobs:
1016
quality_checks:
1117
runs-on: ubuntu-22.04
1218
steps:
19+
- uses: actions/setup-java@v4
20+
if: ${{ inputs.install_java }}
21+
with:
22+
java-version: '21'
23+
distribution: 'corretto'
24+
1325
- name: Checkout code
1426
uses: actions/checkout@v4
1527
with:
1628
ref: ${{ env.BRANCH_NAME }}
1729
fetch-depth: 0
1830

31+
# Must be done before anything installs, or it will check dependencies for secrets too.
32+
- name: Ensure .gitallowed exists, for secret scanning
33+
run: |
34+
if [ ! -f ".gitallowed" ]; then
35+
echo "Creating empty .gitallowed file"
36+
touch .gitallowed
37+
fi
38+
echo "./nhsd-rules-deny.txt:10" >> .gitallowed
39+
echo "Allowing the following regex patterns:"
40+
cat .gitallowed
41+
42+
- name: Install git-secrets
43+
run: |
44+
sudo apt-get update
45+
sudo apt-get install -y git curl
46+
git clone https://github.com/awslabs/git-secrets.git /tmp/git-secrets
47+
cd /tmp/git-secrets
48+
sudo make install
49+
50+
- name: Download regex patterns
51+
run: |
52+
curl -L https://raw.githubusercontent.com/NHSDigital/software-engineering-quality-framework/main/tools/nhsd-git-secrets/nhsd-rules-deny.txt -o nhsd-rules-deny.txt
53+
54+
- name: Configure git-secrets
55+
run: |
56+
git-secrets --register-aws
57+
git-secrets --add-provider -- cat nhsd-rules-deny.txt
58+
59+
- name: Run secrets scan
60+
run: |
61+
git-secrets --scan-history .
62+
1963
# using git commit sha for version of action to ensure we have stable version
2064
- name: Install asdf
2165
uses: asdf-vm/actions/setup@05e0d2ed97b598bfce82fd30daf324ae0c4570e6
@@ -258,9 +302,17 @@ jobs:
258302

259303
- name: Generate and check SBOMs
260304
uses: NHSDigital/eps-action-sbom@main
261-
305+
306+
- name: "check is SONAR_TOKEN exists"
307+
env:
308+
super_secret: ${{ secrets.SONAR_TOKEN }}
309+
if: ${{ env.super_secret != '' }}
310+
run: echo "RUN_SONAR=true" >> "$GITHUB_ENV"
311+
312+
262313
- name: SonarCloud Scan
263314
uses: SonarSource/sonarcloud-github-action@master
315+
if: ${{ env.RUN_SONAR == 'true' }}
264316
env:
265317
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
266318
SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}

‎.pre-commit-config.yaml‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
repos:
2+
- repo: local
3+
hooks:
4+
- id: git-secrets
5+
name: Git Secrets
6+
description: git-secrets scans commits, commit messages, and --no-ff merges to prevent adding secrets into your git repositories.
7+
entry: bash
8+
args:
9+
- -c
10+
- 'docker run -v "$LOCAL_WORKSPACE_FOLDER:/src" git-secrets --pre_commit_hook'
11+
language: system

‎README.md‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,12 +5,49 @@ A workflow to run the quality checks for EPS repositories. The steps executed by
55
- **Generate and Check SBOMs**: Creates Software Bill of Materials (SBOMs) to track dependencies for security and compliance. Uses [THIS](https://github.com/NHSDigital/eps-action-sbom) action.
66
- **Run Linting**
77
- **Run Unit Tests**
8+
- **Scan git history for secrets**: Scans for secret-like patterns, using https://github.com/NHSDigital/software-engineering-quality-framework/blob/main/tools/nhsd-git-secrets/git-secrets
89
- **SonarCloud Scan**: Performs code analysis using SonarCloud to detect quality issues and vulnerabilities.
910
- **Validate CloudFormation Templates** (*Conditional*): If CloudFormation, AWS SAM templates or CDK are present, runs `cfn-lint` (SAM and cloudformation only) and `cfn-guard` to validate templates against AWS best practices and security rules.
1011
- **CDK Synth** (*Conditional*): Runs `make cdk-synth` if packages/cdk folder exists
1112
- **Check Licenses**: Runs `make check-licenses`.
1213
- **Check Python Licenses** (*Conditional*): If the project uses Poetry, scans Python dependencies for incompatible licenses.
1314

15+
The secret scanning also has a dockerfile, which can be run against a repo in order to scan it manually (or as part of pre-commit hooks). This can be done like so:
16+
```bash
17+
docker build -f https://raw.githubusercontent.com/NHSDigital/eps-workflow-quality-checks/refs/tags/v3.0.0/dockerfiles/nhsd-git-secrets.dockerfile -t git-secrets .
18+
docker run -v /path/to/repo:/src git-secrets --scan-history .
19+
```
20+
For usage of the script, see the [source repo](https://github.com/NHSDigital/software-engineering-quality-framework/blob/main/tools/nhsd-git-secrets/git-secrets). Generally, you will either need `--scan -r .` or `--scan-history .`. The arguments default to `--scan -r .`, i.e. scanning the current state of the code.
21+
22+
In order to enable the pre-commit hook for secret scanning (to prevent developers from committing secrets in the first place), add the following to the `.devcontainer/devcontainer.json` file:
23+
```json
24+
{
25+
"remoteEnv": { "LOCAL_WORKSPACE_FOLDER": "${localWorkspaceFolder}" },
26+
"postAttachCommand": "docker build -f https://raw.githubusercontent.com/NHSDigital/eps-workflow-quality-checks/refs/tags/v3.0.0/dockerfiles/nhsd-git-secrets.dockerfile -t git-secrets . && pre-commit install --install-hooks -f",
27+
"features": {
28+
"ghcr.io/devcontainers/features/docker-outside-of-docker:1": {
29+
"version": "latest",
30+
"moby": "true",
31+
"installDockerBuildx": "true"
32+
}
33+
}
34+
}
35+
```
36+
And the this pre-commit hook to the `.pre-commit-config.yaml` file:
37+
```yaml
38+
repos:
39+
- repo: local
40+
hooks:
41+
- id: git-secrets
42+
name: Git Secrets
43+
description: git-secrets scans commits, commit messages, and --no-ff merges to prevent adding secrets into your git repositories.
44+
entry: bash
45+
args:
46+
- -c
47+
- 'docker run -v "$LOCAL_WORKSPACE_FOLDER:/src" git-secrets --pre_commit_hook'
48+
language: system
49+
```
50+
1451
# Usage
1552
1653
## Inputs

‎dockerfiles/nhsd-git-secrets.dockerfile‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
# This dockerfile allows you to run the NHS flavour of secret-scanning on the mounted directory.
2+
# It assumes nothing about the local filesystem, so can be built remotely by using
3+
# docker build -t git-secrets -f https://raw.githubusercontent.com/NHSDigital/eps-workflow-quality-checks/refs/tags/<VERSION>/dockerfiles/nhsd-git-secrets.dockerfile .
4+
# When run:
5+
# docker run -v /path/to/code:/src git-secrets [ARGS]
6+
# It will default to scanning the local directory (note that it must be a git repository) as it currently is (no history scan).
7+
# However, script arguments (https://github.com/NHSDigital/software-engineering-quality-framework/blob/main/tools/nhsd-git-secrets/git-secrets)
8+
# can also be supplied. Remember to provide the trailing `.`, or the script would assume
9+
# you want to scan STDIN.
10+
# The default arguments are `--scan -r .
11+
12+
FROM ubuntu:latest
13+
14+
RUN echo "Installing required modules"
15+
RUN apt-get update
16+
RUN apt-get -y install curl git build-essential
17+
18+
WORKDIR /secrets-scanner
19+
20+
RUN echo "Downloading secrets scanner"
21+
RUN curl https://codeload.github.com/awslabs/git-secrets/tar.gz/master | tar -xz --strip=1 git-secrets-master
22+
23+
RUN echo "Installing secrets scanner"
24+
RUN make install
25+
26+
RUN echo "Downloading regex files from engineering-framework"
27+
RUN curl https://codeload.github.com/NHSDigital/software-engineering-quality-framework/tar.gz/main | tar -xz --strip=3 software-engineering-quality-framework-main/tools/nhsd-git-secrets/nhsd-rules-deny.txt
28+
29+
RUN echo '#!/usr/bin/env bash\n\
30+
\n\
31+
git config --global --add safe.directory /src\n\
32+
# Register additional providers: adds AWS by default \n\
33+
echo "Configuring secrets scanner" \n\
34+
/secrets-scanner/git-secrets --register-aws \n\
35+
/secrets-scanner/git-secrets --add-provider -- cat /secrets-scanner/nhsd-rules-deny.txt \n\
36+
\n\
37+
/secrets-scanner/git-secrets $@ \n ' >> /entrypoint.sh
38+
RUN chmod +x /entrypoint.sh
39+
40+
WORKDIR /src
41+
ENTRYPOINT [ "/entrypoint.sh" ]
42+
CMD [ "--scan", "-r", "." ]

0 commit comments

Comments
 (0)