Skip to content

Commit 8845fe2

Browse files
andyblundellstriggy77
andyblundell
and
striggy77
authored
Secret scanning notes (#131)
Co-authored-by: Adam S <[email protected]>
1 parent e8bcd20 commit 8845fe2

File tree

2 files changed

+33
-7
lines changed

2 files changed

+33
-7
lines changed

‎practices/security.md‎

Lines changed: 32 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -41,8 +41,8 @@ The remainder of this page gives more detailed and specific recommendations to b
4141
### Big picture
4242

4343
- Understand what **data** is processed or stored in the system
44-
- Assess the data classification e.g. personal confidential data (PCD), aggregate data, anonymised data, publicly available information. See [Health and social care data risk model](https://digital.nhs.uk/data-and-information/looking-after-information/data-security-and-information-governance/nhs-and-social-care-data-off-shoring-and-the-use-of-public-cloud-services/health-and-social-care-data-risk-model)
45-
- Understand governance and compliance requirements which apply, e.g. [NHS guidance](https://digital.nhs.uk/data-and-information/looking-after-information/data-security-and-information-governance/nhs-and-social-care-data-off-shoring-and-the-use-of-public-cloud-services/health-and-social-care-cloud-risk-framework), GDPR
44+
- Assess the data classification e.g. personal confidential data (PCD), aggregate data, anonymised data, publicly available information. See [Health and social care data risk model](https://digital.nhs.uk/data-and-information/looking-after-information/data-security-and-information-governance/nhs-and-social-care-data-off-shoring-and-the-use-of-public-cloud-services)
45+
- Understand governance and compliance requirements which apply, e.g. [NHS guidance](https://digital.nhs.uk/data-and-information/looking-after-information/data-security-and-information-governance/nhs-and-social-care-data-off-shoring-and-the-use-of-public-cloud-services), GDPR
4646
- Consider whether the data being processed is all **necessary** for the system to function, or whether it could be reduced to minimise risk
4747
- Prefer use of managed services to reduce attack surface where possible
4848
- Keep **audit** log(s) of user actions, software and infrastructure changes (e.g. git, CI/CD, [CloudTrail](https://aws.amazon.com/cloudtrail/))
@@ -67,10 +67,36 @@ The remainder of this page gives more detailed and specific recommendations to b
6767
- Prevent **[clickjacking](https://sudo.pagerduty.com/for_engineers/#clickjacking)** with `X-Frame-Options`
6868
- Be careful not to **leak information**, e.g. error messages, stack traces, headers
6969
- **Don't trust** yourself or others!
70-
- Code should be automatically scanned for secrets or other sensitive data using standalone tools like [GitGuardian](https://www.gitguardian.com/) or [awslabs git-secrets](https://github.com/awslabs/git-secrets) or built in tools in [GitLab](https://docs.gitlab.com/ee/user/application_security/secret_detection/) or [GitHub](https://docs.github.com/en/github/administering-a-repository/about-secret-scanning).
71-
- Review the configuration of whichever tools you use, for example [awslabs git-secrets](https://github.com/awslabs/git-secrets) does not scan for IP addresses by default, but you can change that by adding more regex expressions
72-
- TO DO: add config to register regex for IP4 and IP6
73-
- Ideally, run a tool like [OWASP SEDATED](https://github.com/OWASP/SEDATED) or [awslabs git-secrets](https://github.com/awslabs/git-secrets) locally on developer laptops (to catch secrets at source) with a backstop of a centralised tool such as [GitHub](https://docs.github.com/en/github/administering-a-repository/about-secret-scanning)
70+
- Code must be automatically scanned for secrets or other sensitive data:
71+
- It is recommended to use RegEx expressions to catch any potential issues in your code.
72+
73+
<details><summary>Base set of recommended RegEx expressions for scanning (click to expand)</summary>
74+
75+
``` yml
76+
Banned:
77+
78+
'[0-9a-fA-F]{1,4}:[0-9a-fA-F]{1,4}:[0-9a-fA-F]{1,4}:[0-9a-fA-F]{1,4}:[0-9a-fA-F]{1,4}:[0-9a-fA-F]{1,4}:[0-9a-fA-F]{1,4}:[0-9a-fA-F]{1,4}' # IPv6
79+
'[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}' # IPv4
80+
'[a-z]{2}-[a-z-]*-[1,2,3]\.rds\.amazonaws\.com' # AWS RDS (Aurora) endpoint
81+
'rds\.[a-z]{2}-[a-z-]*-[1,2,3]\.amazonaws\.com' # AWS RDS endpoint
82+
'dynamodb\.[a-z]{2}-[a-z-]*-[1,2,3]\.amazonaws\.com' # AWS DynamoDB endpoint
83+
'[a-z]{2}-[a-z-]*-[1,2,3]\.es\.amazonaws\.com' # AWS Elasticsearch endpoint
84+
'[a-z]*[1-3]\.cache\.amazonaws\.com' # AWS ElastiCache endpoint
85+
'hooks\.slack\.com/services/T[a-zA-Z0-9]*/B[a-zA-Z0-9]*/[a-zA-Z0-9]*' # Slack webhook URL
86+
'-----BEGIN[[:blank:]]CERTIFICATE-----' # SSL PEM certificate
87+
'-----BEGIN[[:blank:]]PRIVATE[[:blank:]]KEY-----' # SSL PEM key
88+
89+
Allowed:
90+
91+
'(000000000000|123456789012)' # AWS mock account numbers
92+
'(127\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}|10\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}|172\.(1[6-9|2[0-9]|3[0-1])\.[0-9]{1,3}\.[0-9]{1,3}|192\.168\.[0-9]{1,3}\.[0-9]{1,3}|0\.0\.0\.0|8\.8\.8\.8|8\.8\.4\.4|208\.67\.222\.222|208\.67\.220\.220)' # IPv4 exceptions
93+
```
94+
</details>
95+
- To catch any issues early and to minimise potential exposure, scan code on developer machines *before* code is committed to the code repository. Recommended solution options:
96+
- [awslabs git-secrets](https://github.com/awslabs/git-secrets)
97+
- [GitLeaks](https://github.com/zricethezav/gitleaks)
98+
- As a backstop, *also* enable server-side scanning within the code repository. Recommended solution options:
99+
- TO DO: more details... for example in [GitHub](https://docs.github.com/en/github/administering-a-repository/about-secret-scanning)
74100
- Be wary of any 3rd party JavaScript included on the page, e.g. for A/B testing, analytics
75101
- Pin dependencies at known versions to avoid unexpected updates
76102
- Scan dependencies for vulnerabilities, e.g. using [OWASP Dependency Check](https://www.owasp.org/index.php/OWASP_Dependency_Check) or [Snyk](https://snyk.io/)

‎quality-checks.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -38,7 +38,7 @@ We rate our applications against each of these checks as follows:
3838
| Integration tests | Functionality | Universal | Check interactions with other components and dependant systems. e.g. across microservices, authentication layers, database, third-party systems. Ideally includes full end-to-end testing across all components | When components are developed in isolation, it's vital that we regularly test them working together. Changes in one component can break the whole system | Builds fail if any tests fail | | |
3939
| API / contract tests | Functionality | Contextual | Check whether the API interface adheres to the agreed contract | Any API interface is an integration point with another component or a software systems. An extra care has to be taken to ensure compatibility and stability of that integration are maintained so that we don't break applications that depend on our APIs | Builds fail if any tests fail | Postman | [Automate Your API Tests with Postman](https://www.postman.com/use-cases/api-testing-automation/) |
4040
| UI tests | Functionality | Contextual | Check that the user interface components behave as expected, particularly checking the visual elements to verify that they are functioning according to requirements | As the only aspects of software that end users come into contact with, it is essential that these elements behave as expected and allow users to get only what they need from our software applications | Builds fail if any tests fail | | |
41-
| Secret scanning | Security | Universal | Check for secrets (e.g. passwords, IP addresses, etc) accidentally included in software code | This protects us against accidentally leaking secrets (in source code) which could compromise the security of the application | Builds fail if any of these patterns (with some exceptions) are found: IPv4/IPv6 address, domain name, AWS account number, password/token/key, high entropy e.g. SSH key | Git pre-commit hook, TBC | TBC |
41+
| Secret scanning | Security | Universal | Check for secrets (e.g. passwords, IP addresses, etc) accidentally included in software code | This protects us against accidentally leaking secrets (in source code) which could compromise the security of the application | Covering all of the patterns listed in the [security practices](/practices/security.md#application-level-security):<br/><br/>Full repository (including history) scan, and all secrets removed<br/><br/>And:<br/><br/>Local (on developer laptop) scanning to block commits containing the patterns <br/><br/>And:<br/><br/>Server-side scanning within the code repository for new commits containing the patterns| One-off scanning, and ongoing local scanning: [Git-secrets](https://github.com/awslabs/git-secrets)<br/>[GitLeaks](https://github.com/zricethezav/gitleaks)<br/><br/><br/>Code-repository scanning: TBC | [Security practices ("don't trust yourself or others")](/practices/security.md#application-level-security)|
4242
| Security code analysis | Security | Universal | Check for indications of possible security issues (for example injection weaknesses) | This gives fast feedback about security issues: it's not as thorough as security testing, but it's much quicker to execute, so both are important to achieve both rapid and thorough security testing | | | |
4343
| Security testing | Security | Contextual | Check for security issues (for example injection weaknesses) | More thorough than security code scanning, but much slower to execute, so both are important to achieve both rapid and thorough security testing | | | |
4444
| Dependency scanning | Security | Universal | Check for security issues and vulnerabilities in dependent areas of code that are outside of our direct control | Without this we have no way of knowing of any issues or security vulnerabilities of third party components that we are not responsible for | | | |

0 commit comments

Comments
 (0)