Merge pull request #3344 from DefectDojo/release/1.10.0

Release: Merge release into master from: release/1.10.0

Author: Maffooch

.dockerignore

@@ -2,3 +2,4 @@
 .gitignore
 *.md
 .env*
+**/local_settings.py

.gitignore

@@ -56,7 +56,7 @@ coverage.xml
 # Django stuff:
 *.log
 *.pot
-**/settings.py
+**/local_settings.py
 
 # Vim swapfiles
 *.swp
@@ -119,6 +119,13 @@ Pipfile*
 docker/certs/*
 !docker/certs/readme.txt
 
+#ignore locally added extra fixtures
+docker/extra_fixtures/*
+!docker/extra_fixtures/readme.txt
+docker/extra_settings/*
+!docker/extra_settings/README.md
+
+
 # Helm dependencies
 helm/defectdojo/charts
 

CONTRIBUTING.md

@@ -4,7 +4,7 @@
 
 Before submitting, please ensure that you are using the latests code by performing a `git pull`.
 
-Please include your operating system, your operating system version number (16.04, 10.6, etc), and the dojo install you are using (setup.bash, docker, etc).
+Please include your operating system name, your operating system version number (16.04, 18.6, etc), and the dojo install type you are using (setup.bash, docker, etc).
 
 Bugs that do not have this information will be closed.
 
@@ -14,17 +14,23 @@ Here are a few things to keep in mind when making changes to DefectDojo.
 
 ## Modifying DefectDojo and Testing
 
-Please use [these test scripts](./tests) to test your changes. These are the exact scripts we run in our [Travis Build](https://travis-ci.org/OWASP/django-DefectDojo).
+Please use [these test scripts](./tests) to test your changes. These are the scripts we run in our [integration tests](DOCKER.md#run-the-tests-with-docker).
 
-For changes that require additional settings, settings.dist.py is the file you want to change. Settings.py is created by setup.bash from settings.dist.py
+For changes that require additional settings, you can now use local_settings.py file. See the logging section below for more information.
 
 ## Python3 version
-For compatibility reasons, the code in dev branch should be python3.5 compliant.
+For compatibility reasons, the code in dev branch should be python3.6 compliant.
 
 ## Logging
-Logging is configured in `settings.dist.py`.
+Logging is configured in `settings.dist.py` and can be tuned using a `local_settings.py`, see [template for local_settings.py](dojo/settings/template-local_settings)
+Specific logger can be added. For example to activate logs related to the deduplication, change the level from DEBUG to INFO in `local_settings.py`:
 
-Specific logger can be added. For example to activate logs related to the deduplication, change the level from DEBUG to INFO in:
+
+```
+LOGGING['loggers']['dojo.specific-loggers.deduplication']['level'] = 'DEBUG'
+```
+
+Or you can modify `settings.dist.py` directly, but this adds the risk of having conflicts when `settings.dist.py` gets updated upstream. 
 
 ```
           'dojo.specific-loggers.deduplication': {
@@ -34,6 +40,10 @@ Specific logger can be added. For example to activate logs related to the dedupl
         }
 ```
 
+## Debug Toolbar
+In the `dojo/settings/template-local_settings.py` you'll find instructions on how to enable the [Django Debug Toolbar](https://github.com/jazzband/django-debug-toolbar).
+This toolbar allows you to debug SQL queries, and shows some other interesting information.
+
 ## Submitting Pull Requests
 
 The following are things to consider before submitting a pull request to
@@ -53,7 +63,7 @@ DefectDojo.
 
 0. Pull requests should be submitted to the 'dev' branch.
 
-0. In dev branch, the code should be python 3.5 compliant.
+0. In dev branch, the code should be python 3.6 compliant.
 
 [dojo_settings]: /dojo/settings/settings.dist.py "DefectDojo settings file"
 [setup_py]: /setup.py "Python setup script"

DOCKER.md

@@ -49,6 +49,7 @@ or
 docker-compose build nginx
 ```
 
+> **_NOTE:_**  It's possible to add extra fixtures in folder "/docker/extra_fixtures".
 
 ## Run with Docker compose in release mode
 To run the application based on previously built image (or based on dockerhub images if none was locally built), run: 
@@ -68,7 +69,6 @@ In this setup, you need to rebuild django and/or nginx images after each code ch
 For development, use: 
 
 ```zsh
-cp dojo/settings/settings.dist.py dojo/settings/settings.py
 docker/setEnv.sh dev
 docker-compose build
 docker-compose up
@@ -94,9 +94,7 @@ To update changes in static resources, served by nginx, just refresh the browser
 
 *Notes about volume permissions*
 
-*The manual copy of settings.py is sometimes required once after cloning the repository, on linux hosts when the host files cannot be modified from within the django container. In that case that copy in entrypoint-uwsgi-dev.sh fails.* 
-
-*Another way to fix this is changing `USER 1001` in Dockerfile.django to match your user uid and then rebuild the images. Get your user id with* 
+*If you run into permission issues with the mounted volumes, a way to fix this is changing `USER 1001` in Dockerfile.django to match your user uid and then rebuild the images. Get your user id with* 
 
 ```
 id -u
@@ -109,7 +107,6 @@ If you want to be able to step in your code, you can activate ptvsd.Server.
 You can launch your local dev instance of DefectDojo as
 
 ```zsh
-cp dojo/settings/settings.dist.py dojo/settings/settings.py
 docker/setEnv.sh ptvsd
 docker-compose up
 ```
@@ -264,7 +261,6 @@ The integration-tests are under `tests`
 This will run all unit-tests and leave the uwsgi container up: 
 
 ```
-cp dojo/settings/settings.dist.py dojo/settings/settings.py
 docker/setEnv.sh unit_tests
 docker-compose up
 ```
@@ -295,7 +291,6 @@ python manage.py test dojo.unittests.test_dependency_check_parser.TestDependency
 This will run all integration-tests and leave the containers up: 
 
 ```
-cp dojo/settings/settings.dist.py dojo/settings/settings.py
 docker/setEnv.sh integration_tests
 docker-compose up
 ```

Dockerfile.django

@@ -17,6 +17,7 @@ RUN \
     postgresql-client \
     xmlsec1 \
     git \
+    uuid-runtime \
     && \
   apt-get clean && \
   rm -rf /var/lib/apt/lists && \
@@ -43,8 +44,10 @@ RUN \
     libmariadb3 \
     xmlsec1 \
     git \
+    uuid-runtime \    
     # only required for the dbshell (used by the initializer job)
     postgresql-client \
+    curl \
     && \
   apt-get clean && \
   rm -rf /var/lib/apt/lists && \
@@ -71,15 +74,21 @@ COPY \
   /
 COPY wsgi.py manage.py tests/unit-tests.sh ./
 COPY dojo/ ./dojo/
-# Legacy installs need the modified settings.py, do not remove!
-RUN \
-  cp dojo/settings/settings.dist.py dojo/settings/settings.py
+
+# Add extra fixtures to docker image which are loaded by the initializer
+COPY docker/extra_fixtures/* /app/dojo/fixtures/
+
 COPY tests/ ./tests/
 RUN \
-  rm -f /readme.txt && \
+  # Remove placeholder copied from docker/certs
+  rm -f /readme.txt && \ 
+  # Remove placeholder copied from docker/extra_fixtures
+  rm -f dojo/fixtures/readme.txt && \
   mkdir -p dojo/migrations && \
   chmod g=u dojo/migrations && \
   true
+# Fallback for safety parser, if installation does't allow internet connectivity
+RUN curl -sS -o dojo/tools/safety/insecure_full.json https://raw.githubusercontent.com/pyupio/safety-db/master/data/insecure_full.json
 USER root
 RUN \
     adduser --system --no-create-home --disabled-password --gecos '' \

Dockerfile.nginx

@@ -55,16 +55,15 @@ RUN pip3 install \
 COPY components/ ./components/
 COPY manage.py ./
 COPY dojo/ ./dojo/
-RUN \
-  cp dojo/settings/settings.dist.py dojo/settings/settings.py
+
 RUN \
   cd components && \
   yarn && \
   cd .. && \
-  python3 manage.py collectstatic --noinput && \
+  env DD_SECRET_KEY='.' python3 manage.py collectstatic --noinput && \
   true
 
-FROM nginx:1.19.3-alpine@sha256:a3c6118edc80de4a5aaf2711b7742c25d4d2da54325bae465205cb386afa79ee
+FROM nginx:1.19.4-alpine@sha256:f9ddfb3fd9590a3b6ba095939b7a5aee110a6fb397922e2684d6e189e78329c9
 ARG uid=1001
 ARG appuser=defectdojo
 COPY --from=collectstatic /app/static/ /usr/share/nginx/html/static/

KUBERNETES.md

@@ -32,7 +32,8 @@ helm repo update
 
 Helm >= v3
 ```zsh
-helm repo add stable https://kubernetes-charts.storage.googleapis.com/
+helm repo add stable https://charts.helm.sh/stable
+helm repo add bitnami https://charts.bitnami.com/bitnami
 helm repo update
 ```
 Then pull the dependent charts:
@@ -42,20 +43,20 @@ helm dependency update ./helm/defectdojo
 
 Now, install the helm chart into minikube.
 
-If you have setup an ingress controller: 
+If you have setup an ingress controller:
 ```zsh
 DJANGO_INGRESS_ENABLED=true
 ```
-else: 
+else:
 ```zsh
 DJANGO_INGRESS_ENABLED=false
 ```
 
-If you have configured TLS: 
+If you have configured TLS:
 ```zsh
 DJANGO_INGRESS_ACTIVATE_TLS=true
 ```
-else: 
+else:
 ```zsh
 DJANGO_INGRESS_ACTIVATE_TLS=false
 ```
@@ -91,9 +92,9 @@ helm install \
   --set createMysqlSecret=true \
   --set createPostgresqlSecret=true
 ```
-Note that you need only one of: 
+Note that you need only one of:
 - postgresql or mysql
-- rabbitmq or redis 
+- rabbitmq or redis
 
 It usually takes up to a minute for the services to startup and the
 status of the containers can be viewed by starting up ```minikube dashboard```.
@@ -141,7 +142,7 @@ Use the same commands as before but add:
 ```
 
 ### Installing from a private registry
-If you have stored your images in a private registry, you can install defectdojo chart with (helm 3). 
+If you have stored your images in a private registry, you can install defectdojo chart with (helm 3).
 
 - First create a secret named "defectdojoregistrykey" based on the credentials that can pull from the registry: see https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
 - Then install the chart with the same commands as before but adding:
@@ -167,7 +168,7 @@ docker build --build-arg http_proxy=http://myproxy.com:8080 --build-arg https_pr
 ### Upgrade the chart
 If you want to change kubernetes configuration of use an updated docker image (evolution of defectDojo code), upgrade the application:
 ```
-kubectl delete job defectdojo-initializer 
+kubectl delete job defectdojo-initializer
 helm upgrade  defectdojo ./helm/defectdojo/ \
    --set django.ingress.enabled=${DJANGO_INGRESS_ENABLED} \
    --set django.ingress.activateTLS=${DJANGO_INGRESS_ACTIVATE_TLS}
@@ -317,7 +318,7 @@ kubectl logs $(kubectl get pod --selector=defectdojo.org/component=${POD} \
 # Open a shell in a specific pod
 kubectl exec -it $(kubectl get pod --selector=defectdojo.org/component=${POD} \
   -o jsonpath="{.items[0].metadata.name}") -- /bin/bash
-# Or: 
+# Or:
 kubectl exec defectdojo-django-<xxx-xxx> -c uwsgi -it /bin/sh
 
 # Open a Python shell in a specific pod
@@ -337,8 +338,8 @@ Helm >= v3
 helm uninstall defectdojo
 ```
 
-To remove persistent objects not removed by uninstall (this will remove any database):  
+To remove persistent objects not removed by uninstall (this will remove any database):
 ```
 kubectl delete secrets defectdojo defectdojo-redis-specific defectdojo-rabbitmq-specific defectdojo-postgresql-specific defectdojo-mysql-specific
 kubectl delete pvc data-defectdojo-rabbitmq-0 data-defectdojo-postgresql-0
-```
+```

MAINTAINERS.md

@@ -31,6 +31,9 @@ Furthermore, maintainers that have not had any activities (commits, PR, PR revie
 
 ### Maintainers
 * Anthony Pipia (@apipia)
-* Saurabh (@dr3dd589)
 * Jannik Jürgens (@alles-klar)
 * Pascal Trovatelli (@ptrovatelli)
+
+### Reviewers
+* Saurabh (@dr3dd589)
+* Romain Aviolat (@xens)

SPONSORING.md

@@ -1,4 +1,4 @@
-On April 5th, 2018, OWASP clarified their sponsorship requirements to note that time, software, or any other quantifiable contribution can be counted towards the $1000 threshold outlined by the [OWASP Global Policy](https://www.owasp.org/index.php/Project_Sponsorship_Operational_Guidelines).
+On April 5th, 2018, OWASP clarified their sponsorship requirements to note that time, software, or any other quantifiable contribution can be counted towards the $1000 threshold outlined by the [OWASP Global Policy](https://owasp.org/www-policy/operational/project-sponsorship).
 
 Below is our sponsorship guidelines to provide further clarification specific to our project for non-monetary contributions:
 

components/package.json

@@ -1,6 +1,6 @@
 {
   "name": "DefectDojo",
-  "version": "1.9.3",
+  "version": "1.10.0",
   "dependencies": {
     "JUMFlot": "jumjum123/JUMFlot#*",
     "bootstrap": "^3.4.0",
@@ -19,7 +19,7 @@
     "datatables.net-dt": "^1.10.22",
     "drmonty-datatables-plugins": "^1.0.0",
     "drmonty-datatables-responsive": "^1.0.0",
-    "easymde": "^2.12.1",
+    "easymde": "^2.13.0",
     "flot": "flot/flot#~0.8.3",
     "flot-axis": "markrcote/flot-axislabels#*",
     "font-awesome": "^4.0.0",