Slow Import speeds

[chunacatsunflower]

I am hoping that someone in the community may have some advice for me to help support a deployment of DefectDojo.

I have installed Defectdojo in an air-gapped environment, and we are using it to collect vulnerability reports run on a variety of software projects, including a large monolithic build, which generates something on the order of 12,000 vulnerabilities in a single semgrep report (mostly medium and low severity). After some tweaking of our services and the timeouts/upload limits, I have been able to upload this report to DefectDojo, however it takes well over an hour to process.

I would like to integrate the defectdojo upload into our build pipelines, however this slow upload speed makes it unreasonable to do so. I have done several steps to improve the performance, specifically, enabling multi-threaded celery event handlers, boosting the memory and cpu allocation for the django and postgres containers on the back end (although we don't have too much headroom to go further), and a few other things over the course of the last several weeks. I'm not sure what else I should look into here.

Has anyone else encountered this kind of performance issue? Does anyone have ideas to improve the speed of processing the upload?

[valentijnscholten]

Do you mean upload or import speed? What size (number of findings) is the report? What scan type? What version of defect dojo? What type of deployment? What resources allocated to the containers?

[chunacatsunflower]

A single Semgrep SARIF report that contains over 12000 findings takes more than one hour to (edit:) import.

We are running Open Source defectodojo 2.49.2 in a kubernetes environment; django, celery-worker and postgres nodes are all allocated 2 CPU and 4 GB memory; the others are at default resources allocation defined by the provided helm configuration.

[valentijnscholten]

That sounds slow compared to the 14000 finding jfrog sample file that imports in 8 minutes on my local laptop running docker compose: https://github.com/DefectDojo/django-DefectDojo/blob/master/unittests/scans/jfrog_xray_unified/very_many_vulns.json.
Could you try that as a comparison? In my experience Defect Dojo is quite database heavy as Django performs lots of (small) queries. So Database performance and latency is crucial. Could there be any limit on the postgres side that hinder performance?

[chunacatsunflower]

Thanks for that info, it is very helpful to have a benchmark.

I have used that same file as a test on our kubenetes based deployment using the provided helm charts, and it initially took approximately 30 minutes to complete. I've been tweaking some postgres parameters this week (batching transactions, commit delays, etc.), and have gotten it down to around 13 minutes, so it definitely seems like a database bottleneck, especially if it can run in 8 minutes on a local deployment.

I'll keep looking into the deployment; maybe I can spot some differences in config between docker-compose and the helm charts.

[valentijnscholten]

Additional info: The 8 minute is for the foreground part in the uwsgi container that creates the findings. There will be some additional post processing in the celery workers that runs beyond the mentioned 8 minutes.

[mtesauro]

BTW, this is a very handy tool when tuning Postgres:

https://pgtune.leopard.in.ua/

HTH