Recommended Approach for CI/CD Import Scans Where Each Engagement Is a Pipeline Run

[eabraham2623]

I use a community instance of DefectDojo and follow this CI/CD integration model:

• Product Type = GitLab Subgroup
• Product = GitLab Project/Repository
• Engagement = Individual Jenkins pipeline run for that GitLab Project/Repository

Format:

https://<gitlab_domain>/<parent_group>/<product_type>/<product>.git

Each engagement is created by the pipeline (triggered by a git push to a branch) and performs:

• Source code vulnerability scans on the repository
• Container image scans on images produced by the project

This results in:

• 2 source code scan reports
• 3 × (number of containers) container scan reports

So, at minimum, 5 tests are imported per engagement.

Because there may be no significant changes between pipeline runs, the findings imported into Dojo can often be identical to those from previous runs.

I have enabled deduplication in the system-level UI settings, but I’m encountering inconsistent results, sometimes identical findings are recognized as duplicates, sometimes not, even when the reports are unchanged.

Note: I’m using import-scan instead of reimport-scan, since each engagement corresponds to a unique pipeline run.

Question:
How can I ensure duplicates are labeled correctly even with deduplication enabled? Is this ultimately a limitation of my chosen engagement model, and would switching to a different engagement structure so I can use reimport-scan be more effective?

[valentijnscholten]

In general this should work, but sometimes the hash_code configuration is not 100% able to match duplicate findings. Do you have some examples of the inconsisten results?

[eabraham2623]

Ok, so here’s one example:

I have two engagements. For simplicity, I imported two Grype scans, one into each engagement.

Both scans have 303 total findings.

We are assuming the Grype reports are nearly identical. The only difference is the container name being scanned, but in practice, it’s the same container when built.

In one engagement, there are 0 active findings, while in the other there are 59 active findings for some reason. The engagement with 59 active findings was imported after the one with 0.

Here’s one of the findings that doesn’t seem to be counted as a duplicate. It is flagged as a duplicate in the first imported report, but not in the one imported afterward. This may be because there were other equivalent imports prior to this example, but if that were the case, I would have expected it to be counted as a duplicate in both reports.

Finding:

**Result message:** A medium vulnerability in rpm package: libyaml, version 0.1.7-5.el8 was found in image <artifactory_domain>/<path_to_image>/container4:hotfix-TESTING-632a3f4 at: /var/lib/rpm/Packages NEWLINE **Rule name:** RpmMatcherExactDirectMatch NEWLINE **Rule short description:** CVE-2024-35328 medium vulnerability for libyaml package NEWLINE **Rule full description:** A flaw was found in the libyaml library. Setting a YAML string with the yaml_parser_set_input_string function to be parsed by the yaml_parser_parse function can cause an infinite loop, resulting in a denial of service in the application linked to the library.

I have attached a csv file that goes more into detail on the finding I have shared.

Let me know if you need more information from me.
example_finding_from_two_engagements.csv

[mtesauro]

I am wondering if you did deduplication at the engagement level instead of the default product-level - would your problem be resolved?

https://docs.defectdojo.com/en/working_with_findings/finding_deduplication/enabling_product_deduplication/

There's a nice diagram here - though it might be relocated in the future:
https://docs.defectdojo.com/en/open_source/archived_docs/usage/features/#deduplication

[eabraham2623]

@mtesauro

Good suggestion. I actually ran into that issue before submitting this question and have since resolved that particular problem. Currently, each time I import a report into Dojo, deduplication_on_engagement is set to false. However, I am still experiencing inconsistent results.

I have enabled "Deduplicate findings" at the System Settings level. The documentation you shared shows it under Pro Settings. Since I am using the community version, I’m not sure if there is a significant difference there, but I wanted to note it in case it matters.