Unify how document re-ingestion behaves for all document types

[dejanb]

Currently there's a mismatch in how different document types will handle re-ingestion (SBOMs and CSAFs will not be ingested again, while OSVs and CVEs will).

With #1237 the feature to prevent duplicate SBOMs nodes ingestion was introduced. Although support for advisories was added the feature was never fully implemented in this way.

CSAF ingestor does the check in its own way by querying the database for the existing record.

At the moment we are interested in re-ingesting CVE files, which works and does not create any unwanted side-effects.

It would be good to unify how this works for all document types. In the normal usage re-ingestion should be disabled to prevent errors, but there has to be a parameter that would allow it to happen (we would need to reingest documents to add new data that we are not handling at the moment, or fix errors in the ingestion process). This has to be propagated to importers as well.

[ctron]

CSAF ingestor does the check in its own way by querying the database for the existing record.

I think is it how it works with SBOMs and some other less used types too. I also think that it's not completely true:

https://github.com/trustification/trustify/blob/216d9d443ea5b71b0e374aa94b59b1f139487d77/modules/ingestor/src/graph/advisory/mod.rs#L136-L144

This is basically an early return, skipping work if the entry existed already.

I do agree that we should unify this however. We should have a clear strategy of what makes a document unique, what happens when a unique/duplicate ID gets uploaded, and how we handle re-processing of documents.

The problem is that original sources might no longer have those documents. Assuming a CI workflow pushing documents to trustify, those documents would never be stored elsewhere.

In cases where we need to re-process documents, we should:

• Similar to migrations, record that we need to re-process
• Continue re-processing (from the S3/filesystem storage)
• Track the status, as this might take some time
• Ensure that the system can handle requests while re-processing (maybe returning incorrect data during that time)

[dejanb]

Yes, the early return works for the advisory entity, but in the whole process the outcome is not handled and the rest of the ingestion process continues (and the vulnerability is updated beforehand)

https://github.com/trustification/trustify/blob/216d9d443ea5b71b0e374aa94b59b1f139487d77/modules/ingestor/src/service/advisory/cve/loader.rs#L89

What I was referfing to CSAF is this logic that does this job on the document level

https://github.com/trustification/trustify/blob/216d9d443ea5b71b0e374aa94b59b1f139487d77/modules/ingestor/src/service/advisory/csaf/loader.rs#L100

I agree with the rest. Perhaps updating document data from the internal storage is the best way forward.

[ctron]

I think we should create an ADR for this

[ctron]

ADR: #1913