Skip to content

Ingest pipeline breaks order after last file #63

@juagargi

Description

@juagargi

The ingest command ingests files into the DB.

The processing pipeline is:

  1. Open file in Processor
  2. Parse file, send each certificate to CertificateProcessor
  3. In multiple routines, create batches of certificates, in CertificateProcessor
  4. For each batch, in an independent routine, update the DB.
  5. The CertificateProcessor waits until all its batches are updated.
  6. The Processor waits until the CertificateProcessor is done
  7. The Processor calls OnBundleFinished if the # of certs is a bundle, or at end

The bug appears:

  • When there is no bundle size defined: the call to OnBundleFinished is done before the CertProcessor is done.
  • When there is a defined bundle size: the CertProcessor is still running.

This bug makes the last batches not to be ready for the next steps, done on OnBundleFinished,
thus creating an incomplete view of the certificate landscape represented by the files.

We need to:

  • Just move the code to solve the first bug.
  • On each bundle:
    • Close the parsedCertCh and wait for CertProcessor to finish.
    • Create a new parsedCertCh and a new CertProcessor

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions