Skip to content

Performance optimisations for the computation of the evaluation metrics and the data loading #192

@nikos-livathinos

Description

@nikos-livathinos

We want to speedup the evaluation runtime:

  1. Parallelize the computation of evaluation metrics. This can be easily done for those metrics where each sample can be computed independently from each other.
  2. Redesign the data loading to pre-fetch and cache the ground truth and prediction documents from the disk. The loaded DoclingDocument objects can be then shared across all evaluators, instead of having each evaluator reloading the data. This is particularly useful when we run on slow storage, where I/O time dominates.
  3. Refactor the API/CLI to allow the computation of multiple modalities in one go. This is essential to take advantage of the point 2 as it maximizes the usage of the pre-loaded data.

For the computation of the evaluation metrics, we want:

  • Parallelize TableEvaluator to compute TEDS scores of each table independently.
  • Parallelize PixelLayoutEvaluator to compute confusion matrices and metrics for each page independently.
  • Parallelize MarkDownTextEvaluator to compute text metrics for each page independently.
    ...

For the optimizations in the data loading, we want:

  • Load and cache ground truth DoclingDocument objects from a parquet dataset.
  • Load and cache prediction DoclingDocument objects from a parquet dataset.
  • Load and cache prediction DoclingDocument objects from externally provided files (dt, json, yaml, etc.)
  • Refactor the evaluate() method and the CLI to receive multiple modalities.

Metadata

Metadata

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions