Indexer Reliability and Performance Epic

[artntek]

We have a functional indexer codebase, but there are some reliability and performance aspects that need improvement:

1. Index-failure handling

• After being consumed from RabbitMQ, an indexing task can fail for many reasons, which can usually be categorized into:
  • Retrievable: the task failed due to a temporary issue (e.g. network, pod restart, etc.), and can be re-tried with the likelihood of success
  • Terminal: the task will never succeed, so should not be retried. Instead, the operator should be notified of the failure, the reason, and steps to resolve
• see:
  • How to Acknowledge Failed Index Messages in RabbitMQ  #23
  • Improve error handling for Solr failures in dataone-indexer #240

2. Some indexing tasks take a long time; e.g:

• Indexing EML docs with lots of annotations, requiring the use of OntologyModelService - See Investigate the performance issue in DataONE indexer #34
• Resource maps are not indexed until all referenced objects are indexed. An alternative approach has been proposed - New Approach to index resource map objects #101

3. We should be able to assign priorities to Index tasks - see Priority Indexer Queues #103 (e.g. big re-indexes should be low priority background jobs, so as not to disrupt people submitting new data packages or edits)
4. Would be good to have a periodic auditing task that determines if we have objects that are missing from solr and reindexes them.

Already Done:

• Metacat resubmits index tasks that were not successfully submitted to RabbitMQ (Metacat Issue #1603)

[taojing2002]

We already have the ticket!
Matthew and I had a short talk this afternoon. We believe even we carefully design the resubmitting mechanism, it still can't be fully reliable. We must have an auditing mechanism at the end. So maybe the first step to have a auditing mechanism. The auditing mechanism may have two index tasks:

1. First one is to check very short of time window and to run very frequently. It is used for handling the scenarios needing a quick response, such as the client's editing.
2. Second one is to check longer time window and to run much less frequently. It is used to handling some historical objects without index document.

[artntek]

See Metacat issue: Create an Indexer Auditing Mechanism To Run in K8s