Skip to content

EventSource scaling config for gapUpdate #11

@adamconrad7

Description

@adamconrad7

During some recent testing of gapConfig with many new collections (10+), I observed some failures in gapUpdate due to timeout. I believe this happens because gapUpdate locks the collection while calculating the updated gaps. This lock exists to prevent a data race when concurrent updates are made on the same rows. This solution is not perfect, however, as it can result in effective deadlock when many concurrent gapUpdate executions wait for the lock on the same collection. This problem is greatly exaggerated when many new collections are initially processed. If the same load is applied from granules in existing collections, there is much less contention. Why this discrepancy in performance exist between fresh, newly added collections and existing ones is not clear to me, perhaps creating initial gaps created on an empty gap table is more expensive than when there are existing gaps. To mitigate this, the concurrency of gapUpdate are limited by the maximum_concurrency parameter on the EventSource trigger. I intended to reduce the scaling limit from 16 to 8, which eliminated the timeout issue I observed, but this change was never merged.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions