-
Notifications
You must be signed in to change notification settings - Fork 0
EventSource scaling config for gapUpdate #11
Description
During some recent testing of gapConfig with many new collections (10+), I observed some failures in gapUpdate due to timeout. I believe this happens because gapUpdate locks the collection while calculating the updated gaps. This lock exists to prevent a data race when concurrent updates are made on the same rows. This solution is not perfect, however, as it can result in effective deadlock when many concurrent gapUpdate executions wait for the lock on the same collection. This problem is greatly exaggerated when many new collections are initially processed. If the same load is applied from granules in existing collections, there is much less contention. Why this discrepancy in performance exist between fresh, newly added collections and existing ones is not clear to me, perhaps creating initial gaps created on an empty gap table is more expensive than when there are existing gaps. To mitigate this, the concurrency of gapUpdate are limited by the maximum_concurrency parameter on the EventSource trigger. I intended to reduce the scaling limit from 16 to 8, which eliminated the timeout issue I observed, but this change was never merged.