fix: defer pull jobs refresh to prevent timeout with many sites#185
Merged
fix: defer pull jobs refresh to prevent timeout with many sites#185
Conversation
Addresses timeout issues when adding or editing syndication sites in installations with many configured syndication sites. The synchronous refresh_pull_jobs() operation becomes prohibitively expensive at scale, causing requests to timeout during site administration. This change introduces deferred background processing via WordPress cron. When sites or site groups are modified, instead of immediately clearing and rescheduling all pull jobs, the operation is deferred to a one-time cron event scheduled 60 seconds in the future. A transient-based debounce mechanism prevents duplicate scheduling when multiple rapid changes occur. The solution maintains functional correctness whilst eliminating timeout issues by moving the expensive operation out of the request lifecycle. Fixes #86
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When a syn_site post or syn_sitegroup term is saved or deleted, the system previously triggered an immediate synchronous refresh of pull jobs. This approach becomes problematic when many sites are configured, as the refresh operation can take considerable time and lead to timeout errors, making it impossible to add or edit sites.
The root cause is that
refresh_pull_jobs()iterates through all configured sites and performs API calls, which becomes increasingly expensive as the number of sites grows. Running this synchronously within the save/delete request lifecycle means the HTTP request must wait for all these operations to complete.This change defers the refresh operation to a background cron event scheduled 60 seconds in the future. The implementation includes debouncing through a transient to prevent duplicate scheduling when multiple sites are modified in quick succession. The transient expires after 2 minutes, whilst any existing scheduled event is cleared before scheduling a new one to ensure only one refresh runs.
The solution maintains data consistency by ensuring the refresh still occurs, just asynchronously rather than blocking the HTTP request. This approach is particularly effective for bulk operations where multiple sites might be saved in quick succession—rather than triggering multiple expensive refresh operations, they are debounced into a single deferred event.
Unit tests verify the debouncing logic, event scheduling, transient management, and correct integration with the existing site and site group change handlers.
Fixes #86