Feature/1584 batched job

[i-just]

Description

Feeds are now processed as batched jobs.

Notes:

• Pagination is still supported, as it was before. If you have a feed that has 3 pages, each page has, e.g. 200 primary elements to be processed, then each page will take two batches.
• services/Process.php->processFeed() - the last parameter ($feedData) is no longer optional
• services/Process.php->afterProcessFeed() - now accepts the 4th param ($startTime); it’s needed for the time calculations to work correctly for the paginated view; if omitted, the time calculation will still work, but as before - timing only the last page
• services/Process.php->beforeProcessFeed() is now split into 2 methods, beforeProcessFeed(), which runs before each page (if the feed is not paginated, then there’s only one page), and getFeedSettings(), which runs before each batch
• Processing X elements finished in <number>s => it looks like this was a bit buggy before for paginated feeds; it reported the total number of processed items but only really measured the time of the past page (fixed now)
• it looks like the logs were always a bit wonky for the paginated feeds; they were always nesting one of the processed elements under Finished preparing for feed processing. because the $stepKey was not cleared as expected (fixed now)

Related issues

#1584

[jamesmacwhite]

This is literally the best thing ever for the Feed Me, can't wait for this to be merged!

[janhenckens]

Hey @i-just, I was trying to install this branch to see how I could integrate this wit Scout but I see it requires Craft 4 here https://github.com/craftcms/feed-me/blob/feature/1584-batched-job/composer.json#L33 but the PR is against the 5.x branch.

Will this be merged for both or it something wrong with that? Thanks!

[i-just]

Hi @janhenckens, if approved, this will be merged into Feed Me 5.x (for Craft CMS 4) and Feed Me 6.x (for Craft CMS 5).

[janhenckens]

Hi @janhenckens, if approved, this will be merged into Feed Me 5.x (for Craft CMS 4) and Feed Me 6.x (for Craft CMS 5).

Oh, my bad - I got the plugin & craft versions confused.

[jamesmacwhite]

Hope it does get approved. Queues with low TTR or data that isn't able to be paginated will benefit greatly with batched job support natively from Feed Me.

[plcdnl]

Sorry to disturb, but it would be really great to have this pull request approved! This feature would be super helpful 🤞

[angrybrad]

This is a draft PR and isn’t finalized, yet.

[angrybrad]

Just released Feed Me 5.11.0 and 6.8.0 with this.

[jamesmacwhite]

Amazing!