PERF: Move per-post language detection job into batches using redis #177

nattsw · 2024-11-20T05:08:52Z

Currently, upon post serialization, we spawn a
new job to detect the language of the post. This
could potentially flood sidekiq on larger sites.

This PR uses redis to keep track of posts that
need language detection, then schedules them in
a 5-minute-ly job, with a maximum of 1000 posts
at a time.

Note for the reviewer: This plugin doesn't seem to be using autoloading as it should yet, but that will come in a separate PR.

Currently, upon post serialization, we spawn a new job to detect the language of the post. This could potentially flood sidekiq on larger sites. This PR uses redis to keep track of posts that need language detection, then schedules them in a 5-minute-ly job, with a maximum of 1000 posts at a time.

plugin.rb

spec/serializers/post_serializer_spec.rb

spec/jobs/detect_posts_translation_spec.rb

plugin.rb

Drenmi · 2024-11-20T06:42:13Z

What are the consequences if we build up a bit of a backlog in this queue? Just need to wait a bit longer for the translations to happen?

nattsw · 2024-11-20T07:25:03Z

@Drenmi for now I believe waiting is fine for language to be detected. Note that this PR involves language detection, not the translation itself.

~~The current behaviour also doesn't instantly update the post, since it is an asynchronous job triggered during serialization which adds custom fields.~~ Actually I lied -- adding the post custom field does allow the post to be updated through the magic of message bus. The 🌐 icon appears after the job is executed.

nattsw · 2024-11-20T10:09:36Z

@jjaffeux I've incorporated your suggestions :)

nattsw force-pushed the use-scheduled-batch-job branch from d35ea01 to add68d4 Compare November 20, 2024 05:11