Supporting batch processing

[janko]

In our Rails app I would like to sometimes use Sequel instead of Active Record for running data migrations. The one feature I'm missing in Sequel is Active Record's #in_batches, which returns datasets representing a batch of records, which makes it easier to perform bulk operations without actually loading records into memory.

One classic example is backfilling a newly added column:

Record.in_batches do |batch|
  batch.update_all(new_column: "some_value")
end

From what I could tell, the closest Sequel offers is #paged_each, which is the equivalent of #find_each. It could technically be used to implement the above:

Record.order(:id).select(:id).naked.paged_each(row_per_fetch: BATCH_SIZE).each_slice(BATCH_SIZE) do |batch_data|
  batch_ids = batch_data.map { |data| data[:id] }
  batch = Record.where(id: batch_ids)
  # ...
end

Since this isn't that efficient, Active Record 7.1 will optimize this using an approach that seems similar to Sequel's :filter strategy for #paged_each. Would you be open to a PR adding support for batch processing like the one Active Record 7.1 will have?

[jeremyevans]

It looks like this could be implemented using an option to paged_each, to yield each dataset instead of yielding each row? I'd be open to that approach.

[janko]

My mistake, I overlooked the part after the loop 👍

[jeremyevans]

OK, I pushed that change: 802b5de

[jeremyevans]

I added support for a :skip_transaction option to paged_each: 89cdc61

[janko]

Looks great! ❤️ I did some benchmarks, and in my dataset it was ~3x faster than Active Record 7.0 batching, and maybe ~20% faster than upcoming Active Record 7.1 implementation.

Given that it now provides a #paged_datasets method, I'm wondering if something more generic like paged_operations would be a better name of the plugin. In my particular case I would use it for paged inserts, which is neither an update nor delete.

Maybe it would also make sense to use the batching terminology, since it doesn't share common logic with #paged_each. So, we could have #batch_update, #batch_delete, #batches and #batch_size. Just an idea 🙂

[jeremyevans]

I'm fine renaming it to paged_operations. Can you send a pull request for that?

I don't want to change from page to batch, as they describe the same type of operation, and page is Sequel's normal terminology for that. While the plugin does not share an implementation with paged_each, the point is the same, to handle operations on a large dataset using multiple queries that each operate on part of the dataset.