Skip to content

Feat: Batch transform support (BatchTransformerInterface)#60

Merged
bpolaszek merged 3 commits intomasterfrom
feat/batch-transform
Feb 11, 2026
Merged

Feat: Batch transform support (BatchTransformerInterface)#60
bpolaszek merged 3 commits intomasterfrom
feat/batch-transform

Conversation

@bpolaszek
Copy link
Owner

Summary

  • Add BatchTransformerInterface — a new interface (separate from TransformerInterface) for transforming items in batches instead of one-by-one
  • Add CallableBatchTransformer — callable-based implementation
  • Add batchSize option to EtlConfiguration to control chunk size
  • Modify IterableProcessor to chunk extracted items via iterable_chunk() when a batch transformer is detected
  • Add processItemBatch() to EtlExecutor for batch processing orchestration
  • Move bentools/iterable-functions from require-dev to require

Use case

Transformers that make HTTP calls can now send N requests concurrently instead of waiting for each response sequentially:

$executor = (new EtlExecutor())
    ->transformWith(new CallableBatchTransformer(
        function (array $items, EtlState $state): array {
            return $httpClient->sendConcurrent(
                array_map(fn ($item) => new Request('GET', $item['url']), $items)
            );
        }
    ))
    ->withOptions(new EtlConfiguration(batchSize: 10))
    ->process($source);

Design decisions

  • BatchTransformerInterface does not extend TransformerInterface (PHP prevents parameter type narrowing mixedarray)
  • Batch mode activates when transformer implements BatchTransformerInterface (regardless of batchSize)
  • nextTick callbacks are consumed between batches, not between items within a batch
  • SkipRequest during extraction removes the item from the batch; StopRequest processes collected items then stops
  • When transformer is not BatchTransformerInterface, batchSize is ignored (zero behavioral change)

Test plan

  • Happy path: N items, batch size M, all transformed correctly
  • Partial last batch (7 items, batchSize=3 → batches of 3, 3, 1)
  • Empty input
  • SkipRequest during ExtractEvent → item removed from batch
  • StopRequest during ExtractEvent → processes collected items, then stops
  • SkipRequest during TransformEvent → result skipped
  • Fan-out in batch mode (transformer yields more results than inputs)
  • batchSize=1 with BatchTransformerInterface → works with chunks of 1
  • Non-batch transformer with batchSize > 1 → gracefully ignored
  • Integration with flushFrequency
  • ExtractEvent fires per item, TransformEvent fires per result
  • batchSize larger than total items → single batch
  • EtlConfiguration validation (batchSize < 1 throws)
  • All 147 tests pass, PHPStan clean, 99.3% coverage

🤖 Generated with Claude Code

@codecov
Copy link

codecov bot commented Feb 10, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100.00%. Comparing base (26fdf50) to head (c92c636).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files
@@             Coverage Diff             @@
##              master       #60   +/-   ##
===========================================
  Coverage     100.00%   100.00%           
- Complexity       352       372   +20     
===========================================
  Files             61        62    +1     
  Lines            916       966   +50     
===========================================
+ Hits             916       966   +50     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Base automatically changed from feat/generator-fan-out to master February 11, 2026 13:16
bpolaszek and others added 3 commits February 11, 2026 14:16
Allow transforming multiple items at once instead of one-by-one,
enabling use cases like concurrent HTTP requests.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@bpolaszek bpolaszek force-pushed the feat/batch-transform branch from f5662a5 to c92c636 Compare February 11, 2026 13:16
@bpolaszek bpolaszek merged commit 9386ed5 into master Feb 11, 2026
6 checks passed
@bpolaszek bpolaszek deleted the feat/batch-transform branch February 11, 2026 13:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant