MAGE-1109: Add Batching Optimizer feature #1797
Conversation
There was a problem hiding this comment.
This is just fantastic work @damcou !! 🙌
I do see some issues if you wouldn't mind taking a look and had a few suggestions.
I also think we should add some language that says something along the lines that these numbers are estimates only and that indexing activity should be monitored after making changes to ensure batches are not exceeding the recommended size of 10 MB.
| const INCREASED_MARGIN = 50; | ||
|
|
||
| const DEFAULT_SAMPLE_SIZE = 20; | ||
| const LARGE_SAMPLE_SIZE = 100; |
There was a problem hiding this comment.
I think we should keep the default but allow the user to explicitly set the sample size to whatever they want.
There was a problem hiding this comment.
This was my first idea, but for some reason, I'm not allowed to do it, apparently it's because the store_id parameter is an array ?
There was a problem hiding this comment.
I'm not sure what was causing that but I just added a POC for a sample size option (did not add for margin). Is this what you were going for or something different?
| * Arbitrary increased margin to ensure not to exceed recommended batch size when catalog is a mix between complex and other product types | ||
| * (i.e. with a lot of record sizes variations) | ||
| */ | ||
| const INCREASED_MARGIN = 50; |
There was a problem hiding this comment.
Just like with sample size I think having an option to set the margin makes sense.
| { | ||
| return [ | ||
| new InputOption( | ||
| self::LARGE_SAMPLE_OPTION, |
There was a problem hiding this comment.
I would recommend replacing this with two options: margin and sample size
| $standardDeviation = $this->getStandardDeviation($sample, $sizeAverage); | ||
| $this->output->writeln('<comment>Standard Deviation</comment> : ' . $standardDeviation); | ||
|
|
||
| $recommendedBatchCountLow = $this->getRecommendedBatchCount($sizeAverage, $standardDeviation, self::INCREASED_MARGIN); |
There was a problem hiding this comment.
If we know the max possible size in the sample we can recommend an absolute floor that would be below the standard deviation.
| */ | ||
| protected function getSizeAverage(array $sizes): int | ||
| { | ||
| if (count($sizes) <= 1) { |
There was a problem hiding this comment.
| if (count($sizes) <= 1) { | |
| if (empty($sizes)) { |
That earlier suggestion was intended for the standard deviation n-1 adjustment which doesn't apply here. Here we can just check for 0.
This PR contains the Batching Optimizer CLI command which include the following process:
store_idargument).simple,virtual,downloadable,giftcard) and "complex products" (configurable,bundle,grouped) in the catalog. (for example, a sample of 20 products from a catalog composed of 40% of simple products and 60% of complex products will have 8 simple products and 12 complex products).