[BUG] Tackling increased memory usage when loading product models in a loop #5091
Description
Is there an existing issue for this?
- I have searched the existing issues
Current Behavior
When looping though a collection of let's say 60k products in a very simple manner like
if(!empty($product)){
$product->clearInstance();
}
$product = Mage::getModel('catalog/product')->load($productId);
unset($product)You can see that memory keeps stockpiling until you finally go out of memory depending on how many products you have. It seems that garbage collection can't happen because references are still held.
Expected Behavior
EIther memory increases in each iteration and garbage collection clears it every now and then or the memory doesn't increase by each iteration
Steps To Reproduce
I am providing a simple script that can help simulate this but make sure you have a huge catalog and php-memprof installed to run it https://github.com/BitOne/php-meminfo
Here's the full script
<?php
/*
* Copyright (c) 2025. IOWEB TECHNOLOGIES
*/
error_reporting(E_ALL);
ini_set('display_errors', '1');
set_time_limit(0);
if (PHP_SAPI !== 'cli') {
echo 'This script must be executed via the CLI.' . PHP_EOL;
exit(1);
}
require_once dirname(__DIR__) . '/app/Mage.php';
if (class_exists('Varien_Profiler')) {
Varien_Profiler::disable();
}
umask(0);
Mage::app('admin')->setUseSessionInUrl(false);
Mage::app()->setCurrentStore(Mage_Core_Model_App::ADMIN_STORE_ID);
$productResource = Mage::getResourceModel('catalog/product_collection');
$productIds = $productResource->getAllIds();
if (!$productIds) {
echo 'No product IDs found.' . PHP_EOL;
exit(0);
}
$cliArgs = $argv;
array_shift($cliArgs);
$total = count($productIds);
$maxIterations = $total;
$meminfoEnabled = function_exists('meminfo_dump');
$shouldFlushSingletons = true;
foreach ($cliArgs as $arg) {
if ($arg === '--no-memprof') {
$meminfoEnabled = false;
continue;
}
if ($arg === '--keep-singletons') {
$shouldFlushSingletons = false;
continue;
}
if (is_numeric($arg)) {
$maxIterations = max(1, (int)$arg);
}
}
$meminfoDir = Mage::getBaseDir('var') . DS . 'meminfo';
if ($meminfoEnabled && !is_dir($meminfoDir) && !mkdir($meminfoDir, 0777, true) && !is_dir($meminfoDir)) {
echo sprintf('Unable to create meminfo output directory: %s%s', $meminfoDir, PHP_EOL);
$meminfoEnabled = false;
}
/**
* Format bytes in a readable string (e.g. 12.45 MB).
*/
$formatBytes = static function ($bytes) {
$units = ['B', 'KB', 'MB', 'GB', 'TB'];
$bytes = max((int)$bytes, 0);
$precision = 2;
$pow = $bytes > 0 ? floor(log($bytes, 1024)) : 0;
$pow = min($pow, count($units) - 1);
$bytes /= pow(1024, $pow);
return round($bytes, $precision) . ' ' . $units[$pow];
};
echo sprintf('Found %d product IDs. Processing up to %d iteration(s).%s', $total, $maxIterations, PHP_EOL);
$previousRealUsage = memory_get_usage(false);
$previousAllocatedUsage = memory_get_usage(true);
/**
* Flush per-request singleton caches that would otherwise grow forever in a long CLI run.
*/
$flushSingletonCaches = static function () {
try {
$observer = Mage::getSingleton('cataloginventory/observer');
} catch (Exception $exception) {
return;
}
if (!$observer) {
return;
}
$properties = [
'_stockItemsArray',
'_checkedProductsQty',
'_checkedQuoteItems',
'_itemsForReindex',
];
$reflection = new ReflectionObject($observer);
foreach ($properties as $propertyName) {
if (!$reflection->hasProperty($propertyName)) {
continue;
}
$property = $reflection->getProperty($propertyName);
$property->setAccessible(true);
$property->setValue($observer, []);
}
};
foreach ($productIds as $index => $productId) {
if(!empty($product)){
$product->clearInstance();
}
$product = Mage::getModel('catalog/product')->load($productId);
$currentRealUsage = memory_get_usage(false);
$currentAllocatedUsage = memory_get_usage(true);
$peakUsage = memory_get_peak_usage(true);
$deltaReal = $currentRealUsage - $previousRealUsage;
$deltaAllocated = $currentAllocatedUsage - $previousAllocatedUsage;
echo sprintf(
'[%d/%d] Product ID %s | real: %s | realΔ: %s | alloc: %s | allocΔ: %s | peak: %s%s',
$index + 1,
$total,
$productId,
$formatBytes($currentRealUsage),
$formatBytes($deltaReal),
$formatBytes($currentAllocatedUsage),
$formatBytes($deltaAllocated),
$formatBytes($peakUsage),
PHP_EOL
);
$previousRealUsage = $currentRealUsage;
$previousAllocatedUsage = $currentAllocatedUsage;
$product = null;
if ($shouldFlushSingletons) {
$flushSingletonCaches();
}
if ($meminfoEnabled) {
$dumpFile = sprintf(
'%s/meminfo-product-%05d-%s.json',
$meminfoDir,
$index + 1,
$productId
);
if ($handle = fopen($dumpFile, 'w')) {
meminfo_dump($handle);
fclose($handle);
echo sprintf(' ↳ meminfo dump created: %s%s', $dumpFile, PHP_EOL);
} else {
echo sprintf(' ↳ failed to open meminfo dump file: %s%s', $dumpFile, PHP_EOL);
}
}
if ($index + 1 >= $maxIterations) {
echo sprintf('Reached iteration limit (%d).%s', $maxIterations, PHP_EOL);
break;
}
}You can just drop it in the main directory of your installation and execute it.
The script bootstraps Mage, loads every product ID sequentially, prints memory stats (real vs alloc vs peak), optionally dumps php-meminfo snapshots per iteration, and can flush Magento singleton caches between loads.
First run it with
php -d extension=meminfo.so ioweb/test-memory-increase-per-iteration.php --no-memprof --keep-singletons
You will notice the memory is increasing steadily like this
WIth memprof you can track that the Mage_CatalogInventory_Model_Observer is holding references to objects in an internal cache so no matter if you clearInstance or unset the variable, the object is never freed from memory.
In order to test this cause I added a function to flush these properties
/**
* Flush per-request singleton caches that would otherwise grow forever in a long CLI run.
*/
$flushSingletonCaches = static function () {
try {
$observer = Mage::getSingleton('cataloginventory/observer');
} catch (Exception $exception) {
return;
}
if (!$observer) {
return;
}
$properties = [
'_stockItemsArray',
'_checkedProductsQty',
'_checkedQuoteItems',
'_itemsForReindex',
];
$reflection = new ReflectionObject($observer);
foreach ($properties as $propertyName) {
if (!$reflection->hasProperty($propertyName)) {
continue;
}
$property = $reflection->getProperty($propertyName);
$property->setAccessible(true);
$property->setValue($observer, []);
}
};Then integrated into the php script and re-run it without the flag --keep-singletons
php -d extension=meminfo.so ioweb/test-memory-increase-per-iteration.php --no-memprof
You will notice at regular intervals a small pause to do garbage collection and the memory instantly drops
While not perfect this is to illustrate that this observer is causing a memory leak when loading products.
Whether by design or not, if it's holding references to objects imho it should have a way to destroy those references , via it's own clear method. Otherwise methods like clearInstance should take care of this to prevent the leak.
There are other causes as well, the Varien_Profiler is always on recording behind the scenes unless explicitly turned off and some other caches that keep increasing the size eventually but the difference is negligible compared to this observer.
Within 10k products it can reach up to 1.2 GB of memory
With clearing the observer I've reached over 60k products with <350mb of memory.
What do you think about it? What would be a good approach to handle this?
I'm pretty sure all collection iterations suffer from similar causes and it's hard to track all but products are very common when you try to export feeds.
Environment
- OpenMage:All versions
- php:7.4 to 8.2Anything else?
No response