WIP Breaking Change for DataValidation

Fix #797 (marked stale in 2018, but now reopened). Fix #4091. DataValidation is specified in the Xml at the sheet level rather than the cell level. PhpSpreadsheet, however, currently requires a cell and a data validation object for each cell in a range. As a consequence, it may exhaust memory allocating objects that are never really needed. If it reads a spreadsheet which applies DataValidation to an entire column, the reader creates a million cells and a million DataValidation objects. An additional problem is that, when it saves the spreadsheet, it creates a row entry for each of the million rows even if they contain no data.

The code is changed to locate DataValidation for a cell by running the DataValidation list to see if the Sqref for any entry matches the cell in question. This eliminates the need to require a DataValidation entry for each cell; indeed, it eliminates the need to create a cell just because it is subject to DataValidation. Initial tests are very encouraging. The memory-exhausting spreadsheet from 797 required over 1GB of memory. With this change in place, that is reduced to 6MB, and DataValidation works as expected for all cells in the range.

This change is a work in progress. It needs formal tests. I need to see how it works with inserting or removing rows and columns. I need to see how Excel handles conflicting entries. My early tests indicate that it at least doesn't do anything irrational (like claiming corruption) when dealing with conflicts. It appears that, if I have a DataValidation applied to column A, and another applied to cell A5, whichever appears first in the Xml is the ultimate arbiter of what rule applies to A5.

If this change is merged, it will be a breaking range. It will require a new major release (probably PhpSpreadsheet 4.0), and will not happen for at least a couple of months, and probably later.

Author: oleibman

src/PhpSpreadsheet/Reader/Xlsx/DataValidations.php

@@ -3,6 +3,7 @@
 namespace PhpOffice\PhpSpreadsheet\Reader\Xlsx;
 
 use PhpOffice\PhpSpreadsheet\Cell\Coordinate;
+use PhpOffice\PhpSpreadsheet\Cell\DataValidation;
 use PhpOffice\PhpSpreadsheet\Reader\Xlsx;
 use PhpOffice\PhpSpreadsheet\Worksheet\Worksheet;
 use SimpleXMLElement;
@@ -38,28 +39,23 @@ public function load(): void
             $range = strtoupper((string) $dataValidation['sqref']);
             $rangeSet = explode(' ', $range);
             foreach ($rangeSet as $range) {
-                $stRange = $this->worksheet->shrinkRangeToFit($range);
-
-                // Extract all cell references in $range
-                foreach (Coordinate::extractAllCellReferencesInRange($stRange) as $reference) {
-                    // Create validation
-                    $docValidation = $this->worksheet->getCell($reference)->getDataValidation();
-                    $docValidation->setType((string) $dataValidation['type']);
-                    $docValidation->setErrorStyle((string) $dataValidation['errorStyle']);
-                    $docValidation->setOperator((string) $dataValidation['operator']);
-                    $docValidation->setAllowBlank(filter_var($dataValidation['allowBlank'], FILTER_VALIDATE_BOOLEAN));
-                    // showDropDown is inverted (works as hideDropDown if true)
-                    $docValidation->setShowDropDown(!filter_var($dataValidation['showDropDown'], FILTER_VALIDATE_BOOLEAN));
-                    $docValidation->setShowInputMessage(filter_var($dataValidation['showInputMessage'], FILTER_VALIDATE_BOOLEAN));
-                    $docValidation->setShowErrorMessage(filter_var($dataValidation['showErrorMessage'], FILTER_VALIDATE_BOOLEAN));
-                    $docValidation->setErrorTitle((string) $dataValidation['errorTitle']);
-                    $docValidation->setError((string) $dataValidation['error']);
-                    $docValidation->setPromptTitle((string) $dataValidation['promptTitle']);
-                    $docValidation->setPrompt((string) $dataValidation['prompt']);
-                    $docValidation->setFormula1(Xlsx::replacePrefixes((string) $dataValidation->formula1));
-                    $docValidation->setFormula2(Xlsx::replacePrefixes((string) $dataValidation->formula2));
-                    $docValidation->setSqref($range);
-                }
+                $docValidation = new DataValidation();
+                $docValidation->setType((string) $dataValidation['type']);
+                $docValidation->setErrorStyle((string) $dataValidation['errorStyle']);
+                $docValidation->setOperator((string) $dataValidation['operator']);
+                $docValidation->setAllowBlank(filter_var($dataValidation['allowBlank'], FILTER_VALIDATE_BOOLEAN));
+                // showDropDown is inverted (works as hideDropDown if true)
+                $docValidation->setShowDropDown(!filter_var($dataValidation['showDropDown'], FILTER_VALIDATE_BOOLEAN));
+                $docValidation->setShowInputMessage(filter_var($dataValidation['showInputMessage'], FILTER_VALIDATE_BOOLEAN));
+                $docValidation->setShowErrorMessage(filter_var($dataValidation['showErrorMessage'], FILTER_VALIDATE_BOOLEAN));
+                $docValidation->setErrorTitle((string) $dataValidation['errorTitle']);
+                $docValidation->setError((string) $dataValidation['error']);
+                $docValidation->setPromptTitle((string) $dataValidation['promptTitle']);
+                $docValidation->setPrompt((string) $dataValidation['prompt']);
+                $docValidation->setFormula1(Xlsx::replacePrefixes((string) $dataValidation->formula1));
+                $docValidation->setFormula2(Xlsx::replacePrefixes((string) $dataValidation->formula2));
+                $docValidation->setSqref($range);
+                $this->worksheet->setDataValidation($range, $docValidation);
             }
         }
     }

src/PhpSpreadsheet/Worksheet/Worksheet.php

@@ -3265,6 +3265,14 @@ public function getDataValidation(string $cellCoordinate): DataValidation
         if (isset($this->dataValidationCollection[$cellCoordinate])) {
             return $this->dataValidationCollection[$cellCoordinate];
         }
+        foreach ($this->dataValidationCollection as $dataValidation) {
+            $sqref = $dataValidation->getSqref() ?? '';
+            if (str_contains($sqref, ':')) {
+                if (Coordinate::coordinateIsInsideRange($sqref, $cellCoordinate)) {
+                    return $dataValidation;
+                }
+            }
+        }
 
         // else create data validation
         $this->dataValidationCollection[$cellCoordinate] = new DataValidation();
@@ -3297,7 +3305,17 @@ public function setDataValidation(string $cellCoordinate, ?DataValidation $dataV
      */
     public function dataValidationExists(string $coordinate): bool
     {
-        return isset($this->dataValidationCollection[$coordinate]);
+        if (isset($this->dataValidationCollection[$coordinate])) {
+            return true;
+        }
+        foreach ($this->dataValidationCollection as $dataValidation) {
+            $sqref = $dataValidation->getSqref() ?? '';
+            if (str_contains($sqref, ':')) {
+                if (Coordinate::coordinateIsInsideRange($sqref, $coordinate)) {
+                    return true;
+                }
+            }
+        }
     }
 
     /**

src/PhpSpreadsheet/Writer/Xlsx/Worksheet.php

@@ -1313,8 +1313,17 @@ private function writeSheetData(XMLWriter $objWriter, PhpspreadsheetWorksheet $w
         $cellsByRow = [];
         foreach ($worksheet->getCoordinates() as $coordinate) {
             [$column, $row] = Coordinate::coordinateFromString($coordinate);
-            $cellsByRow[$row] = $cellsByRow[$row] ?? '';
-            $cellsByRow[$row] .= "{$column},";
+            if (!isset($cellsByRow[$row])) {
+                $pCell = $worksheet->getCell("$column$row");
+                $xfi = $pCell->getXfIndex();
+                $cellValue = $pCell->getValue();
+                $writeValue = $cellValue !== '' && $cellValue !== null;
+                if (!empty($xfi) || $writeValue) {
+                    $cellsByRow[$row] = "{$column},";
+                }
+            } else {
+                $cellsByRow[$row] .= "{$column},";
+            }
         }
 
         $currentRow = 0;