Support for CBOR update queries (#1148)

Author: thomascorthals

CHANGELOG.md

@@ -6,6 +6,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 ### Added
+- CBOR formatted update requests
 
 ### Changed
 

composer.json

@@ -24,6 +24,7 @@
     "require-dev": {
         "ext-curl": "*",
         "ext-iconv": "*",
+        "2tvenom/cborencode": "^1.0",
         "escapestudios/symfony2-coding-standard": "^3.11",
         "nyholm/psr7": "^1.8",
         "php-http/guzzle7-adapter": "^1.0",
@@ -34,8 +35,12 @@
         "phpunit/phpunit": "^10.5",
         "rawr/phpunit-data-provider": "^3.3",
         "roave/security-advisories": "dev-master",
+        "spomky-labs/cbor-php": "^3.1",
         "symfony/event-dispatcher": "^5.0 || ^6.0 || ^7.0"
     },
+    "suggest": {
+        "spomky-labs/cbor-php": "Needed to use CBOR formatted requests with Solr 9.3+"
+    },
     "prefer-stable": true,
     "config": {
         "sort-packages": true,

docs/plugins.md

@@ -14,6 +14,7 @@ This can be done very easily with this plugin, you can simply keep feeding docum
 ### Some notes
 
 -   Solarium issues JSON formatted update requests by default. If you require XML specific functionality, you can set the request format to XML on the plugin instance. XML requests are slower than JSON.
+-   Solr 9.3 and higher also supports CBOR formatted update requests. You can set the request format to CBOR on the plugin instance if your documents adhere to [current limitations](queries/update-query/best-practices-for-updates.md#known-cbor-limitations).
 -   You can set a custom buffer size. The default is 100 documents, a safe value. By increasing this you can get even better performance, but depending on your document size at some level you will run into memory or request limits. A value of 1000 has been successfully used for indexing 200k documents.
 -   You can use the createDocument method with array input, but you can also manually create document instance and use the addDocument(s) method.
 -   With buffer size X an update request will be sent to Solr for each X docs. You can just keep feeding docs. These buffer flushes don’t include a commit. This is done on purpose. You can add a commit when you’re done, or you can use the Solr auto commit feature.

docs/queries/update-query/best-practices-for-updates.md

@@ -40,6 +40,31 @@ $update = $client->createUpdate();
 $update->setRequestFormat($update::REQUEST_FORMAT_XML);
 ```
 
-### Raw XML update commands
+#### Raw XML update commands
 
 Solarium makes it easy to build update commands without having to know the underlying XML structure. If you already have XML formatted update commands, you can add them directly to an update query. Make sure they are valid as Solarium will not check this, and set the [XML request format](#xml-vs-json-formatted-update-requests) on the update query.
+
+### CBOR formatted update requests
+
+Since Solr 9.3, Solr also supports the [CBOR format for indexing](https://solr.apache.org/guide/solr/latest/indexing-guide/indexing-with-cbor.html). While CBOR requests might be faster to handle by Solr, they are significantly slower and require more memory to build in Solarium than JSON or XML requests. Benchmark your own use cases to determine if this is the right choice for you.
+
+In order to use CBOR with Solarium, you need to install the `spomky-labs/cbor-php` library separately.
+
+```sh
+composer require spomky-labs/cbor-php
+```
+
+```php
+// get an update query instance
+$update = $client->createUpdate();
+
+// set CBOR request format
+$update->setRequestFormat($update::REQUEST_FORMAT_CBOR);
+```
+
+#### Known CBOR limitations
+
+As outlined in [SOLR-17510](https://issues.apache.org/jira/browse/SOLR-17510?focusedCommentId=17892000#comment-17892000), CBOR formatted updates currently have some limitations.
+
+- You can only add documents, other commands such as delete and commit aren't supported yet.
+- There is no support for atomic updates.

docs/queries/update-query/building-an-update-query/building-an-update-query.md

@@ -1,5 +1,5 @@
 An update query has options and commands. These commands and options are instructions for the client classes to build and execute a request and return the correct result. In the following sections both the options and commands will be discussed in detail.
-You can also take a look at the [XML](https://solr.apache.org/guide/uploading-data-with-index-handlers.html#xml-formatted-index-updates) or [JSON](https://solr.apache.org/guide/uploading-data-with-index-handlers.html#json-formatted-index-updates) request formats for more information about the underlying Solr update handler.
+You can also take a look at the [XML](https://solr.apache.org/guide/uploading-data-with-index-handlers.html#xml-formatted-index-updates), [JSON](https://solr.apache.org/guide/uploading-data-with-index-handlers.html#json-formatted-index-updates), or [CBOR](https://solr.apache.org/guide/solr/latest/indexing-guide/indexing-with-cbor.html) request formats for more information about the underlying Solr update handler.
 
 Options
 -------
@@ -10,7 +10,7 @@ However, if you do need to customize them for a special case, you can.
 
 ### RequestFormat
 
-Solarium issues JSON formatted update requests by default. Set this to XML if you require XML specific functionality.
+Solarium issues JSON formatted update requests by default. Set this to XML if you require XML specific functionality. You can also set this to CBOR if you use Solr 9.3 or higher and your use case falls within [current limitations](../best-practices-for-updates.md#known-cbor-limitations).
 
 ### ResultClass
 

docs/queries/update-query/update-query.md

@@ -5,3 +5,4 @@ Update queries allow you to add, delete, commit, optimize and rollback commands.
 -   Always use a database or other persistent storage as the source for building documents to add. Don't be tempted to emulate an update command by selecting a document, altering it and adding it. Almost all schemas will have fields that are indexed and not stored. You will lose the data in those fields.
 -   The best way to use update queries is also related to your Solr config. If you are for instance using the autocommit feature of Solr you probably don't want to use a commit command in your update queries. Make sure you know the configuration details of the Solr core you use.
 -   Some functionality is only available with XML formatted or JSON formatted update queries, but not both. Set the appropriate request format if necessary.
+-   Solr 9.3 and higher also supports CBOR formatted update queries. Be aware that there are some [current limitations](best-practices-for-updates.md#known-cbor-limitations) with this request format.

examples/7.5.3-plugin-bufferedupdate-benchmarks.php

@@ -2,27 +2,31 @@
 
 require_once(__DIR__.'/init.php');
 
+use Composer\InstalledVersions;
 use Solarium\Core\Client\Adapter\TimeoutAwareInterface;
 use Solarium\Core\Client\Request;
+use Solarium\QueryType\Update\Query\Query;
 
 set_time_limit(0);
-ini_set('memory_limit', ini_get('suhosin.memory_limit') ?: '-1');
+ini_set('memory_limit', -1);
 ob_implicit_flush(true);
 @ob_end_flush();
 
 htmlHeader();
 
-if (!isset($weight) || !isset($requestFormat)) {
+if (!isset($weight) || !isset($addRequestFormat) || !isset($delRequestFormat)) {
     echo <<<'EOT'
         <h1>Usage</h1>
 
-        <p>This file is intended to be included by a script that sets two variables:</p>
+        <p>This file is intended to be included by a script that sets three variables:</p>
 
         <dl>
             <dt><code>$weight</code></dt>
             <dd>Either <code>''</code> for the regular plugins or <code>'lite'</code> for the lite versions.</dd>
-            <dt><code>$requestFormat</code></dt>
+            <dt><code>$addRequestFormat</code></dt>
             <dd>Any of the <code>Solarium\QueryType\Update\Query\Query::REQUEST_FORMAT_*</code> constants.</dd>
+            <dt><code>$delRequestFormat</code></dt>
+            <dd><code>Solarium\QueryType\Update\Query\Query::REQUEST_FORMAT_JSON</code> or <code>REQUEST_FORMAT_XML</code>.</dd>
         </dl>
 
         <h2>Example</h2>
@@ -35,7 +39,8 @@
         use Solarium\QueryType\Update\Query\Query;
 
         $weight = '';
-        $requestFormat = Query::REQUEST_FORMAT_JSON;
+        $addRequestFormat = Query::REQUEST_FORMAT_JSON;
+        $delRequestFormat = Query::REQUEST_FORMAT_JSON;
 
         require(__DIR__.'/7.5.3-plugin-bufferedupdate-benchmarks.php');
         </pre>
@@ -46,6 +51,14 @@
     exit;
 }
 
+if (in_array(Query::REQUEST_FORMAT_CBOR, [$addRequestFormat, $delRequestFormat]) && !InstalledVersions::isInstalled('spomky-labs/cbor-php')) {
+    echo '<h2>Note: The CBOR benchmark requires spomky-labs/cbor-php</h2>';
+
+    htmlFooter();
+
+    exit;
+}
+
 echo '<h2>Note: These benchmarks can take some time to run!</h2>';
 
 // create a client instance and don't let the adapter timeout
@@ -76,10 +89,10 @@
 $addBuffer = $client->getPlugin($addPlugin = 'bufferedadd'.$weight);
 $delBuffer = $client->getPlugin($delPlugin = 'buffereddelete'.$weight);
 
-$addBuffer->setRequestFormat($requestFormat);
-$delBuffer->setRequestFormat($requestFormat);
+$addBuffer->setRequestFormat($addRequestFormat);
+$delBuffer->setRequestFormat($delRequestFormat);
 
-echo '<h3>'.$addPlugin.' / '.$delPlugin.' ('.strtoupper($requestFormat).')</h3>';
+echo '<h3>'.$addPlugin.' ('.strtoupper($addRequestFormat).') / '.$delPlugin.' ('.strtoupper($delRequestFormat).')</h3>';
 echo '<table><thead>';
 echo '<tr><th>add buffer size</th><th>add time</th>';
 echo '<th>delete buffer size</th><th>delete time</th>';

examples/7.5.3.0-plugin-bufferedupdate-benchmarks-build-only.php

@@ -0,0 +1,119 @@
+<?php
+
+require_once(__DIR__.'/init.php');
+
+use Composer\InstalledVersions;
+use Solarium\Core\Client\Response;
+use Solarium\Core\Event\Events;
+use Solarium\Core\Event\PreExecuteRequest;
+use Solarium\QueryType\Update\Query\Query;
+
+set_time_limit(0);
+ini_set('memory_limit', -1);
+ob_implicit_flush(true);
+@ob_end_flush();
+
+htmlHeader();
+
+echo '<h2>Note: These benchmarks build the requests but don\'t execute them</h2>';
+
+$requestFormats = [
+    Query::REQUEST_FORMAT_XML,
+    Query::REQUEST_FORMAT_JSON,
+];
+
+if (InstalledVersions::isInstalled('spomky-labs/cbor-php')) {
+    $requestFormats[] = Query::REQUEST_FORMAT_CBOR;
+} else {
+    echo '<h2>Note: The CBOR benchmark requires spomky-labs/cbor-php</h2>';
+}
+
+// memory usage is only useful with PHP 8.2+, earlier version don't allow resetting between benchmarks
+$withMemoryUsage = function_exists('memory_reset_peak_usage');
+
+// create a client instance
+$client = new Solarium\Client($adapter, $eventDispatcher, $config);
+
+// autoload the buffered add plugin
+$addBuffer = $client->getPlugin('bufferedaddlite');
+
+// return a dummy response instead of executing the query
+$response = new Response('', ['HTTP/1.0 200 OK']);
+
+// keep measures for individual build times
+$start = 0;
+$buildTimes = [];
+
+$client->getEventDispatcher()->addListener(
+    Events::PRE_EXECUTE_REQUEST,
+    function (PreExecuteRequest $event) use ($response, &$start, &$buildTimes) {
+        $buildTimes[] = (hrtime(true) - $start) / 1000000;
+        $event->setResponse($response);
+        $start = hrtime(true);
+    }
+);
+
+$docs = 1200000;
+
+foreach ($requestFormats as $requestFormat) {
+    $addBuffer->setRequestFormat($requestFormat);
+
+    echo '<h3>'.strtoupper($requestFormat).'</h3>';
+    echo '<table><thead>';
+    echo '<tr><th rowspan="2">buffer size</th><th colspan="5">build time</th>';
+    if ($withMemoryUsage) {
+        echo '<th rowspan="2">mem peak usage</th>';
+    }
+    echo '</tr>';
+    echo '<tr><th>min</th><th>max</th><th>mean</th><th>median</th><th>total</th></tr>';
+    echo '</thead><tbody style="text-align:right">';
+
+    foreach ([2000, 200, 20, 2] as $flushes) {
+        $bufferSize = $docs / $flushes;
+
+        $addBuffer->setBufferSize($bufferSize);
+
+        echo '<tr><td>'.$bufferSize.'</td>';
+
+        $buildTimes = [];
+
+        if ($withMemoryUsage) {
+            memory_reset_peak_usage();
+        }
+
+        $start = hrtime(true);
+
+        for ($i = 0; $i < $docs; ++$i) {
+            $data = [
+                'id' => sprintf('test-%08d', $i),
+                'name' => 'test for buffered add',
+                'cat' => ['solarium-test', 'solarium-test-bufferedadd'],
+            ];
+            $addBuffer->createDocument($data);
+        }
+
+        sort($buildTimes);
+        $halfway = $flushes / 2;
+        $total = array_sum($buildTimes);
+        $min = reset($buildTimes);
+        $max = end($buildTimes);
+        $mean = $total / $flushes;
+        $median = ($buildTimes[$halfway - 1] + $buildTimes[$halfway]) / 2;
+        echo '<td>'.(int) $min.' ms</td>';
+        echo '<td>'.(int) $max.' ms</td>';
+        echo '<td>'.(int) $mean.' ms</td>';
+        echo '<td>'.(int) $median.' ms</td>';
+        echo '<td>'.(int) $total.' ms</td>';
+
+        if ($withMemoryUsage) {
+            $memoryPeakUsage = memory_get_peak_usage() / 1024;
+            echo '<td>'.(int) $memoryPeakUsage.' KiB</td>';
+        }
+
+        echo '</tr>';
+    }
+
+    echo '</tbody></table>';
+}
+
+htmlFooter();

examples/7.5.3.1-plugin-bufferedupdate-benchmarks-xml.php

@@ -5,6 +5,7 @@
 use Solarium\QueryType\Update\Query\Query;
 
 $weight = '';
-$requestFormat = Query::REQUEST_FORMAT_XML;
+$addRequestFormat = Query::REQUEST_FORMAT_XML;
+$delRequestFormat = Query::REQUEST_FORMAT_XML;
 
 require(__DIR__.'/7.5.3-plugin-bufferedupdate-benchmarks.php');

examples/7.5.3.2-plugin-bufferedupdate-lite-benchmarks-xml.php

@@ -5,6 +5,7 @@
 use Solarium\QueryType\Update\Query\Query;
 
 $weight = 'lite';
-$requestFormat = Query::REQUEST_FORMAT_XML;
+$addRequestFormat = Query::REQUEST_FORMAT_XML;
+$delRequestFormat = Query::REQUEST_FORMAT_XML;
 
 require(__DIR__.'/7.5.3-plugin-bufferedupdate-benchmarks.php');