Support for CBOR update queries

CHANGELOG.md

@@ -6,6 +6,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 ### Added
+- CBOR formatted update requests
 
 ### Changed
 

composer.json

@@ -24,6 +24,7 @@
     "require-dev": {
         "ext-curl": "*",
         "ext-iconv": "*",
+        "2tvenom/cborencode": "^1.0",
         "escapestudios/symfony2-coding-standard": "^3.11",
         "nyholm/psr7": "^1.8",
         "php-http/guzzle7-adapter": "^1.0",
@@ -34,8 +35,12 @@
         "phpunit/phpunit": "^10.5",
         "rawr/phpunit-data-provider": "^3.3",
         "roave/security-advisories": "dev-master",
+        "spomky-labs/cbor-php": "^3.1",
         "symfony/event-dispatcher": "^5.0 || ^6.0 || ^7.0"
     },
+    "suggest": {
+        "spomky-labs/cbor-php": "Needed to use CBOR formatted requests with Solr 9.3+"
+    },
     "prefer-stable": true,
     "config": {
         "sort-packages": true,

docs/plugins.md

@@ -14,6 +14,7 @@ This can be done very easily with this plugin, you can simply keep feeding docum
 ### Some notes
 
 -   Solarium issues JSON formatted update requests by default. If you require XML specific functionality, you can set the request format to XML on the plugin instance. XML requests are slower than JSON.
+-   Solr 9.3 and higher also supports CBOR formatted update requests. You can set the request format to CBOR on the plugin instance if your documents adhere to [current limitations](../queries/update-query/best-practices-for-updates#known-cbor-limitations).
 -   You can set a custom buffer size. The default is 100 documents, a safe value. By increasing this you can get even better performance, but depending on your document size at some level you will run into memory or request limits. A value of 1000 has been successfully used for indexing 200k documents.
 -   You can use the createDocument method with array input, but you can also manually create document instance and use the addDocument(s) method.
 -   With buffer size X an update request will be sent to Solr for each X docs. You can just keep feeding docs. These buffer flushes don’t include a commit. This is done on purpose. You can add a commit when you’re done, or you can use the Solr auto commit feature.

docs/queries/update-query/best-practices-for-updates.md

@@ -40,6 +40,31 @@ $update = $client->createUpdate();
 $update->setRequestFormat($update::REQUEST_FORMAT_XML);
 ```
 
-### Raw XML update commands
+#### Raw XML update commands
 
 Solarium makes it easy to build update commands without having to know the underlying XML structure. If you already have XML formatted update commands, you can add them directly to an update query. Make sure they are valid as Solarium will not check this, and set the [XML request format](#xml-vs-json-formatted-update-requests) on the update query.
+
+### CBOR vs JSON formatted update requests
+
+Since Solr 9.3, Solr also supports the [CBOR format for indexing](https://solr.apache.org/guide/solr/latest/indexing-guide/indexing-with-cbor.html).
+
+In order to use CBOR, you need to install the `spomky-labs/cbor-php` library first.
+
+```sh
+composer require spomky-labs/cbor-php
+```
+
+```php
+// get an update query instance
+$update = $client->createUpdate();
+
+// set CBOR request format
+$update->setRequestFormat($update::REQUEST_FORMAT_CBOR);
+```
+
+#### Known CBOR limitations
+
+As outlined in [SOLR-17510](https://issues.apache.org/jira/browse/SOLR-17510?focusedCommentId=17892000#comment-17892000), CBOR formatted updates currently have some limitations.
+
+- You can only add documents, other commands such as delete and commit aren't supported yet.
+- There is no support for atomic updates.

docs/queries/update-query/building-an-update-query/building-an-update-query.md

@@ -10,7 +10,7 @@ However, if you do need to customize them for a special case, you can.
 
 ### RequestFormat
 
-Solarium issues JSON formatted update requests by default. Set this to XML if you require XML specific functionality.
+Solarium issues JSON formatted update requests by default. Set this to XML if you require XML specific functionality. You can also set this to CBOR if you use Solr 9.3 or higher and your use case falls within [current limitations](../../best-practices-for-updates#known-cbor-limitations).
 
 ### ResultClass
 

docs/queries/update-query/update-query.md

@@ -5,3 +5,4 @@ Update queries allow you to add, delete, commit, optimize and rollback commands.
 -   Always use a database or other persistent storage as the source for building documents to add. Don't be tempted to emulate an update command by selecting a document, altering it and adding it. Almost all schemas will have fields that are indexed and not stored. You will lose the data in those fields.
 -   The best way to use update queries is also related to your Solr config. If you are for instance using the autocommit feature of Solr you probably don't want to use a commit command in your update queries. Make sure you know the configuration details of the Solr core you use.
 -   Some functionality is only available with XML formatted or JSON formatted update queries, but not both. Set the appropriate request format if necessary.
+-   Solr 9.3 and higher also supports CBOR formatted update queries. Be aware that there are some [current limitations](../best-practices-for-updates#known-cbor-limitations) with this request format.

examples/7.5.3-plugin-bufferedupdate-benchmarks.php

@@ -12,17 +12,19 @@
 
 htmlHeader();
 
-if (!isset($weight) || !isset($requestFormat)) {
+if (!isset($weight) || !isset($addRequestFormat) || !isset($delRequestFormat)) {
     echo <<<'EOT'
         <h1>Usage</h1>
 
-        <p>This file is intended to be included by a script that sets two variables:</p>
+        <p>This file is intended to be included by a script that sets three variables:</p>
 
         <dl>
             <dt><code>$weight</code></dt>
             <dd>Either <code>''</code> for the regular plugins or <code>'lite'</code> for the lite versions.</dd>
-            <dt><code>$requestFormat</code></dt>
+            <dt><code>$addRequestFormat</code></dt>
             <dd>Any of the <code>Solarium\QueryType\Update\Query\Query::REQUEST_FORMAT_*</code> constants.</dd>
+            <dt><code>$delRequestFormat</code></dt>
+            <dd><code>Solarium\QueryType\Update\Query\Query::REQUEST_FORMAT_JSON</code> or <code>REQUEST_FORMAT_XML</code>.</dd>
         </dl>
 
         <h2>Example</h2>
@@ -35,7 +37,8 @@
         use Solarium\QueryType\Update\Query\Query;
 
         $weight = '';
-        $requestFormat = Query::REQUEST_FORMAT_JSON;
+        $addRequestFormat = Query::REQUEST_FORMAT_JSON;
+        $delRequestFormat = Query::REQUEST_FORMAT_JSON;
 
         require(__DIR__.'/7.5.3-plugin-bufferedupdate-benchmarks.php');
         </pre>
@@ -76,10 +79,10 @@
 $addBuffer = $client->getPlugin($addPlugin = 'bufferedadd'.$weight);
 $delBuffer = $client->getPlugin($delPlugin = 'buffereddelete'.$weight);
 
-$addBuffer->setRequestFormat($requestFormat);
-$delBuffer->setRequestFormat($requestFormat);
+$addBuffer->setRequestFormat($addRequestFormat);
+$delBuffer->setRequestFormat($delRequestFormat);
 
-echo '<h3>'.$addPlugin.' / '.$delPlugin.' ('.strtoupper($requestFormat).')</h3>';
+echo '<h3>'.$addPlugin.' ('.strtoupper($addRequestFormat).') / '.$delPlugin.' ('.strtoupper($delRequestFormat).')</h3>';
 echo '<table><thead>';
 echo '<tr><th>add buffer size</th><th>add time</th>';
 echo '<th>delete buffer size</th><th>delete time</th>';

examples/7.5.3.1-plugin-bufferedupdate-benchmarks-xml.php

@@ -5,6 +5,7 @@
 use Solarium\QueryType\Update\Query\Query;
 
 $weight = '';
-$requestFormat = Query::REQUEST_FORMAT_XML;
+$addRequestFormat = Query::REQUEST_FORMAT_XML;
+$delRequestFormat = Query::REQUEST_FORMAT_XML;
 
 require(__DIR__.'/7.5.3-plugin-bufferedupdate-benchmarks.php');

examples/7.5.3.2-plugin-bufferedupdate-lite-benchmarks-xml.php

@@ -5,6 +5,7 @@
 use Solarium\QueryType\Update\Query\Query;
 
 $weight = 'lite';
-$requestFormat = Query::REQUEST_FORMAT_XML;
+$addRequestFormat = Query::REQUEST_FORMAT_XML;
+$delRequestFormat = Query::REQUEST_FORMAT_XML;
 
 require(__DIR__.'/7.5.3-plugin-bufferedupdate-benchmarks.php');

examples/7.5.3.3-plugin-bufferedupdate-benchmarks-json.php

@@ -5,6 +5,7 @@
 use Solarium\QueryType\Update\Query\Query;
 
 $weight = '';
-$requestFormat = Query::REQUEST_FORMAT_JSON;
+$addRequestFormat = Query::REQUEST_FORMAT_JSON;
+$delRequestFormat = Query::REQUEST_FORMAT_JSON;
 
 require(__DIR__.'/7.5.3-plugin-bufferedupdate-benchmarks.php');

examples/7.5.3.4-plugin-bufferedupdate-lite-benchmarks-json.php

@@ -5,6 +5,7 @@
 use Solarium\QueryType\Update\Query\Query;
 
 $weight = 'lite';
-$requestFormat = Query::REQUEST_FORMAT_JSON;
+$addRequestFormat = Query::REQUEST_FORMAT_JSON;
+$delRequestFormat = Query::REQUEST_FORMAT_JSON;
 
 require(__DIR__.'/7.5.3-plugin-bufferedupdate-benchmarks.php');

examples/7.5.3.5-plugin-bufferedupdate-benchmarks-cbor.php

@@ -0,0 +1,12 @@
+<?php
+
+require_once(__DIR__.'/init.php');
+
+use Solarium\QueryType\Update\Query\Query;
+
+$weight = '';
+$addRequestFormat = Query::REQUEST_FORMAT_CBOR;
+// CBOR can only be used to add documents
+$delRequestFormat = Query::REQUEST_FORMAT_JSON;
+
+require(__DIR__.'/7.5.3-plugin-bufferedupdate-benchmarks.php');

examples/7.5.3.6-plugin-bufferedupdate-lite-benchmarks-cbor.php

@@ -0,0 +1,12 @@
+<?php
+
+require_once(__DIR__.'/init.php');
+
+use Solarium\QueryType\Update\Query\Query;
+
+$weight = 'lite';
+$addRequestFormat = Query::REQUEST_FORMAT_CBOR;
+// CBOR can only be used to add documents
+$delRequestFormat = Query::REQUEST_FORMAT_JSON;
+
+require(__DIR__.'/7.5.3-plugin-bufferedupdate-benchmarks.php');

examples/execute_all.php

@@ -163,6 +163,8 @@
         '7.5.3.2-plugin-bufferedupdate-lite-benchmarks-xml.php', // takes too long for a workflow, can be run manually
         '7.5.3.3-plugin-bufferedupdate-benchmarks-json.php', // takes too long for a workflow, can be run manually
         '7.5.3.4-plugin-bufferedupdate-lite-benchmarks-json.php', // takes too long for a workflow, can be run manually
+        '7.5.3.5-plugin-bufferedupdate-benchmarks-cbor.php', // takes too long for a workflow, can be run manually
+        '7.5.3.6-plugin-bufferedupdate-lite-benchmarks-cbor.php', // takes too long for a workflow, can be run manually
         '7.9-plugin-nowaitforresponserequest.php', // there is no default suggester included with techproducts
     ];
 

examples/index.html

@@ -205,6 +205,8 @@ <h2>Examples</h2>
                         <li><a href="7.5.3.2-plugin-bufferedupdate-lite-benchmarks-xml.php">7.5.3.2 Lite plugins with XML requests</a></li>
                         <li><a href="7.5.3.3-plugin-bufferedupdate-benchmarks-json.php">7.5.3.3 Regular plugins with JSON requests</a></li>
                         <li><a href="7.5.3.4-plugin-bufferedupdate-lite-benchmarks-json.php">7.5.3.4 Lite plugins with JSON requests</a></li>
+                        <li><a href="7.5.3.5-plugin-bufferedupdate-benchmarks-cbor.php">7.5.3.5 Regular plugins with CBOR requests</a></li>
+                        <li><a href="7.5.3.6-plugin-bufferedupdate-lite-benchmarks-cbor.php">7.5.3.6 Lite plugins with CBOR requests</a></li>
                     </ul>
                 </ul>
                 <li><a href="7.6-plugin-prefetchiterator.php">7.6 Prefetch iterator for select queries</a></li>

src/Core/Client/Request.php

@@ -46,6 +46,11 @@ class Request extends Configurable implements RequestParamsInterface
      */
     const METHOD_PUT = 'PUT';
 
+    /**
+     * Content-Type for CBOR payloads.
+     */
+    const CONTENT_TYPE_APPLICATION_CBOR = 'application/cbor';
+
     /**
      * Content-Type for JSON payloads.
      */

src/Plugin/BufferedAdd/BufferedAdd.php

@@ -118,7 +118,16 @@ public function commit(?bool $overwrite = null, ?bool $softCommit = null, ?bool
         }
 
         $this->updateQuery->add(null, $command);
-        $this->updateQuery->addCommit($event->getSoftCommit(), $event->getWaitSearcher(), $event->getExpungeDeletes());
+
+        if ($this->updateQuery::REQUEST_FORMAT_CBOR === $this->getRequestFormat()) {
+            $this->updateQuery->addParam('commit', true);
+            $this->updateQuery->addParam('softCommit', $event->getSoftCommit());
+            $this->updateQuery->addParam('waitSearcher', $event->getWaitSearcher());
+            $this->updateQuery->addParam('expungeDeletes', $event->getExpungeDeletes());
+        } else {
+            $this->updateQuery->addCommit($event->getSoftCommit(), $event->getWaitSearcher(), $event->getExpungeDeletes());
+        }
+
         $result = $this->client->update($this->updateQuery, $this->getEndpoint());
         $this->clear();
 

src/Plugin/BufferedAdd/BufferedAddLite.php

@@ -201,7 +201,16 @@ public function commit(?bool $overwrite = null, ?bool $softCommit = null, ?bool
         }
 
         $this->updateQuery->add(null, $command);
-        $this->updateQuery->addCommit($softCommit, $waitSearcher, $expungeDeletes);
+
+        if ($this->updateQuery::REQUEST_FORMAT_CBOR === $this->getRequestFormat()) {
+            $this->updateQuery->addParam('commit', true);
+            $this->updateQuery->addParam('softCommit', $softCommit);
+            $this->updateQuery->addParam('waitSearcher', $waitSearcher);
+            $this->updateQuery->addParam('expungeDeletes', $expungeDeletes);
+        } else {
+            $this->updateQuery->addCommit($softCommit, $waitSearcher, $expungeDeletes);
+        }
+
         $result = $this->client->update($this->updateQuery, $this->getEndpoint());
         $this->clear();
 

src/Plugin/BufferedDelete/BufferedDeleteLite.php

@@ -9,6 +9,7 @@
 
 namespace Solarium\Plugin\BufferedDelete;
 
+use Solarium\Exception\InvalidArgumentException;
 use Solarium\Exception\RuntimeException;
 use Solarium\Plugin\AbstractBufferedUpdate\AbstractBufferedUpdate;
 use Solarium\Plugin\BufferedDelete\Delete\Id as DeleteById;
@@ -113,6 +114,26 @@ public function getDeletes(): array
         return $this->buffer;
     }
 
+    /**
+     * Set the request format for the updates.
+     *
+     * Use UpdateQuery::REQUEST_FORMAT_JSON or UpdateQuery::REQUEST_FORMAT_XML as value.
+     *
+     * @param string $requestFormat
+     *
+     * @throws InvalidArgumentException
+     *
+     * @return self Provides fluent interface
+     */
+    public function setRequestFormat(string $requestFormat): self
+    {
+        if ($this->updateQuery::REQUEST_FORMAT_CBOR === strtolower($requestFormat)) {
+            throw new InvalidArgumentException('Unsupported request format: CBOR can only be used to add documents');
+        }
+
+        return parent::setRequestFormat($requestFormat);
+    }
+
     /**
      * Flush any buffered deletes to Solr.
      *

src/QueryType/Update/Query/Query.php

@@ -23,6 +23,7 @@
 use Solarium\QueryType\Update\Query\Command\Optimize as OptimizeCommand;
 use Solarium\QueryType\Update\Query\Command\RawXml as RawXmlCommand;
 use Solarium\QueryType\Update\Query\Command\Rollback as RollbackCommand;
+use Solarium\QueryType\Update\RequestBuilder\Cbor as CborRequestBuilder;
 use Solarium\QueryType\Update\RequestBuilder\Json as JsonRequestBuilder;
 use Solarium\QueryType\Update\RequestBuilder\Xml as XmlRequestBuilder;
 use Solarium\QueryType\Update\ResponseParser;
@@ -67,6 +68,11 @@ class Query extends BaseQuery
      */
     const COMMAND_ROLLBACK = 'rollback';
 
+    /**
+     * CBOR request format.
+     */
+    const REQUEST_FORMAT_CBOR = 'cbor';
+
     /**
      * JSON request format.
      */
@@ -97,6 +103,7 @@ class Query extends BaseQuery
      * @var array
      */
     protected $requestFormats = [
+        self::REQUEST_FORMAT_CBOR => CborRequestBuilder::class,
         self::REQUEST_FORMAT_JSON => JsonRequestBuilder::class,
         self::REQUEST_FORMAT_XML => XmlRequestBuilder::class,
     ];