BTS-1854 | String comparisons and sorting order may differ between core (ICU 64) and JavaScript (ICU 73)

site/content/3.12/release-notes/version-3.12/incompatible-changes-in-3-12.md

@@ -170,6 +170,23 @@ onward and will be removed in a future version.
 You can use [Stream Transactions](../../develop/transactions/stream-transactions.md)
 instead in most cases, and in some cases AQL can be sufficient.
 
+## Incompatibilities with Unicode text between core and JavaScript
+
+ArangoDB 3.12 uses the ICU library for Unicode handling in version 64 for its core
+(ArangoSearch, AQL, RocksDB) but version 73 in [JavaScript contexts](../../develop/javascript-api/_index.md).
+If you compare or sort string values with JavaScript and with the core, the values
+may not match between the two or have a different order. This is due to changes
+in the Unicode standard and the binary representation of strings for comparisons.
+
+You can be affected if you use JavaScript-based features like Foxx microservices
+or user-defined AQL functions (UDFs), compare or sort strings in them, and
+Unicode characters for which the standard has changed between the two ICU versions
+are involved.
+
+{{< comment >}}
+TODO: May become relevant later should we upgrade the core ICU.
+If not, we still might want to incorporate some of this into the reference docs.
+
 ## Breaking changes to the `collation` Analyzer
 
 The [`collation` Analyzer](../../index-and-search/analyzers.md#collation) lets
@@ -190,6 +207,7 @@ or older versions may behave in unpredicted ways.
 Note that sorting by the output of the `collation` Analyzer like
 `SORT TOKENS(<text>, <collationAnalyzer>)` is still not a supported feature and
 doesn't produce meaningful results.
+{{< /comment >}}
 
 ## Control character escaping in audit log
 

site/content/3.12/release-notes/version-3.12/known-issues-in-3-12.md

@@ -46,3 +46,4 @@ Note that this page does not list all open issues.
 | **Date Added:** 2024-03-21 <br> **Component:** arangod <br> **Deployment Mode:** All <br> **Description:** When creating an `inverted` index with the `inBackground` option enabled, HTTP API calls like `http://localhost:8529/_api/index?collection=<coll>&withHidden=true` don't return the `isBuilding` and `progress` attributes and the progress of the index building can thus not be observed. <br> **Affected Versions:** 3.10.13, 3.11.7, 3.12.x <br> **Fixed in Versions:** - <br> **Reference:** [BTS-1788](https://arangodb.atlassian.net/browse/BTS-1788) (internal) |
 | **Date Added:** 2024-03-28 <br> **Component:** arangod <br> **Deployment Mode:** Cluster <br> **Description:** During startup or upgrade from a previous minor version, Agent nodes crash if the `--cluster.force-one-shard` option is enabled. Workaround: Don't use the `--cluster.force-one-shard` option (or set it to `false`) for Agents. <br> **Affected Versions:** 3.12.0 <br> **Fixed in Versions:** 3.12.1 <br> **Reference:** [BTS-1839](https://arangodb.atlassian.net/browse/BTS-1839) (internal) |
 | **Date Added:** 2024-03-28 <br> **Component:** arangod <br> **Deployment Mode:** Cluster <br> **Description:** In a cluster, creating an EnterpriseGraph fails in OneShard databases (created with the option `{"sharding": "single"}`). EnterpriseGraphs can still be created in a single server deployment, if the sharding option was not set to `single` during the database creation. <br> **Affected Versions:** 3.12.x <br> **Fixed in Versions:** - <br> **Reference:** [BTS-1841](https://arangodb.atlassian.net/browse/BTS-1841) (internal) |
+| **Date Added:** 2024-04-24 <br> **Component:** arangod <br> **Deployment Mode:** All <br> **Description:** ArangoDB uses the ICU library for Unicode handling in version 64 for its core (ArangoSearch, AQL, RocksDB) but version 73 in [JavaScript contexts](../../develop/javascript-api/_index.md) since v3.12.0. If you compare or sort string values with JavaScript and with the core, the values may not match or have a different order. This is due to changes in the Unicode standard and the binary representation of strings for comparisons. You can be affected if you use JavaScript-based features like Foxx microservices or user-defined AQL functions (UDFs), compare or sort strings in them, and Unicode characters for which the standard has changed between the two ICU versions are involved. <br> **Affected Versions:** 3.12.x <br> **Fixed in Versions:** - <br> **Reference:** [BTS-1854](https://arangodb.atlassian.net/browse/BTS-1854) (internal) |

site/content/3.12/release-notes/version-3.12/whats-new-in-3-12.md

@@ -716,8 +716,9 @@ full, log entries are written synchronously until the queue has space again.
 ### V8 and ICU library upgrades
 
 The bundled V8 JavaScript engine has been upgraded from version 7.9.317 to
-12.1.165. As part of this upgrade, the bundled Unicode character handling library
-ICU has been upgraded as well, from version 64.2 to 73.1.
+12.1.165. As part of this upgrade, the Unicode character handling library
+ICU has been upgraded as well, from version 64.2 to 73.1 (but only for
+JavaScript contexts, see [Incompatible changes in ArangoDB 3.12](incompatible-changes-in-3-12.md#incompatibilities-with-unicode-text-between-core-and-javascript)).
 
 Note that ArangoDB's build of V8 has pointer compression disabled to allow for
 more than 4 GB of heap memory.

site/content/3.13/release-notes/version-3.12/incompatible-changes-in-3-12.md

@@ -170,6 +170,23 @@ onward and will be removed in a future version.
 You can use [Stream Transactions](../../develop/transactions/stream-transactions.md)
 instead in most cases, and in some cases AQL can be sufficient.
 
+## Incompatibilities with Unicode text between core and JavaScript
+
+ArangoDB 3.12 uses the ICU library for Unicode handling in version 64 for its core
+(ArangoSearch, AQL, RocksDB) but version 73 in [JavaScript contexts](../../develop/javascript-api/_index.md).
+If you compare or sort string values with JavaScript and with the core, the values
+may not match between the two or have a different order. This is due to changes
+in the Unicode standard and the binary representation of strings for comparisons.
+
+You can be affected if you use JavaScript-based features like Foxx microservices
+or user-defined AQL functions (UDFs), compare or sort strings in them, and
+Unicode characters for which the standard has changed between the two ICU versions
+are involved.
+
+{{< comment >}}
+TODO: May become relevant later should we upgrade the core ICU.
+If not, we still might want to incorporate some of this into the reference docs.
+
 ## Breaking changes to the `collation` Analyzer
 
 The [`collation` Analyzer](../../index-and-search/analyzers.md#collation) lets
@@ -190,6 +207,7 @@ or older versions may behave in unpredicted ways.
 Note that sorting by the output of the `collation` Analyzer like
 `SORT TOKENS(<text>, <collationAnalyzer>)` is still not a supported feature and
 doesn't produce meaningful results.
+{{< /comment >}}
 
 ## Control character escaping in audit log
 

site/content/3.13/release-notes/version-3.12/known-issues-in-3-12.md

@@ -46,3 +46,4 @@ Note that this page does not list all open issues.
 | **Date Added:** 2024-03-21 <br> **Component:** arangod <br> **Deployment Mode:** All <br> **Description:** When creating an `inverted` index with the `inBackground` option enabled, HTTP API calls like `http://localhost:8529/_api/index?collection=<coll>&withHidden=true` don't return the `isBuilding` and `progress` attributes and the progress of the index building can thus not be observed. <br> **Affected Versions:** 3.10.13, 3.11.7, 3.12.x <br> **Fixed in Versions:** - <br> **Reference:** [BTS-1788](https://arangodb.atlassian.net/browse/BTS-1788) (internal) |
 | **Date Added:** 2024-03-28 <br> **Component:** arangod <br> **Deployment Mode:** Cluster <br> **Description:** During startup or upgrade from a previous minor version, Agent nodes crash if the `--cluster.force-one-shard` option is enabled. Workaround: Don't use the `--cluster.force-one-shard` option (or set it to `false`) for Agents. <br> **Affected Versions:** 3.12.0 <br> **Fixed in Versions:** 3.12.1 <br> **Reference:** [BTS-1839](https://arangodb.atlassian.net/browse/BTS-1839) (internal) |
 | **Date Added:** 2024-03-28 <br> **Component:** arangod <br> **Deployment Mode:** Cluster <br> **Description:** In a cluster, creating an EnterpriseGraph fails in OneShard databases (created with the option `{"sharding": "single"}`). EnterpriseGraphs can still be created in a single server deployment, if the sharding option was not set to `single` during the database creation. <br> **Affected Versions:** 3.12.x <br> **Fixed in Versions:** - <br> **Reference:** [BTS-1841](https://arangodb.atlassian.net/browse/BTS-1841) (internal) |
+| **Date Added:** 2024-04-24 <br> **Component:** arangod <br> **Deployment Mode:** All <br> **Description:** ArangoDB uses the ICU library for Unicode handling in version 64 for its core (ArangoSearch, AQL, RocksDB) but version 73 in [JavaScript contexts](../../develop/javascript-api/_index.md) since v3.12.0. If you compare or sort string values with JavaScript and with the core, the values may not match or have a different order. This is due to changes in the Unicode standard and the binary representation of strings for comparisons. You can be affected if you use JavaScript-based features like Foxx microservices or user-defined AQL functions (UDFs), compare or sort strings in them, and Unicode characters for which the standard has changed between the two ICU versions are involved. <br> **Affected Versions:** 3.12.x <br> **Fixed in Versions:** - <br> **Reference:** [BTS-1854](https://arangodb.atlassian.net/browse/BTS-1854) (internal) |

site/content/3.13/release-notes/version-3.12/whats-new-in-3-12.md

@@ -716,8 +716,9 @@ full, log entries are written synchronously until the queue has space again.
 ### V8 and ICU library upgrades
 
 The bundled V8 JavaScript engine has been upgraded from version 7.9.317 to
-12.1.165. As part of this upgrade, the bundled Unicode character handling library
-ICU has been upgraded as well, from version 64.2 to 73.1.
+12.1.165. As part of this upgrade, the Unicode character handling library
+ICU has been upgraded as well, from version 64.2 to 73.1 (but only for
+JavaScript contexts, see [Incompatible changes in ArangoDB 3.12](incompatible-changes-in-3-12.md#incompatibilities-with-unicode-text-between-core-and-javascript)).
 
 Note that ArangoDB's build of V8 has pointer compression disabled to allow for
 more than 4 GB of heap memory.