Misc release notes and changes for 3.12, highlight Read from followers feature (#345)

Simran-B · web-flow · commit c3748ef6a788 · 2023-11-22T09:48:22.000+01:00
* Projection optimizer rule changes

* Shorten menu title

* API changes, formatting

* DOC-597 | Improvement to Hot Backups

* --log.max-queued-entries startup option

* AQL operation title formatting

* Add Read from followers in clusters to EE features

* Sync release notes between versions

* Adjustable Stream Transactions size

* ArangoSearch file descriptor metric, headline tweak

* Fix metric help string, add to API changes

* Clarify CircleCI upstream references
diff --git a/CIRCLECI.md b/CIRCLECI.md
@@ -75,9 +75,9 @@ arguments are invoked:
 | Parameter type | Name | Value |
 |:---------------|:-----|:------|
 | string | `workflow` | `generate` |
-| string | `arangodb-3_10` | `{string in PR Template at 3.10}` |
-| string | `arangodb-3_11` | `{string in PR Template at 3.11}` |
-| string | `arangodb-3_12` | `{string in PR Template at 3.12}` |
+| string | `arangodb-3_10` | [Upstream reference](#upstream-reference) for 3.10 |
+| string | `arangodb-3_11` | [Upstream reference](#upstream-reference) for 3.11 |
+| string | `arangodb-3_12` | [Upstream reference](#upstream-reference) for 3.12 |
 | string | `generators` | `examples` |
 | string | `deploy-url` | `deploy-preview-{PR_NUMBER}` |
 
@@ -93,9 +93,9 @@ arguments are invoked:
 | Parameter type | Name | Value |
 |:---------------|:-----|:------|
 | string | `workflow` | `generate` |
-| string | `arangodb-3_10` | `{string in PR Template at 3.10}` |
-| string | `arangodb-3_11` | `{string in PR Template at 3.11}` |
-| string | `arangodb-3_12` | `{string in PR Template at 3.12}` |
+| string | `arangodb-3_10` | [Upstream reference](#upstream-reference) for 3.10 |
+| string | `arangodb-3_11` | [Upstream reference](#upstream-reference) for 3.11 |
+| string | `arangodb-3_12` | [Upstream reference](#upstream-reference) for 3.12 |
 | string | `generators` | `examples` |
 | string | `deploy-url` | `deploy-preview-{PR_NUMBER}` |
 | boolean | `commit-generated` | `true` |
@@ -121,13 +121,38 @@ or for multiple versions.
 | Parameter type | Name | Value |
 |:---------------|:-----|:------|
 | string | `workflow` | `generate` |
-| string | `arangodb-3_10` | `{string in PR Template at 3.10}` |
-| string | `arangodb-3_11` | `{string in PR Template at 3.11}` |
+| string | `arangodb-3_10` | [Upstream reference](#upstream-references) for 3.10 |
+| string | `arangodb-3_11` | [Upstream reference](#upstream-references) for 3.11 |
 | string | `generators` | `examples` |
 | boolean | `commit-generated` | `true` |
 | string | `deploy-url` | `deploy-preview-{PR_NUMBER}` |
 | string | `override` | `http,^aql.*` |
 
+### Upstream references
+
+Documentation pull requests specify upstream references like so:
+
+```markdown
+- 3.10: 
+- 3.11: https://github.com/arangodb/arangodb/pull/12345
+- 3.12: arangodb/enterprise-preview:devel-nightly
+```
+
+The above example indicates that ArangoDB versions 3.11 and 3.12 contain changes
+relevant to the docs PR, but 3.10 does not. Relevant changes are typically
+behavior changes of _arangod_ that will be visible in documentation examples.
+
+For 3.11, a link to a PR in the `arangodb/arangodb` repository is given. It is
+used by the GitHub integration to determine the feature branch to compile and
+use for generating examples. Do not specify a link when manually triggering a
+pipeline in CircleCI but the **branch name** (like `feature/new-aql-function`)!
+
+For 3.12, an ArangoDB Enterprise Edition image hosted on
+[Docker Hub](https://hub.docker.com/) is specified. Using Docker images has the
+advantage that the compilation of ArangoDB can be skipped, making the example
+generation faster. Of course, this requires that an image containing relevant
+changes to ArangoDB exists.
+
 ## Release workflow for ArangoDB releases
 
 To run a release job for a new ArangoDB patch release (e.g. 3.11.4),
diff --git a/site/content/3.10/about-arangodb/features/_index.md b/site/content/3.10/about-arangodb/features/_index.md
@@ -63,6 +63,7 @@ See all [Enterprise Edition Features](enterprise-edition.md).
 {{% /comment %}}
 | Only regular cluster deployments | **OneShard** deployment option to store all collections of a database on a single cluster node, to combine the performance of a single server and ACID semantics with a fault-tolerant cluster setup |
 | ACID transactions for multi-document / multi-collection queries on single servers, for single document operations in clusters, and for multi-document queries in clusters for collections with a single shard | In addition, ACID transactions for multi-collection queries using the OneShard feature |
+| Always read from leader shards in clusters | Optionally allow dirty reads to **read from followers** to scale reads |
 | TLS key and certificate rotation | In addition, **key rotation for JWT secrets** and **server name indication** (SNI) |
 | Built-in user management and authentication | Additional **LDAP authentication** option |
 | Only server logs | **Audit log** of server interactions |
diff --git a/site/content/3.10/about-arangodb/features/enterprise-edition.md b/site/content/3.10/about-arangodb/features/enterprise-edition.md
@@ -68,6 +68,10 @@ features outlined below. For additional information, see
   stored values, primary sort columns, and primary key columns in memory to
   improve the performance of Views and inverted indexes.
 
+- [**Read from followers in clusters**](../../develop/http-api/documents.md#read-from-followers):
+  Allow dirty reads so that Coordinators can read from any shard replica and not
+  only from the leader, for scaling reads.
+
 ## Querying
 
 - [**Pregel in Cluster**](../../data-science/pregel/_index.md#prerequisites):
diff --git a/site/content/3.10/about-arangodb/features/highlights-by-version.md b/site/content/3.10/about-arangodb/features/highlights-by-version.md
@@ -65,6 +65,10 @@ aliases:
   Optimized data loading for AQL traversal queries if only a few document
   attributes are accessed.
 
+- [**Read from followers in clusters**](../../develop/http-api/documents.md#read-from-followers):
+  Allow dirty reads so that Coordinators can read from any shard replica and not
+  only from the leader, for scaling reads.
+  
 Also see [What's New in 3.10](../../release-notes/version-3.10/whats-new-in-3-10.md).
 
 ## Version 3.9
diff --git a/site/content/3.10/develop/transactions/stream-transactions.md b/site/content/3.10/develop/transactions/stream-transactions.md
@@ -45,7 +45,7 @@ on the Coordinator to ensure that abandoned transactions cannot block the
 cluster from operating properly:
 
 - Maximum idle timeout of up to **120 seconds** between operations.
-- Maximum transaction size of **128 MB** per DB-Server.
+- Maximum transaction size of **128 MB** (per DB-Server in clusters).
 
 These limits are also enforced for Stream Transactions on single servers.
 
@@ -103,7 +103,7 @@ Additionally, `options` can have the following optional attributes:
   waiting on collection locks. This option is only meaningful when using
   `exclusive` locks. If not specified, a default value is used. Setting
   `lockTimeout` to `0` makes ArangoDB not time out waiting for a lock.
-- `maxTransactionSize`: Transaction size limit in bytes.
+- `maxTransactionSize`: Transaction size limit in bytes. Can be at most 128 MiB.
 
 The method returns an object that lets you run supported operations as part of
 the transactions, get the status information, and commit or abort the transaction.
diff --git a/site/content/3.10/release-notes/version-3.10/whats-new-in-3-10.md b/site/content/3.10/release-notes/version-3.10/whats-new-in-3-10.md
@@ -1048,7 +1048,12 @@ that have copies of the data. Therefore, the read throughput is higher.
 
 This feature is only available in the Enterprise Edition.
 
-For more information, see [Read from followers](../../develop/http-api/documents.md#read-from-followers).
+For more information, see [Read from followers](../../develop/http-api/documents.md#read-from-followers)
+in the HTTP API documentation.
+
+The JavaScript API supports an `allowDirtyReads` option for
+[AQL queries](../../aql/how-to-invoke-aql/with-arangosh.md#allowdirtyreads) and
+[reading documents](../../develop/javascript-api/@arangodb/collection-object.md#collectiondocumentobject--options).
 
 ## Improved shard rebalancing
 
diff --git a/site/content/3.11/about-arangodb/features/_index.md b/site/content/3.11/about-arangodb/features/_index.md
@@ -63,6 +63,7 @@ See all [Enterprise Edition Features](enterprise-edition.md).
 {{% /comment %}}
 | Only regular cluster deployments | **OneShard** deployment option to store all collections of a database on a single cluster node, to combine the performance of a single server and ACID semantics with a fault-tolerant cluster setup |
 | ACID transactions for multi-document / multi-collection queries on single servers, for single document operations in clusters, and for multi-document queries in clusters for collections with a single shard | In addition, ACID transactions for multi-collection queries using the OneShard feature |
+| Always read from leader shards in clusters | Optionally allow dirty reads to **read from followers** to scale reads |
 | TLS key and certificate rotation | In addition, **key rotation for JWT secrets** and **server name indication** (SNI) |
 | Built-in user management and authentication | Additional **LDAP authentication** option |
 | Only server logs | **Audit log** of server interactions |
diff --git a/site/content/3.11/about-arangodb/features/enterprise-edition.md b/site/content/3.11/about-arangodb/features/enterprise-edition.md
@@ -68,6 +68,10 @@ features outlined below. For additional information, see
   stored values, primary sort columns, and primary key columns in memory to
   improve the performance of Views and inverted indexes.
 
+- [**Read from followers in clusters**](../../develop/http-api/documents.md#read-from-followers):
+  Allow dirty reads so that Coordinators can read from any shard replica and not
+  only from the leader, for scaling reads.
+
 ## Querying
 
 - [**Pregel in Cluster**](../../data-science/pregel/_index.md#prerequisites):
diff --git a/site/content/3.11/about-arangodb/features/highlights-by-version.md b/site/content/3.11/about-arangodb/features/highlights-by-version.md
@@ -91,6 +91,10 @@ Also see [What's New in 3.11](../../release-notes/version-3.11/whats-new-in-3-11
   Optimized data loading for AQL traversal queries if only a few document
   attributes are accessed.
 
+- [**Read from followers in clusters**](../../develop/http-api/documents.md#read-from-followers):
+  Allow dirty reads so that Coordinators can read from any shard replica and not
+  only from the leader, for scaling reads.
+  
 Also see [What's New in 3.10](../../release-notes/version-3.10/whats-new-in-3-10.md).
 
 ## Version 3.9
diff --git a/site/content/3.11/develop/transactions/stream-transactions.md b/site/content/3.11/develop/transactions/stream-transactions.md
@@ -45,7 +45,7 @@ on the Coordinator to ensure that abandoned transactions cannot block the
 cluster from operating properly:
 
 - Maximum idle timeout of up to **120 seconds** between operations.
-- Maximum transaction size of **128 MB** per DB-Server.
+- Maximum transaction size of **128 MB** (per DB-Server in clusters).
 
 These limits are also enforced for Stream Transactions on single servers.
 
@@ -103,7 +103,7 @@ Additionally, `options` can have the following optional attributes:
   waiting on collection locks. This option is only meaningful when using
   `exclusive` locks. If not specified, a default value is used. Setting
   `lockTimeout` to `0` makes ArangoDB not time out waiting for a lock.
-- `maxTransactionSize`: Transaction size limit in bytes.
+- `maxTransactionSize`: Transaction size limit in bytes. Can be at most 128 MiB.
 
 The method returns an object that lets you run supported operations as part of
 the transactions, get the status information, and commit or abort the transaction.
diff --git a/site/content/3.11/release-notes/version-3.10/api-changes-in-3-10.md b/site/content/3.11/release-notes/version-3.10/api-changes-in-3-10.md
@@ -126,6 +126,23 @@ if the request body was an empty array. Example:
 
 Now, a request like this succeeds and returns an empty array as response.
 
+#### Limit to the number of databases in a deployment
+
+<small>Introduced in: v3.10.10</small>
+
+The new `--database.max-databases` startup option can cap the number of databases
+and creating databases using the `POST /_api/database` endpoint can thus now fail
+for this reason if your deployment is at or above the configured maximum. Example:
+
+```json
+{
+  "code": 400,
+  "error": true,
+  "errorMessage": "unable to create additional database because it would exceed the configured maximum number of databases (2)",
+  "errorNum": 32
+}
+```
+
 ### Endpoint return value changes
 
 - Since ArangoDB 3.8, there have been two APIs for retrieving the metrics in two
@@ -807,6 +824,18 @@ The following metrics have been added:
 | `arangodb_file_descriptors_limit` | System limit for the number of open files for the arangod process. |
 | `arangodb_file_descriptors_current` | Number of file descriptors currently opened by the arangod process. |
 
+---
+
+<small>Introduced in: v3.10.11</small>
+
+The following metrics have been added to improve the observability of in-memory
+cache subsystem:
+
+- `rocksdb_cache_free_memory_tasks_total`
+- `rocksdb_cache_free_memory_tasks_duration_total`
+- `rocksdb_cache_migrate_tasks_total`
+- `rocksdb_cache_migrate_tasks_duration_total`
+
 #### Pregel API
 
 When loading the graph data into memory, a `"loading"` state is now returned by
diff --git a/site/content/3.11/release-notes/version-3.10/whats-new-in-3-10.md b/site/content/3.11/release-notes/version-3.10/whats-new-in-3-10.md
@@ -1048,7 +1048,12 @@ that have copies of the data. Therefore, the read throughput is higher.
 
 This feature is only available in the Enterprise Edition.
 
-For more information, see [Read from followers](../../develop/http-api/documents.md#read-from-followers).
+For more information, see [Read from followers](../../develop/http-api/documents.md#read-from-followers)
+in the HTTP API documentation.
+
+The JavaScript API supports an `allowDirtyReads` option for
+[AQL queries](../../aql/how-to-invoke-aql/with-arangosh.md#allowdirtyreads) and
+[reading documents](../../develop/javascript-api/@arangodb/collection-object.md#collectiondocumentobject--options).
 
 ## Improved shard rebalancing
 
@@ -1384,6 +1389,20 @@ attempt to create an additional database fails with error
 if other databases are dropped first. The default value for this option is
 unlimited, so an arbitrary amount of databases can be created.
 
+### Configurable maximum for queued log entries
+
+<small>Introduced in: v3.10.12</small>
+
+The new `--log.max-queued-entries` startup option lets you configure how many
+log entries are queued in a background thread.
+
+Log entries are pushed on a queue for asynchronous writing unless you enable the
+`--log.force-direct` startup option. If you use a slow log output (e.g. syslog),
+the queue might grow and eventually overflow.
+
+You can configure the upper bound of the queue with this option. If the queue is
+full, log entries are written synchronously until the queue has space again.
+
 ## Miscellaneous changes
 
 ### Optimizer rules endpoint
@@ -1574,6 +1593,15 @@ The following system metrics have been added:
 | `arangodb_file_descriptors_limit` | System limit for the number of open files for the arangod process. |
 | `arangodb_file_descriptors_current` | Number of file descriptors currently opened by the arangod process. |
 
+### More instant Hot Backups
+
+<small>Introduced in: v3.10.10, v3.11.3</small>
+
+Cluster deployments no longer wait for all in-progress transactions to get
+committed when a user requests a Hot Backup. The waiting could cause deadlocks
+and thus Hot Backups to fail, in particular in ArangoGraph. Now, Hot Backups are
+created immediately and commits have to wait until the backup process is done.
+
 ## Client tools
 
 ### arangobench
diff --git a/site/content/3.12/about-arangodb/features/_index.md b/site/content/3.12/about-arangodb/features/_index.md
@@ -63,6 +63,7 @@ See all [Enterprise Edition Features](enterprise-edition.md).
 {{% /comment %}}
 | Only regular cluster deployments | **OneShard** deployment option to store all collections of a database on a single cluster node, to combine the performance of a single server and ACID semantics with a fault-tolerant cluster setup |
 | ACID transactions for multi-document / multi-collection queries on single servers, for single document operations in clusters, and for multi-document queries in clusters for collections with a single shard | In addition, ACID transactions for multi-collection queries using the OneShard feature |
+| Always read from leader shards in clusters | Optionally allow dirty reads to **read from followers** to scale reads |
 | TLS key and certificate rotation | In addition, **key rotation for JWT secrets** and **server name indication** (SNI) |
 | Built-in user management and authentication | Additional **LDAP authentication** option |
 | Only server logs | **Audit log** of server interactions |
diff --git a/site/content/3.12/about-arangodb/features/enterprise-edition.md b/site/content/3.12/about-arangodb/features/enterprise-edition.md
@@ -72,6 +72,10 @@ features outlined below. For additional information, see
   Retrieve search results for the highest-ranking matches from Views faster by
   defining a list of sort expressions to optimize.
 
+- [**Read from followers in clusters**](../../develop/http-api/documents.md#read-from-followers):
+  Allow dirty reads so that Coordinators can read from any shard replica and not
+  only from the leader, for scaling reads.
+
 ## Querying
 
 - [**Pregel in Cluster**](../../data-science/pregel/_index.md#prerequisites):
diff --git a/site/content/3.12/about-arangodb/features/highlights-by-version.md b/site/content/3.12/about-arangodb/features/highlights-by-version.md
@@ -105,6 +105,10 @@ Also see [What's New in 3.11](../../release-notes/version-3.11/whats-new-in-3-11
   Optimized data loading for AQL traversal queries if only a few document
   attributes are accessed.
 
+- [**Read from followers in clusters**](../../develop/http-api/documents.md#read-from-followers):
+  Allow dirty reads so that Coordinators can read from any shard replica and not
+  only from the leader, for scaling reads.
+  
 Also see [What's New in 3.10](../../release-notes/version-3.10/whats-new-in-3-10.md).
 
 ## Version 3.9
diff --git a/site/content/3.12/develop/transactions/stream-transactions.md b/site/content/3.12/develop/transactions/stream-transactions.md
@@ -45,10 +45,13 @@ on the Coordinator to ensure that abandoned transactions cannot block the
 cluster from operating properly:
 
 - Maximum idle timeout of up to **120 seconds** between operations.
-- Maximum transaction size of **128 MB** per DB-Server.
+- Maximum transaction size with a default of **128 MiB** (per DB-Server in clusters).
 
 These limits are also enforced for Stream Transactions on single servers.
 
+The maximum size for a single Stream Transaction can be adjusted with the
+`--transaction.streaming-max-transaction-size` startup option.
+
 The default maximum idle timeout is **60 seconds** between operations in a
 single Stream Transaction. The maximum value can be bumped up to at most 120
 seconds by setting the `--transaction.streaming-idle-timeout` startup option.
@@ -103,7 +106,8 @@ Additionally, `options` can have the following optional attributes:
   waiting on collection locks. This option is only meaningful when using
   `exclusive` locks. If not specified, a default value is used. Setting
   `lockTimeout` to `0` makes ArangoDB not time out waiting for a lock.
-- `maxTransactionSize`: Transaction size limit in bytes.
+- `maxTransactionSize`: Transaction size limit in bytes. Can be at most the
+  value of the `--transaction.streaming-max-transaction-size` startup option.
 
 The method returns an object that lets you run supported operations as part of
 the transactions, get the status information, and commit or abort the transaction.
diff --git a/site/content/3.12/release-notes/version-3.10/api-changes-in-3-10.md b/site/content/3.12/release-notes/version-3.10/api-changes-in-3-10.md
@@ -126,6 +126,23 @@ if the request body was an empty array. Example:
 
 Now, a request like this succeeds and returns an empty array as response.
 
+#### Limit to the number of databases in a deployment
+
+<small>Introduced in: v3.10.10</small>
+
+The new `--database.max-databases` startup option can cap the number of databases
+and creating databases using the `POST /_api/database` endpoint can thus now fail
+for this reason if your deployment is at or above the configured maximum. Example:
+
+```json
+{
+  "code": 400,
+  "error": true,
+  "errorMessage": "unable to create additional database because it would exceed the configured maximum number of databases (2)",
+  "errorNum": 32
+}
+```
+
 ### Endpoint return value changes
 
 - Since ArangoDB 3.8, there have been two APIs for retrieving the metrics in two
@@ -807,6 +824,18 @@ The following metrics have been added:
 | `arangodb_file_descriptors_limit` | System limit for the number of open files for the arangod process. |
 | `arangodb_file_descriptors_current` | Number of file descriptors currently opened by the arangod process. |
 
+---
+
+<small>Introduced in: v3.10.11</small>
+
+The following metrics have been added to improve the observability of in-memory
+cache subsystem:
+
+- `rocksdb_cache_free_memory_tasks_total`
+- `rocksdb_cache_free_memory_tasks_duration_total`
+- `rocksdb_cache_migrate_tasks_total`
+- `rocksdb_cache_migrate_tasks_duration_total`
+
 #### Pregel API
 
 When loading the graph data into memory, a `"loading"` state is now returned by
diff --git a/site/content/3.12/release-notes/version-3.10/whats-new-in-3-10.md b/site/content/3.12/release-notes/version-3.10/whats-new-in-3-10.md
diff --git a/site/content/3.12/release-notes/version-3.11/api-changes-in-3-11.md b/site/content/3.12/release-notes/version-3.11/api-changes-in-3-11.md
diff --git a/site/content/3.12/release-notes/version-3.11/whats-new-in-3-11.md b/site/content/3.12/release-notes/version-3.11/whats-new-in-3-11.md
diff --git a/site/content/3.12/release-notes/version-3.12/api-changes-in-3-12.md b/site/content/3.12/release-notes/version-3.12/api-changes-in-3-12.md
diff --git a/site/content/3.12/release-notes/version-3.12/incompatible-changes-in-3-12.md b/site/content/3.12/release-notes/version-3.12/incompatible-changes-in-3-12.md
diff --git a/site/content/3.12/release-notes/version-3.12/whats-new-in-3-12.md b/site/content/3.12/release-notes/version-3.12/whats-new-in-3-12.md
